📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,274 @@
|
||||
# Advanced RAG Cookbook
|
||||
|
||||
Build advanced RAG functionality with Weaviate.
|
||||
|
||||
Read first:
|
||||
- Basic RAG cookbook, important to start from this base. MUST READ: [Basic RAG Cookbook](./basic_rag.md)
|
||||
|
||||
Docs to reference if needed:
|
||||
- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
|
||||
- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
|
||||
- Vector search: https://docs.weaviate.io/weaviate/search/similarity
|
||||
- Keyword search: https://docs.weaviate.io/weaviate/search/bm25
|
||||
- Hybrid search: https://docs.weaviate.io/weaviate/search/hybrid
|
||||
- Image search: https://docs.weaviate.io/weaviate/search/image
|
||||
|
||||
|
||||
## Core Rules
|
||||
|
||||
First implement the basic strategy from [here](./basic_rag.md). Then modify according to this guide.
|
||||
|
||||
- Use a virtual environment via `venv`
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this install set: `uv add weaviate-client python-dotenv dspy weaviate-agents`
|
||||
- Customise this cookbook to the users specification, ask them for details if not given.
|
||||
|
||||
Assume the user has data already to be used, do not create data unless asked to.
|
||||
|
||||
Instead of following this cookbook, you first must ask the user if they would prefer to use the Weaviate Query Agent. If so, all steps in this guide can be implemented with the query agent which does advanced RAG out of the box.
|
||||
|
||||
Query agent docs: https://docs.weaviate.io/agents/query/usage
|
||||
|
||||
## Env Rules
|
||||
|
||||
Mandatory:
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
External provider keys:
|
||||
- Fill only keys actually used by the target Weaviate collection setup.
|
||||
|
||||
## Advanced RAG overview
|
||||
|
||||
* Query re-writer: *Change user input text into a query text using an LLM*
|
||||
* Query decomposition: *Change query into multiple sub-queries each re-written with an LLM*
|
||||
* Filtering: *Use an LLM to define filters on the collection*
|
||||
* Re-ranking: *Score the final results by a more advanced model*
|
||||
* Prompt engineering: *Add chain of thought, Tree of thoughts, ReAct*
|
||||
|
||||
## Query Re-writer
|
||||
|
||||
```python
|
||||
class QueryRewriter(dspy.Signature):
|
||||
"""
|
||||
Rewrite the user's query into a more relevant search term that is a more relevant search term for searching a database.
|
||||
"""
|
||||
input_query: str = dspy.InputField(description="The original user query")
|
||||
rewritten_query: str = dspy.OutputField(
|
||||
description=(
|
||||
"A single search term that is more relevant to the user's query. "
|
||||
"Include only relevant information, it does not need to be a full sentence or question "
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Modify the `query_transformation` function:
|
||||
|
||||
```python
|
||||
def query_transformation(query: str) -> list[str]:
|
||||
lm = dspy.LM(subtask_model_name)
|
||||
answer = dspy.Predict(QueryRewriter)
|
||||
pred = answer(input_query=query, lm=lm)
|
||||
return [pred.rewritten_query]
|
||||
```
|
||||
|
||||
## Query Decomposition
|
||||
|
||||
```python
|
||||
class QueryRewriter(dspy.Signature):
|
||||
"""
|
||||
Rewrite the user's query into a more relevant search terms that are more relevant search term for searching a database.
|
||||
"""
|
||||
input_query: str = dspy.InputField(description="The original user query")
|
||||
rewritten_queries: list[str] = dspy.OutputField(
|
||||
description=(
|
||||
"A list of search terms that are more relevant to the user's query. "
|
||||
"Each entry should include only relevant information, it does not need to be a full sentence or question "
|
||||
"Split independent searches into different entries "
|
||||
"Each entry should be relevant independently that capture a different required search aspect "
|
||||
"Do not repeat similar search terms, each one should have a unique meaning "
|
||||
"Be sparse, do not duplicate search terms "
|
||||
)
|
||||
)
|
||||
|
||||
def query_transformation(query: str) -> list[str]:
|
||||
lm = dspy.LM(subtask_model_name)
|
||||
answer = dspy.Predict(QueryRewriter)
|
||||
pred = answer(input_query=query, lm=lm)
|
||||
return pred.rewritten_queries
|
||||
```
|
||||
|
||||
## LLM-created Filters
|
||||
|
||||
Filters can be specified by the user (for specific use-cases, perhaps), or you can get an LLM to write the filters also. Writing filters requires knowledge of the collection schema. This can be retrieved by advanced methods or a simple version can be used.
|
||||
|
||||
Simple version:
|
||||
|
||||
1. First create structured responses to format filters
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Literal, Any
|
||||
|
||||
class SearchFilter(BaseModel):
|
||||
field: str = Field(description="The field to be filtered on.")
|
||||
operator: Literal["=", "!=", ">", "<"] = Field(description="The operator to be used in conjunction with the value. These are strict operators.")
|
||||
value: Any = Field(description="The value to be used in conjunction with the operator.")
|
||||
|
||||
class Search(BaseModel):
|
||||
filters: list[SearchFilter] = Field(description="The filters to be used in the vector database. This is an AND operation.")
|
||||
|
||||
class SearchCreation(dspy.Signature):
|
||||
"""
|
||||
Create filters and search parameters for a search query in a database.
|
||||
"""
|
||||
query: str = dspy.InputField()
|
||||
schema: list[dict] = dspy.InputField(desc="Schema of the collection to be searched.")
|
||||
data_sample: list[dict] = dspy.InputField(desc="A sample of the data in the collection to be searched.")
|
||||
search: Search = dspy.OutputField(
|
||||
desc=(
|
||||
"Your filters and search parameters, this should be a valid JSON object. "
|
||||
"This should be constructed so that it matches the goal of the user prompt."
|
||||
)
|
||||
)
|
||||
```
|
||||
This requires `schema` and `data_sample` as an input field to the LLM call `SearchCreation`.
|
||||
|
||||
2. Helper function to turn structured response into weaviate filter
|
||||
|
||||
```python
|
||||
def _format_filters(search_filters: list[SearchFilter]):
|
||||
filters = []
|
||||
for search_filter in search_filters:
|
||||
base_filter = Filter.by_property(search_filter.field)
|
||||
if search_filter.operator == "=":
|
||||
filter = base_filter.equal(search_filter.value)
|
||||
elif search_filter.operator == "!=":
|
||||
filter = base_filter.not_equal(search_filter.value)
|
||||
elif search_filter.operator == ">":
|
||||
filter = base_filter.greater_than(search_filter.value)
|
||||
elif search_filter.operator == "<":
|
||||
filter = base_filter.less_than(search_filter.value)
|
||||
filters.append(filter)
|
||||
return Filter.all_of(filters) if filters else None
|
||||
```
|
||||
|
||||
3. Combine
|
||||
|
||||
```python
|
||||
def create_filters(query: str):
|
||||
|
||||
# import client here
|
||||
|
||||
collection = client.collections.use("<collection_name>")
|
||||
|
||||
# Get collection schema (for field names etc.). can replace this with more advanced configuration (like aggregating for unique groups)
|
||||
config = collection.config.get()
|
||||
schema = [{"name": p.name, "type": p.data_type[:]} for p in config.properties]
|
||||
|
||||
# Get a sample of the data in the collection to be searched
|
||||
data_sample = collection.query.fetch_objects(limit=5)
|
||||
|
||||
# Create search parameters
|
||||
search_parameters = dspy.ChainOfThought(SearchCreation)
|
||||
search_parameters_output = search_parameters(query=query, schema=schema, data_sample=data_sample, lm=dspy.LM(subtask_model_name))
|
||||
|
||||
return _format_filters(search_parameters_output.search.filters)
|
||||
```
|
||||
|
||||
These filters can be passed into the `collection.query.near_text` (or equivalent search function).
|
||||
|
||||
## Re-ranking
|
||||
|
||||
Do not modify the user's collection unless requested to do so. Re-ranking requires configuring the collection with a re-ranker, for example:
|
||||
|
||||
```python
|
||||
collection = client.collections.use("<collection_name>")
|
||||
collection.config.update(
|
||||
reranker_config=Reconfigure.Reranker.cohere()
|
||||
)
|
||||
```
|
||||
(this would require a Cohere API key).
|
||||
|
||||
Modify the `retrieve` function
|
||||
|
||||
```python
|
||||
from weaviate.classes.query import Rerank
|
||||
|
||||
def retrieve(query: str, limit: int | None = None, filters = []) -> list[dict]:
|
||||
|
||||
# ...existing code
|
||||
|
||||
response = collection.query.hybrid(
|
||||
query=query,
|
||||
limit=limit,
|
||||
rerank=Rerank(
|
||||
prop="content", # what field to re-rank on
|
||||
query=query # what the search term for the re-ranker should be (same as original in this case)
|
||||
),
|
||||
filters=filters if filters else None
|
||||
)
|
||||
|
||||
# ...existing code
|
||||
```
|
||||
|
||||
## Prompt Engineering
|
||||
|
||||
This step depends on the LLM framework used. You can manually ask the LLM to include reasoning before giving its final answer, adding a reasoning sub-field to be completed before giving the final answer in structured response, or specify in DSPy to use chain-of-thought.
|
||||
|
||||
```python
|
||||
class Generator(dspy.Signature):
|
||||
"""
|
||||
Answer the question based on the context.
|
||||
Do not include any information from external sources, only use the information provided in the context.
|
||||
If you cannot answer the question based on the information provided, say "I don't know".
|
||||
"""
|
||||
context: str | list[dict] = dspy.InputField(desc="The context to answer the question.")
|
||||
query: str = dspy.InputField(desc="The question to answer.")
|
||||
answer: str = dspy.OutputField(desc="The single answer to the question with no additional communication")
|
||||
```
|
||||
|
||||
Modify the `generate` function:
|
||||
|
||||
```python
|
||||
def generate(query: str, context: list[dict]) -> str:
|
||||
lm = dspy.LM(generation_model_name)
|
||||
answer = dspy.Predict(Generator)
|
||||
pred = answer(context=context, query=query, lm=lm)
|
||||
return pred.answer
|
||||
```
|
||||
|
||||
Consider other prompt engineering techniques like ReAct (if necessary but likely overkill), few-shot learning (requires advanced specification), or otherwise.
|
||||
|
||||
## Query Agent
|
||||
|
||||
Skip this guide altogether and use the Weaviate Query Agent.
|
||||
|
||||
```python
|
||||
from weaviate.agents.query import QueryAgent
|
||||
|
||||
# import client here
|
||||
|
||||
qa = QueryAgent(
|
||||
client=client, collections=["Example_Communications_Raw"]
|
||||
)
|
||||
response = qa.search("<user query here>") # just search with no text response
|
||||
response = qa.ask("<user query here>") # search with text response accessible via response.final_answer
|
||||
```
|
||||
|
||||
## Customisation Points
|
||||
|
||||
**LLM framework**
|
||||
|
||||
This guide used DSPy. Follow the guidelines in [here](./basic_rag.md), but most likely you will need an LLM framework involving structured responses.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
|
||||
- For any other issues, refer to the official library/package documentation and use web search extensively for troubleshooting.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Create test scripts to check each function works independently with test data. Tear down tests after completion, or create a proper test suite with pytest (requires install)
|
||||
- User has completed specification of the app.
|
||||
@@ -0,0 +1,360 @@
|
||||
# Agentic RAG Cookbook
|
||||
|
||||
Build RAG-powered AI agents with Weaviate.
|
||||
|
||||
Read first:
|
||||
- Basic Agent cookbook, important to start from this base. MUST READ: [Basic Agent Cookbook](./basic_agent.md)
|
||||
|
||||
Docs to reference if needed:
|
||||
- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
|
||||
- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
|
||||
- Hybrid search: https://docs.weaviate.io/weaviate/search/hybrid
|
||||
- Weaviate Query Agent: https://docs.weaviate.io/agents/query/usage
|
||||
- Elysia: https://weaviate.github.io/elysia/
|
||||
|
||||
|
||||
## Core Rules
|
||||
|
||||
First implement the basic agent from [here](./basic_agent.md). Then modify according to this guide.
|
||||
|
||||
- Use a virtual environment via `venv`
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this install set: `uv add weaviate-client python-dotenv dspy`
|
||||
- Add `weaviate-agents` if using the Query Agent: `uv add "weaviate-client[agents]"`
|
||||
- Add `elysia-ai` if using Elysia: `uv add elysia-ai`
|
||||
- Customise this cookbook to the users specification, ask them for details if not given.
|
||||
|
||||
Assume the user has data already to be used, do not create data unless asked to.
|
||||
|
||||
Instead of following this cookbook, you first must ask the user if they would prefer to use the Weaviate Query Agent or Elysia. If so, skip to the relevant section below.
|
||||
|
||||
- Query Agent docs: https://docs.weaviate.io/agents/query/usage
|
||||
- Elysia docs: https://weaviate.github.io/elysia/
|
||||
|
||||
## Env Rules
|
||||
|
||||
Mandatory:
|
||||
- An LLM provider API key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`)
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
External provider keys:
|
||||
- Fill only keys actually used by the target Weaviate collection setup.
|
||||
|
||||
|
||||
## Agentic RAG Overview
|
||||
|
||||
* Naive RAG tool: *Basic retrieval as a single tool for the RouterAgent*
|
||||
* Hierarchical RAG: *LLM-created filters and search parameters as a sub-agent tool*
|
||||
* Vector DB memory: *Store and retrieve facts across sessions using Weaviate*
|
||||
* Query Agent: *Pre-built agentic RAG service by Weaviate*
|
||||
* Elysia: *Open source agentic framework with built-in query tool*
|
||||
|
||||
|
||||
## Naive RAG Tool
|
||||
|
||||
A simple retrieval tool that the RouterAgent can call. Pass this as a tool to the RouterAgent from the [basic agent cookbook](./basic_agent.md).
|
||||
|
||||
```python
|
||||
from weaviate import connect_to_weaviate_cloud
|
||||
import os
|
||||
|
||||
def retrieve_data(query: str):
|
||||
"""
|
||||
Given a query (free text), return the most relevant documents from the vector database using hybrid search.
|
||||
"""
|
||||
client = connect_to_weaviate_cloud(
|
||||
cluster_url=os.getenv("WEAVIATE_URL", ""),
|
||||
auth_credentials=os.getenv("WEAVIATE_API_KEY", ""),
|
||||
)
|
||||
collection = client.collections.use("<collection_name>")
|
||||
response = collection.query.hybrid(query=query, limit=5)
|
||||
client.close()
|
||||
return f"{[obj.properties for obj in response.objects]}"
|
||||
```
|
||||
|
||||
Customise the search type (`hybrid`, `near_text`, `bm25`), `limit`, and return fields based on the use case.
|
||||
|
||||
|
||||
## Hierarchical RAG (LLM-created Filters)
|
||||
|
||||
Instead of simple retrieval, use an LLM sub-agent to construct filters and search parameters. This makes the tool itself an agent.
|
||||
|
||||
1. Structured response models for filters:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Literal, Any
|
||||
|
||||
class SearchFilter(BaseModel):
|
||||
field: str = Field(description="The field to be filtered on.")
|
||||
operator: Literal["=", "!=", ">", "<"] = Field(description="The operator to be used in conjunction with the value.")
|
||||
value: Any = Field(description="The value to be used in conjunction with the operator.")
|
||||
|
||||
class Search(BaseModel):
|
||||
query: str = Field(description="The search query to be used in the vector database.")
|
||||
filters: list[SearchFilter] = Field(description="The filters to be used in the vector database.")
|
||||
limit: int = Field(description="The number of results to return from the vector database.")
|
||||
|
||||
class SearchCreation(dspy.Signature):
|
||||
"""
|
||||
Create a search query for a vector database.
|
||||
"""
|
||||
user_prompt: str = dspy.InputField()
|
||||
schema: list[dict] = dspy.InputField(desc="Schema of the collection to be searched.")
|
||||
search: Search = dspy.OutputField(
|
||||
desc=(
|
||||
"Your search query and filters, this should be a valid JSON object. "
|
||||
"This should be constructed so that it matches the goal of the user prompt."
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
2. Helper function to convert structured filters to Weaviate filters:
|
||||
|
||||
```python
|
||||
from weaviate.classes.query import Filter
|
||||
|
||||
def format_filters(search_filters: list[SearchFilter]):
|
||||
filters = []
|
||||
for search_filter in search_filters:
|
||||
base_filter = Filter.by_property(search_filter.field)
|
||||
if search_filter.operator == "=":
|
||||
filters.append(base_filter.equal(search_filter.value))
|
||||
elif search_filter.operator == "!=":
|
||||
filters.append(base_filter.not_equal(search_filter.value))
|
||||
elif search_filter.operator == ">":
|
||||
filters.append(base_filter.greater_than(search_filter.value))
|
||||
elif search_filter.operator == "<":
|
||||
filters.append(base_filter.less_than(search_filter.value))
|
||||
return Filter.all_of(filters) if filters else None
|
||||
```
|
||||
|
||||
3. The hierarchical query tool (replaces the naive retrieval tool):
|
||||
|
||||
```python
|
||||
def query_agent_tool(collection_name: str, user_prompt: str):
|
||||
"""
|
||||
Given a query (free text), return the most relevant documents from the vector database using hybrid search with LLM-generated filters.
|
||||
"""
|
||||
client = connect_to_weaviate_cloud(
|
||||
cluster_url=os.getenv("WEAVIATE_URL", ""),
|
||||
auth_credentials=os.getenv("WEAVIATE_API_KEY", ""),
|
||||
)
|
||||
collection = client.collections.use(collection_name)
|
||||
config = collection.config.get()
|
||||
schema = [{"name": p.name, "type": p.data_type[:]} for p in config.properties]
|
||||
|
||||
query_model = dspy.ChainOfThought(SearchCreation)
|
||||
query_output = query_model(
|
||||
user_prompt=user_prompt,
|
||||
schema=schema,
|
||||
lm=dspy.LM("<subtask_model_name>")
|
||||
)
|
||||
|
||||
response = collection.query.hybrid(
|
||||
query=query_output.search.query,
|
||||
filters=format_filters(query_output.search.filters),
|
||||
limit=query_output.search.limit
|
||||
)
|
||||
client.close()
|
||||
return f"{[obj.properties for obj in response.objects]}"
|
||||
```
|
||||
|
||||
Schema information is required for the LLM to construct filters. Fetch dynamically via `collection.config.get()` or provide manually if the schema is stable. Consider enriching the schema with sample data or enumerated values for better filter accuracy.
|
||||
|
||||
|
||||
## Vector Database Memory
|
||||
|
||||
Store and retrieve facts across sessions using Weaviate. Only add this if cross-session persistence is required.
|
||||
|
||||
1. Memory creation signature:
|
||||
|
||||
```python
|
||||
class MemoryCreation(dspy.Signature):
|
||||
user_prompt: str = dspy.InputField()
|
||||
assistant_response: str = dspy.InputField()
|
||||
memory: str = dspy.OutputField(
|
||||
description="A single string representing the most pertinent fact from the user/agent interaction."
|
||||
)
|
||||
```
|
||||
|
||||
2. Add `memories` as an input to `AgentResponse`:
|
||||
|
||||
```python
|
||||
class AgentResponse(dspy.Signature):
|
||||
|
||||
# Input Fields
|
||||
history: dspy.History = dspy.InputField()
|
||||
user_prompt: str = dspy.InputField()
|
||||
available_tools: str = dspy.InputField()
|
||||
memories: list[str] = dspy.InputField(
|
||||
desc="A list of memories from previous conversations, you can use these to inform your response."
|
||||
)
|
||||
|
||||
# Output Fields
|
||||
response: str = dspy.OutputField(
|
||||
description="The response to the user's prompt whilst the tool is running. Update the user on the progress of their request (if a tool is picked), or the final response to the user (if no tool is picked)."
|
||||
)
|
||||
tool: str | None = dspy.OutputField(
|
||||
description="The tool that needs to be used. Return None if no tool is needed."
|
||||
)
|
||||
tool_inputs: Dict[str, Any] | None = dspy.OutputField(
|
||||
description=(
|
||||
"The inputs for the tool. Return an empty dictionary (still include the field) if no inputs are needed. "
|
||||
"The key is the name of the input, the value is the value of the input."
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
3. Add `create_memory` and `retrieve_memories` methods to `RouterAgent`:
|
||||
|
||||
```python
|
||||
from weaviate import connect_to_weaviate_cloud
|
||||
from weaviate.classes.config import Configure
|
||||
|
||||
class RouterAgent:
|
||||
def __init__(self, model: str, memory_model: str | None = None, tools: List[Callable] = []):
|
||||
self.tools: list[Callable] = tools
|
||||
self.model = dspy.LM(model)
|
||||
self.memory_model = dspy.LM(memory_model) if memory_model else dspy.LM(model)
|
||||
self.agent = dspy.ChainOfThought(AgentResponse)
|
||||
self.memory_agent = dspy.Predict(MemoryCreation)
|
||||
self.conversation_history = dspy.History(messages=[])
|
||||
self.weaviate_client = connect_to_weaviate_cloud(
|
||||
cluster_url=os.getenv("WEAVIATE_URL", ""),
|
||||
auth_credentials=os.getenv("WEAVIATE_API_KEY", ""),
|
||||
)
|
||||
|
||||
# ... existing methods from basic_agent.md (add_conversation_history, get_tools_and_descriptions) ...
|
||||
|
||||
def create_memory(self, user_prompt: str, assistant_response: str, tool_result: str):
|
||||
if tool_result:
|
||||
assistant_response += "\n" + tool_result
|
||||
|
||||
result = self.memory_agent(
|
||||
history=self.conversation_history,
|
||||
user_prompt=user_prompt,
|
||||
assistant_response=assistant_response,
|
||||
lm=self.memory_model,
|
||||
)
|
||||
if not self.weaviate_client.collections.exists("Agent_Memory"):
|
||||
self.weaviate_client.collections.create(
|
||||
"Agent_Memory",
|
||||
vector_config=Configure.Vectors.text2vec_weaviate()
|
||||
)
|
||||
|
||||
collection = self.weaviate_client.collections.use("Agent_Memory")
|
||||
collection.data.insert({"user_prompt": user_prompt, "memory": result.memory})
|
||||
return result.memory
|
||||
|
||||
def retrieve_memories(self, user_prompt: str):
|
||||
if not self.weaviate_client.collections.exists("Agent_Memory"):
|
||||
return []
|
||||
collection = self.weaviate_client.collections.use("Agent_Memory")
|
||||
query = collection.query.near_text(query=user_prompt, limit=5)
|
||||
return [memory.properties["memory"] for memory in query.objects]
|
||||
```
|
||||
|
||||
Call `retrieve_memories` at the start of each interaction and pass results to the `memories` field of `AgentResponse`. Call `create_memory` after each interaction.
|
||||
|
||||
Consider using a cheaper model for memory creation (e.g. `memory_model="<cheap_model_name>"`).
|
||||
|
||||
|
||||
## Weaviate Query Agent
|
||||
|
||||
Skip the custom implementation and use the pre-built Weaviate Query Agent for agentic RAG. Handles collection selection, filter construction, and query optimisation automatically.
|
||||
|
||||
```python
|
||||
from weaviate.agents.query import QueryAgent
|
||||
|
||||
# import client here
|
||||
|
||||
qa = QueryAgent(
|
||||
client=client, collections=["<collection_name>"]
|
||||
)
|
||||
response = qa.search("<user query here>") # retrieval only
|
||||
response = qa.ask("<user query here>") # retrieval + text response via response.final_answer
|
||||
```
|
||||
|
||||
The Query Agent is free up to 1000 requests per month. Docs: https://docs.weaviate.io/agents/query/usage
|
||||
|
||||
|
||||
## Elysia
|
||||
|
||||
Elysia is an open source agentic framework with built-in query tools, decision trees, error handling, and automatic retry.
|
||||
|
||||
Setup:
|
||||
|
||||
```python
|
||||
import elysia
|
||||
from elysia.tools.text import FakeTextResponse as TextResponseTool
|
||||
|
||||
elysia.configure(
|
||||
base_model="<model_name>",
|
||||
base_provider="<provider>", # e.g. "anthropic", "openai"
|
||||
logging_level="ERROR"
|
||||
)
|
||||
```
|
||||
|
||||
With custom tools:
|
||||
|
||||
```python
|
||||
tree = elysia.Tree("empty", use_elysia_collections=False)
|
||||
tree.add_tool(TextResponseTool)
|
||||
|
||||
@elysia.tool
|
||||
async def your_tool(param: str):
|
||||
"""Tool description."""
|
||||
return {"result"}
|
||||
|
||||
tree.add_tool(your_tool)
|
||||
response, _ = tree("user query here")
|
||||
```
|
||||
|
||||
With built-in Weaviate query tool (requires preprocessing):
|
||||
|
||||
```python
|
||||
from elysia import preprocess
|
||||
preprocess("<collection_name>")
|
||||
|
||||
tree = elysia.Tree()
|
||||
response, _ = tree(
|
||||
"user query here",
|
||||
collection_names=["<collection_name>"]
|
||||
)
|
||||
```
|
||||
|
||||
Elysia includes built-in error handling, self-healing, and automatic retry. Also available as a standalone app with a frontend UI: https://github.com/weaviate/elysia
|
||||
|
||||
|
||||
## Customisation Points
|
||||
|
||||
**When to use which approach:**
|
||||
|
||||
| Use Case | Recommended Approach |
|
||||
|----------|---------------------|
|
||||
| Single collection, simple queries | Naive RAG tool |
|
||||
| Need filters or operators | Hierarchical RAG or Query Agent |
|
||||
| Multi-step tasks, multiple data sources | Sequential agent with agentic loop |
|
||||
| Cross-session personalisation | Add Vector Database Memory layer |
|
||||
| Production deployment with error handling | Use Elysia or Query Agent |
|
||||
|
||||
**Do not implement multi-agent architectures for simple retrieval tasks.**
|
||||
|
||||
**LLM framework**
|
||||
|
||||
This guide used DSPy. Follow the guidelines in [here](./basic_agent.md), but most likely you will need an LLM framework involving structured responses.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
|
||||
- DSPy signature warnings about missing fields: these can occur when using followup agents without all fields; ensure optional fields are handled.
|
||||
- For any other issues, refer to the official library/package documentation and use web search extensively for troubleshooting.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Create test scripts to check each function works independently with test data. Tear down tests after completion, or create a proper test suite with pytest (requires install)
|
||||
- User has completed specification of the app.
|
||||
@@ -0,0 +1,428 @@
|
||||
# Async Client Usage
|
||||
|
||||
Guide for using the Weaviate Python async client in production applications (FastAPI, async frameworks).
|
||||
|
||||
## 📚 Official Documentation Reference
|
||||
|
||||
**For agents:** If you encounter any issues not covered here, refer to the official Weaviate documentation:
|
||||
|
||||
- **Primary Reference**: [Weaviate Async API Documentation](https://docs.weaviate.io/weaviate/client-libraries/python/async)
|
||||
- **Python Client Reference**: [Weaviate Python Client Docs](https://docs.weaviate.io/weaviate/client-libraries/python)
|
||||
- **API Reference**: [ReadTheDocs - Python Client](https://weaviate-python-client.readthedocs.io/en/stable/)
|
||||
- **Troubleshooting**: [Community Forum](https://forum.weaviate.io/) | [GitHub Issues](https://github.com/weaviate/weaviate-python-client/issues)
|
||||
|
||||
> **Note**: The async client (`WeaviateAsyncClient`) is available in `weaviate-client` v4.7.0+.
|
||||
|
||||
## Connection Methods
|
||||
|
||||
Three instantiation helpers are provided ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#instantiation)):
|
||||
|
||||
### Weaviate Cloud (Recommended)
|
||||
|
||||
```python
|
||||
import weaviate
|
||||
from weaviate.classes.init import Auth
|
||||
|
||||
# Use the official helper function for Weaviate Cloud
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="your-cluster.weaviate.cloud", # Accepts hostname with or without https://
|
||||
auth_credentials=Auth.api_key("your-api-key"),
|
||||
headers={ # Note: parameter is "headers" not "additional_headers"
|
||||
"X-OpenAI-Api-Key": "sk-...",
|
||||
"X-Anthropic-Api-Key": "sk-ant-...",
|
||||
}
|
||||
)
|
||||
|
||||
await client.connect() # Required! Async helpers don't auto-connect
|
||||
```
|
||||
|
||||
**Reference**: [Weaviate Cloud Setup](https://docs.weaviate.io/weaviate/quickstart)
|
||||
|
||||
### Self-Hosted
|
||||
|
||||
```python
|
||||
# For local instances
|
||||
client = weaviate.use_async_with_local()
|
||||
|
||||
# For custom endpoints
|
||||
client = weaviate.use_async_with_custom(
|
||||
http_host="localhost",
|
||||
http_port=8080,
|
||||
http_secure=False,
|
||||
grpc_host="localhost",
|
||||
grpc_port=50051,
|
||||
grpc_secure=False,
|
||||
)
|
||||
|
||||
await client.connect()
|
||||
```
|
||||
|
||||
**Reference**: [Connection Configuration](https://weaviate-python-client.readthedocs.io/en/stable/weaviate.html)
|
||||
|
||||
### Authentication
|
||||
|
||||
Multiple authentication modes are supported ([docs](https://docs.weaviate.io/weaviate/client-libraries/python#authentication)):
|
||||
|
||||
```python
|
||||
from weaviate.classes.init import Auth
|
||||
|
||||
# API Key (most common for Weaviate Cloud)
|
||||
auth = Auth.api_key("your-api-key")
|
||||
|
||||
# Bearer Token (with optional refresh token)
|
||||
auth = Auth.bearer_token("access-token", refresh_token="refresh-token")
|
||||
|
||||
# Client Credentials (OIDC)
|
||||
auth = Auth.client_credentials(client_secret="secret")
|
||||
|
||||
# Client Password (OIDC Resource Owner Password flow)
|
||||
auth = Auth.client_password(username="user", password="pass")
|
||||
|
||||
# Usage
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="your-cluster.weaviate.cloud",
|
||||
auth_credentials=auth,
|
||||
)
|
||||
```
|
||||
|
||||
## Critical Patterns
|
||||
|
||||
### ⚠️ Connection Lifecycle
|
||||
|
||||
**Important**: Unlike synchronous helpers, async helpers **do not connect automatically** ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#instantiation)). You must explicitly call `.connect()` and `.close()`:
|
||||
|
||||
```python
|
||||
# ❌ Wrong - client not connected
|
||||
client = weaviate.use_async_with_weaviate_cloud(...)
|
||||
collections = await client.collections.list_all() # Will fail!
|
||||
|
||||
# ✅ Correct - explicit connect/close
|
||||
client = weaviate.use_async_with_weaviate_cloud(...)
|
||||
await client.connect()
|
||||
collections = await client.collections.list_all()
|
||||
await client.close()
|
||||
```
|
||||
|
||||
### ⚠️ Sync vs Async Methods
|
||||
|
||||
**Key distinction** ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#which-methods-are-async)): Methods involving server requests are async; local operations are synchronous.
|
||||
|
||||
```python
|
||||
# Collection retrieval is SYNC (no await)
|
||||
collection = client.collections.get("MyCollection")
|
||||
|
||||
# Operations on collections are ASYNC (need await)
|
||||
config = await collection.config.get()
|
||||
results = await collection.query.fetch_objects()
|
||||
count = await collection.aggregate.over_all()
|
||||
```
|
||||
|
||||
**Rule:** Getting the collection object is sync; calling methods on it is async.
|
||||
|
||||
### ⚠️ Bulk Operations
|
||||
|
||||
**Important Note** ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#bulk-import-operations)): For large-scale data imports, use the **synchronous client** and its batch operations. The sync client's batch methods already handle concurrency internally and are optimized for bulk operations.
|
||||
|
||||
```python
|
||||
# ✅ For bulk imports, prefer sync client
|
||||
import weaviate
|
||||
|
||||
with weaviate.connect_to_weaviate_cloud(...) as client:
|
||||
collection = client.collections.get("MyCollection")
|
||||
|
||||
# Batch insert handles concurrency automatically
|
||||
with collection.batch.dynamic() as batch:
|
||||
for item in large_dataset:
|
||||
batch.add_object(properties=item)
|
||||
```
|
||||
|
||||
Use the async client for:
|
||||
|
||||
- Web applications (FastAPI, Starlette)
|
||||
- Concurrent request handling
|
||||
- Interactive queries
|
||||
|
||||
Don't use the async client for:
|
||||
|
||||
- Bulk data imports (use sync client instead)
|
||||
|
||||
## Context Manager Pattern (Recommended)
|
||||
|
||||
**Best Practice** ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#using-the-async-context-manager)): Use `async with` to automatically connect/disconnect:
|
||||
|
||||
```python
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import AsyncGenerator
|
||||
|
||||
@asynccontextmanager
|
||||
async def get_weaviate_client(
|
||||
cluster_url: str,
|
||||
api_key: str,
|
||||
provider_headers: dict[str, str] | None = None,
|
||||
) -> AsyncGenerator[weaviate.WeaviateAsyncClient, None]:
|
||||
"""Connect to Weaviate Cloud with automatic cleanup."""
|
||||
# Remove scheme if present
|
||||
hostname = cluster_url.replace("https://", "").replace("http://", "")
|
||||
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url=hostname,
|
||||
auth_credentials=Auth.api_key(api_key),
|
||||
headers=provider_headers,
|
||||
)
|
||||
|
||||
try:
|
||||
await client.connect()
|
||||
yield client
|
||||
finally:
|
||||
await client.close()
|
||||
|
||||
# Usage
|
||||
async def example():
|
||||
async with get_weaviate_client(
|
||||
cluster_url="your-cluster.weaviate.cloud",
|
||||
api_key="your-key",
|
||||
) as client:
|
||||
collections = await client.collections.list_all()
|
||||
```
|
||||
|
||||
> **Note**: When using the context manager, `.connect()` and `.close()` are called automatically.
|
||||
|
||||
## FastAPI Integration
|
||||
|
||||
**Use Case** ([docs](https://docs.weaviate.io/weaviate/client-libraries/python/async#use-cases)): The async client excels in web frameworks like FastAPI for handling concurrent requests.
|
||||
|
||||
Use lifespan management for shared client across requests:
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup: connect to Weaviate
|
||||
app.state.weaviate = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="your-cluster.weaviate.cloud",
|
||||
auth_credentials=Auth.api_key("your-key"),
|
||||
)
|
||||
await app.state.weaviate.connect()
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown: close connection
|
||||
await app.state.weaviate.close()
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
|
||||
@app.get("/collections")
|
||||
async def list_collections():
|
||||
collections = await app.state.weaviate.collections.list_all()
|
||||
return {"collections": list(collections.keys())}
|
||||
```
|
||||
|
||||
**Community Discussion**: [FastAPI Best Practices](https://forum.weaviate.io/t/what-is-the-best-practice-to-use-v4-python-client-for-query-with-fastapi-or-other-async-python-framework/1245)
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### 1. Parameter Name Confusion
|
||||
|
||||
```python
|
||||
# ❌ Wrong - WeaviateAsyncClient() constructor uses different param
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
additional_headers={...} # Wrong parameter name!
|
||||
)
|
||||
|
||||
# ✅ Correct - use "headers" not "additional_headers"
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
headers={...}
|
||||
)
|
||||
```
|
||||
|
||||
### 2. URL Format
|
||||
|
||||
Both formats work with helper functions:
|
||||
|
||||
```python
|
||||
# ✅ Both accepted
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="https://cluster.weaviate.cloud" # With scheme
|
||||
)
|
||||
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="cluster.weaviate.cloud" # Without scheme
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Sync vs Async Function Names
|
||||
|
||||
```python
|
||||
# ❌ Wrong - sync client (cannot use await)
|
||||
client = weaviate.connect_to_weaviate_cloud(...)
|
||||
await client.connect() # TypeError!
|
||||
|
||||
# ✅ Correct - async client
|
||||
client = weaviate.use_async_with_weaviate_cloud(...)
|
||||
await client.connect()
|
||||
```
|
||||
|
||||
**Naming pattern:**
|
||||
|
||||
- Sync: `connect_to_*` (e.g., `connect_to_weaviate_cloud`)
|
||||
- Async: `use_async_with_*` (e.g., `use_async_with_weaviate_cloud`)
|
||||
|
||||
### 4. Port Configuration
|
||||
|
||||
```python
|
||||
# ❌ Wrong - manual port config causes conflicts with Weaviate Cloud
|
||||
client = WeaviateAsyncClient(
|
||||
connection_params=ConnectionParams.from_url(
|
||||
url="https://cluster.weaviate.cloud",
|
||||
grpc_port=443, # Conflict!
|
||||
)
|
||||
)
|
||||
|
||||
# ✅ Correct - use helper function (handles ports automatically)
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url="cluster.weaviate.cloud"
|
||||
)
|
||||
```
|
||||
|
||||
**Rule:** For Weaviate Cloud, always use `use_async_with_weaviate_cloud()` — it handles HTTP (443) and gRPC (50051) ports correctly.
|
||||
|
||||
## Multi-Cluster Example
|
||||
|
||||
Managing connections to multiple Weaviate clusters:
|
||||
|
||||
```python
|
||||
@asynccontextmanager
|
||||
async def get_multi_cluster_clients(
|
||||
clusters: dict[str, dict[str, str]]
|
||||
) -> AsyncGenerator[dict[str, weaviate.WeaviateAsyncClient], None]:
|
||||
"""Connect to multiple Weaviate clusters.
|
||||
|
||||
Args:
|
||||
clusters: Dict of {cluster_id: {"url": "...", "api_key": "..."}}
|
||||
"""
|
||||
clients = {}
|
||||
|
||||
try:
|
||||
# Connect to all clusters
|
||||
for cluster_id, config in clusters.items():
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url=config["url"],
|
||||
auth_credentials=Auth.api_key(config["api_key"]),
|
||||
)
|
||||
await client.connect()
|
||||
clients[cluster_id] = client
|
||||
|
||||
yield clients
|
||||
|
||||
finally:
|
||||
# Close all connections
|
||||
for client in clients.values():
|
||||
await client.close()
|
||||
|
||||
# Usage
|
||||
async def example():
|
||||
clusters = {
|
||||
"prod": {"url": "prod.weaviate.cloud", "api_key": "key1"},
|
||||
"dev": {"url": "dev.weaviate.cloud", "api_key": "key2"},
|
||||
}
|
||||
|
||||
async with get_multi_cluster_clients(clusters) as clients:
|
||||
prod_collections = await clients["prod"].collections.list_all()
|
||||
dev_collections = await clients["dev"].collections.list_all()
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See [Environment Requirements](environment_requirements.md) for provider API keys.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
# Read from environment
|
||||
cluster_url = os.environ["WEAVIATE_URL"]
|
||||
api_key = os.environ["WEAVIATE_API_KEY"]
|
||||
|
||||
# Build provider headers
|
||||
provider_headers = {}
|
||||
if openai_key := os.getenv("OPENAI_API_KEY"):
|
||||
provider_headers["X-OpenAI-Api-Key"] = openai_key
|
||||
if anthropic_key := os.getenv("ANTHROPIC_API_KEY"):
|
||||
provider_headers["X-Anthropic-Api-Key"] = anthropic_key
|
||||
|
||||
client = weaviate.use_async_with_weaviate_cloud(
|
||||
cluster_url=cluster_url,
|
||||
auth_credentials=Auth.api_key(api_key),
|
||||
headers=provider_headers or None,
|
||||
)
|
||||
```
|
||||
|
||||
## Testing Async Code
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weaviate_connection():
|
||||
async with get_weaviate_client(
|
||||
cluster_url="test-cluster.weaviate.cloud",
|
||||
api_key="test-key",
|
||||
) as client:
|
||||
collections = await client.collections.list_all()
|
||||
assert isinstance(collections, dict)
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Pattern | Await? |
|
||||
| ---------------- | --------------------------------------------- | ------- |
|
||||
| Create client | `weaviate.use_async_with_weaviate_cloud(...)` | No |
|
||||
| Connect | `client.connect()` | **Yes** |
|
||||
| Get collection | `client.collections.get("Name")` | No |
|
||||
| List collections | `client.collections.list_all()` | **Yes** |
|
||||
| Query data | `collection.query.fetch_objects()` | **Yes** |
|
||||
| Get config | `collection.config.get()` | **Yes** |
|
||||
| Aggregate | `collection.aggregate.over_all()` | **Yes** |
|
||||
| Close | `client.close()` | **Yes** |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution | Reference |
|
||||
| ----------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| Connection hangs indefinitely | Use context manager or ensure proper `.close()` | [GitHub #753](https://github.com/weaviate/weaviate-python-client/issues/753) |
|
||||
| Multi-worker conflicts (Gunicorn) | Use lifespan management, not startup hooks | [GitHub #1292](https://github.com/weaviate/weaviate-python-client/issues/1292) |
|
||||
| `TypeError: object NoneType can't be used in 'await'` | Use `use_async_with_*` not `connect_to_*` | [Async API Docs](https://docs.weaviate.io/weaviate/client-libraries/python/async) |
|
||||
| Port conflicts with Weaviate Cloud | Use helper functions, not manual `ConnectionParams` | See "Common Pitfalls #4" above |
|
||||
|
||||
### Getting Help
|
||||
|
||||
**For agents:** When encountering errors:
|
||||
|
||||
1. Check the [Common Pitfalls](#common-pitfalls) section above
|
||||
2. Search [Community Forum](https://forum.weaviate.io/) for similar issues
|
||||
3. Check [GitHub Issues](https://github.com/weaviate/weaviate-python-client/issues) for known bugs
|
||||
4. Refer to [official async documentation](https://docs.weaviate.io/weaviate/client-libraries/python/async)
|
||||
5. Review [Python client best practices](https://docs.weaviate.io/weaviate/client-libraries/python/notes-best-practices)
|
||||
|
||||
## Additional Resources
|
||||
|
||||
### Official Documentation
|
||||
|
||||
- **Primary**: [Weaviate Async API](https://docs.weaviate.io/weaviate/client-libraries/python/async)
|
||||
- **Python Client**: [Main Documentation](https://docs.weaviate.io/weaviate/client-libraries/python)
|
||||
- **API Reference**: [ReadTheDocs](https://weaviate-python-client.readthedocs.io/en/stable/)
|
||||
- **Best Practices**: [Notes and Best Practices](https://docs.weaviate.io/weaviate/client-libraries/python/notes-best-practices)
|
||||
|
||||
### Framework Integration
|
||||
|
||||
- [FastAPI Lifespan Events](https://fastapi.tiangolo.com/advanced/events/)
|
||||
- [Python Async Context Managers](https://docs.python.org/3/reference/datamodel.html#asynchronous-context-managers)
|
||||
|
||||
### Community
|
||||
|
||||
- [Weaviate Community Forum](https://forum.weaviate.io/)
|
||||
- [Python Client GitHub](https://github.com/weaviate/weaviate-python-client)
|
||||
- [Weaviate Blog](https://weaviate.io/blog)
|
||||
@@ -0,0 +1,270 @@
|
||||
# Basic Agent Cookbook
|
||||
|
||||
Build a tool-calling AI agent with structured outputs using DSPy.
|
||||
For RAG tools, memory, and framework integrations, [see here](./agentic_rag.md).
|
||||
|
||||
|
||||
|
||||
Docs to reference if needed:
|
||||
- DSPy signatures: https://dspy.ai/learn/programming/signatures/
|
||||
- DSPy language models: https://dspy.ai/learn/programming/language_models/
|
||||
- LiteLLM providers: https://docs.litellm.ai/docs/
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Use a virtual environment via `venv`
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this install set: `uv add dspy python-dotenv`
|
||||
- Customise this cookbook to the users specification, ask them for details if not given.
|
||||
|
||||
## Env Rules
|
||||
|
||||
Mandatory:
|
||||
- An LLM provider API key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`)
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
Optional:
|
||||
- Matching provider keys listed in `environment_requirements.md`
|
||||
|
||||
If the user explicitly requests a non-Weaviate agent, `WEAVIATE_URL` and `WEAVIATE_API_KEY` can be omitted.
|
||||
|
||||
## Agent Response Signature
|
||||
|
||||
The structured output that defines what the LLM returns when selecting tools.
|
||||
|
||||
```python
|
||||
import dspy
|
||||
from typing import Any, Dict
|
||||
|
||||
class AgentResponse(dspy.Signature):
|
||||
|
||||
# Input Fields
|
||||
history: dspy.History = dspy.InputField()
|
||||
user_prompt: str = dspy.InputField()
|
||||
available_tools: str = dspy.InputField()
|
||||
|
||||
# Output Fields
|
||||
response: str = dspy.OutputField(
|
||||
description="The response to the user's prompt whilst the tool is running. Update the user on the progress of their request (if a tool is picked), or the final response to the user (if no tool is picked)."
|
||||
)
|
||||
tool: str | None = dspy.OutputField(
|
||||
description="The tool that needs to be used. Return None if no tool is needed."
|
||||
)
|
||||
tool_inputs: Dict[str, Any] | None = dspy.OutputField(
|
||||
description=(
|
||||
"The inputs for the tool. Return an empty dictionary (still include the field) if no inputs are needed. "
|
||||
"The key is the name of the input, the value is the value of the input."
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Extend `AgentResponse` as needed: add `confidence: float` for certainty scoring, `requires_clarification: bool` for follow-up questions, or modify `description` strings to shape agent behaviour for a specific domain.
|
||||
|
||||
## Router Agent (Single Step)
|
||||
|
||||
Wraps the agent response into a class that manages conversation history and tool execution.
|
||||
|
||||
```python
|
||||
from typing import Callable, List, Tuple
|
||||
|
||||
class RouterAgent:
|
||||
def __init__(self, model: str, tools: List[Callable] = []):
|
||||
self.tools: list[Callable] = tools
|
||||
self.model = dspy.LM(model)
|
||||
self.agent = dspy.ChainOfThought(AgentResponse)
|
||||
self.conversation_history = dspy.History(messages=[])
|
||||
|
||||
def add_conversation_history(self, message: str, response: dspy.Prediction):
|
||||
self.conversation_history.messages.append({"user_prompt": message, **response})
|
||||
|
||||
def get_tools_and_descriptions(self) -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
f"{tool.__name__}:\nDescription: {tool.__doc__ or ''}\nInputs: { {k: v for k, v in tool.__annotations__.items() if k != 'return'} }"
|
||||
for tool in self.tools
|
||||
]
|
||||
)
|
||||
|
||||
def get_response(self, user_prompt: str) -> Tuple[str, str | None]:
|
||||
result = self.agent(
|
||||
history=self.conversation_history,
|
||||
user_prompt=user_prompt,
|
||||
available_tools=self.get_tools_and_descriptions(),
|
||||
lm=self.model,
|
||||
)
|
||||
self.add_conversation_history(message=user_prompt, response=result)
|
||||
if result.tool and result.tool.lower() not in ["null", "none"]:
|
||||
tool_function = next(
|
||||
(tool for tool in self.tools if tool.__name__ == result.tool), None
|
||||
)
|
||||
if tool_function is None:
|
||||
raise ValueError(f"Tool {result.tool} not found")
|
||||
tool_inputs = {k: v for k, v in result.tool_inputs.items() if k != "return"}
|
||||
tool_result = tool_function(**tool_inputs)
|
||||
else:
|
||||
tool_result = None
|
||||
return result.response, tool_result
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```python
|
||||
router = RouterAgent(
|
||||
model="<model_name>", # e.g. claude-sonnet-4-5, gpt-5.2, gemini-2.5-pro
|
||||
tools=[your_tool_function]
|
||||
)
|
||||
response, tool_result = router.get_response("user query here")
|
||||
```
|
||||
|
||||
## Tool Design
|
||||
|
||||
Tools are Python functions. The agent reads `__name__`, `__doc__`, and `__annotations__` to decide when to use them.
|
||||
|
||||
```python
|
||||
def your_tool(param1: str, param2: int) -> str:
|
||||
"""Clear description of what this tool does and when to use it."""
|
||||
# tool logic here
|
||||
return "result as string"
|
||||
```
|
||||
|
||||
Key rules:
|
||||
- Docstrings directly influence when the agent selects the tool. Be specific: "Get current weather conditions for a city" is better than "Get weather".
|
||||
- Type hints guide what inputs the agent provides. Complex types like `filters: List[Dict]` may need additional description in the docstring.
|
||||
- Return strings or string-serializable data.
|
||||
|
||||
|
||||
## Sequential Multi-Step Agent
|
||||
|
||||
For tasks requiring multiple tool calls in succession, add a followup signature and loop.
|
||||
|
||||
Followup signature (receives `tool_output` from the previous step):
|
||||
|
||||
```python
|
||||
class AgentFollowup(dspy.Signature):
|
||||
|
||||
# Input Fields
|
||||
history: dspy.History = dspy.InputField()
|
||||
user_prompt: str = dspy.InputField()
|
||||
tool_output: str = dspy.InputField(description="The output of the previous tool.")
|
||||
available_tools: str = dspy.InputField(
|
||||
description="The available tools and their descriptions."
|
||||
)
|
||||
|
||||
# Output Fields
|
||||
response: str = dspy.OutputField(
|
||||
description="The response to the user's prompt whilst the tool is running. Update the user on the progress of their request (if a tool is picked), or the final response to the user (if no tool is picked)."
|
||||
)
|
||||
tool: str | None = dspy.OutputField(
|
||||
description="The tool that needs to be used. Return None if no tool is needed."
|
||||
)
|
||||
tool_inputs: Dict[str, Any] | None = dspy.OutputField(
|
||||
description="The inputs for the tool. Return an empty dictionary (still include the field) if no inputs are needed. The key is the name of the input, the value is the value of the input.",
|
||||
)
|
||||
```
|
||||
|
||||
Modify `RouterAgent` to loop until the agent stops requesting tools:
|
||||
|
||||
```python
|
||||
class RouterAgent:
|
||||
def __init__(self, model: str, tools: List[Callable] = []):
|
||||
self.tools: list[Callable] = tools
|
||||
self.model = dspy.LM(model)
|
||||
self.agent = dspy.ChainOfThought(AgentResponse)
|
||||
self.followup_agent = dspy.ChainOfThought(AgentFollowup)
|
||||
self.conversation_history = dspy.History(messages=[])
|
||||
|
||||
def add_conversation_history(self, message: str, response: dspy.Prediction):
|
||||
self.conversation_history.messages.append({"user_prompt": message, **response})
|
||||
|
||||
def get_tools_and_descriptions(self) -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
f"{tool.__name__}:\nDescription: {tool.__doc__ or ''}\nInputs: { {k: v for k, v in tool.__annotations__.items() if k != 'return'} }"
|
||||
for tool in self.tools
|
||||
]
|
||||
)
|
||||
|
||||
def get_response(self, user_prompt: str) -> str:
|
||||
result = self.agent(
|
||||
history=self.conversation_history,
|
||||
user_prompt=user_prompt,
|
||||
available_tools=self.get_tools_and_descriptions(),
|
||||
lm=self.model,
|
||||
)
|
||||
self.add_conversation_history(message=user_prompt, response=result)
|
||||
|
||||
max_iter = 10
|
||||
iter = 0
|
||||
|
||||
while result.tool is not None and result.tool.lower() not in ["null", "none"]:
|
||||
iter += 1
|
||||
if iter > max_iter:
|
||||
break
|
||||
|
||||
tool_function = next(
|
||||
(tool for tool in self.tools if tool.__name__ == result.tool), None
|
||||
)
|
||||
if tool_function is None:
|
||||
raise ValueError(f"Tool {result.tool} not found")
|
||||
|
||||
tool_inputs = {k: v for k, v in result.tool_inputs.items() if k != "return"}
|
||||
tool_result = tool_function(**tool_inputs)
|
||||
|
||||
result = self.followup_agent(
|
||||
history=self.conversation_history,
|
||||
tool_output=tool_result,
|
||||
available_tools=self.get_tools_and_descriptions(),
|
||||
lm=self.model,
|
||||
)
|
||||
self.add_conversation_history(message=tool_result, response=result)
|
||||
|
||||
return result.response
|
||||
```
|
||||
|
||||
`max_iter` controls how many tool calls can occur before forced termination. Increase for complex multi-step tasks, decrease to limit costs and runaway loops.
|
||||
|
||||
## User-specific Customisations
|
||||
|
||||
If not specified ask the user about these points before implementing their respective strategies:
|
||||
|
||||
**LLM Framework**
|
||||
|
||||
You can use DSPy (works with all LiteLLM providers) or LiteLLM itself.
|
||||
|
||||
- DSPy: https://dspy.ai/learn/programming/language_models/
|
||||
- LiteLLM: https://docs.litellm.ai/docs/
|
||||
|
||||
Alternatively, users can use a single model provider. What model provider will they use?
|
||||
|
||||
- OpenAI (https://platform.openai.com/docs/libraries)
|
||||
- Anthropic (https://platform.claude.com/docs/)
|
||||
- Google GenAI (https://ai.google.dev/gemini-api/docs/libraries)
|
||||
- Other (such as locally hosted models), use best judgement
|
||||
|
||||
These may require additional installs.
|
||||
|
||||
**Model Selection**
|
||||
|
||||
What model(s) will the user use? Consider a mixed approach: capable model for main agent routing, cheaper model for auxiliary tasks like memory creation.
|
||||
|
||||
**Tools**
|
||||
|
||||
What tools does the user need? List their functions, inputs, and expected outputs. The agent is only as capable as its tools.
|
||||
|
||||
**Single-step vs Multi-step**
|
||||
|
||||
Does the user need a single tool call per query, or should the agent chain multiple tools in sequence? Only use multi-step if the use case requires it.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- DSPy signature warnings about missing fields: ensure all input fields are passed or mark optional fields appropriately.
|
||||
- Tool not found errors: ensure tool function names match exactly what the agent outputs.
|
||||
- Agent loops indefinitely: lower `max_iter` or add more explicit termination conditions.
|
||||
- For any other issues, refer to the official library/package documentation and use web search extensively for troubleshooting.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Create test scripts to check each function works independently with test data. Tear down tests after completion, or create a proper test suite with pytest (requires install)
|
||||
- User has completed specification of the app.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Basic RAG Cookbook
|
||||
|
||||
Build basic RAG functionality with Weaviate.
|
||||
For advanced strategies, [see here](./advanced_rag.md).
|
||||
|
||||
|
||||
Docs to reference if needed:
|
||||
- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
|
||||
- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
|
||||
- Vector search: https://docs.weaviate.io/weaviate/search/similarity
|
||||
- Keyword search: https://docs.weaviate.io/weaviate/search/bm25
|
||||
- Hybrid search: https://docs.weaviate.io/weaviate/search/hybrid
|
||||
- Image search: https://docs.weaviate.io/weaviate/search/image
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Use a virtual environment via `venv`
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this install set: `uv add weaviate-client python-dotenv dspy`
|
||||
- Customise this cookbook to the users specification, ask them for details if not given.
|
||||
|
||||
Assume the user has data already to be used, do not create data unless asked to.
|
||||
|
||||
## Env Rules
|
||||
|
||||
Mandatory:
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
External provider keys:
|
||||
- Fill only keys actually used by the target Weaviate collection setup.
|
||||
|
||||
|
||||
## Weaviate Client
|
||||
|
||||
```python
|
||||
import os
|
||||
from weaviate import connect_to_weaviate_cloud
|
||||
|
||||
client = connect_to_weaviate_cloud(
|
||||
cluster_url=os.getenv("WEAVIATE_URL", ""),
|
||||
auth_credentials=os.getenv("WEAVIATE_API_KEY", ""),
|
||||
headers={
|
||||
"X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
If the user's collections require vectorizer provider keys, set the matching keys listed in `environment_requirements.md`.
|
||||
|
||||
Clients must be closed after completion. Wrap in `try/finally` blocks with `client.close()` (and `client.connect()` to reconnect if needed).
|
||||
|
||||
|
||||
## Multi-tenancy
|
||||
|
||||
Multi-tenancy should be checked via
|
||||
|
||||
```python
|
||||
config = await collection.config.get()
|
||||
config.multi_tenancy_config.enabled # bool
|
||||
```
|
||||
|
||||
e.g.
|
||||
|
||||
```python
|
||||
|
||||
base_collection = client.collections.use(collection_name)
|
||||
|
||||
config = collection.config.get()
|
||||
if config.multi_tenancy_config.enabled:
|
||||
collection = base_collection.with_tenant("<tenant_name>")
|
||||
else:
|
||||
collection = base_collection
|
||||
```
|
||||
|
||||
Tenant names can be obtained via
|
||||
```python
|
||||
all_tenants = list(collection.tenants.get().keys())
|
||||
```
|
||||
|
||||
## Basic Retrieval
|
||||
|
||||
Use collections via
|
||||
|
||||
```python
|
||||
collection = client.collections.use("<collection_name>")
|
||||
```
|
||||
|
||||
Weaviate can use vector, keyword or hybrid search.
|
||||
|
||||
```python
|
||||
collection.query.near_text # semantic (text)
|
||||
collection.query.bm25 # keyword
|
||||
collection.query.hybrid # blend of keyword and semantic
|
||||
```
|
||||
|
||||
It can also do image search
|
||||
|
||||
```python
|
||||
collection.query.near_image(
|
||||
near_image = ... # base 64 representation of image or Path object to image
|
||||
)
|
||||
```
|
||||
|
||||
## Key Code Blocks
|
||||
|
||||
RAG should have 4 pieces of core functionality:
|
||||
|
||||
1. Pre-retrieval
|
||||
2. Retrieval
|
||||
3. Post-retrieval
|
||||
4. Generation
|
||||
|
||||
These should all be separate functions and combined into a single function, leaving scope for later editing or for the user themselves to modify it, to keep it understandable.
|
||||
|
||||
## Pre-retrieval
|
||||
|
||||
Transform the user question into a vector-database style (list of) query(ies). Basic RAG will provide no extra query transformations.
|
||||
|
||||
```python
|
||||
def query_transformation(query: str) -> list[str]:
|
||||
return [query]
|
||||
```
|
||||
|
||||
## Retrieval
|
||||
|
||||
```python
|
||||
def retrieve(
|
||||
query: str,
|
||||
limit: int = 10, # optional
|
||||
filters = [] # optional
|
||||
# additional arguments if required can go here and passed down to the search strategy
|
||||
) -> list[dict]:
|
||||
|
||||
# import client logic here
|
||||
|
||||
collection = client.collections.use("<collection_name>")
|
||||
|
||||
response = collection.query.near_text( # or hybrid, bm25, near_image
|
||||
query=query,
|
||||
limit=limit,
|
||||
filters=filters if filters else None
|
||||
)
|
||||
|
||||
return [
|
||||
{
|
||||
**obj.properties,
|
||||
"uuid": obj.uuid
|
||||
}
|
||||
for obj in response.objects
|
||||
]
|
||||
```
|
||||
|
||||
## Post-Retrieval
|
||||
|
||||
Modify the output of `retrieve`. Basic RAG will provide no extra post-processing. But you can consider adding uniqueness checks, formatting to remove properties, or more.
|
||||
|
||||
```python
|
||||
def process_retrieval_results(objects: list[dict]) -> list[dict]:
|
||||
return objects
|
||||
```
|
||||
|
||||
|
||||
## Generation
|
||||
|
||||
This step depends on your LLM framework, [see below](#user-specific-customisations). Using DSPy:
|
||||
|
||||
```python
|
||||
import dspy
|
||||
def generate(query: str, context: list[dict]) -> str:
|
||||
lm = dspy.LM("<model_name>") # e.g. gpt-5.2, gpt-5-mini, claude-sonnet-4-5, etc.
|
||||
answer = dspy.Predict("context, query -> answer") # inputs: context, query. outputs: answer
|
||||
pred = answer(context=context, query=query, lm=lm)
|
||||
return pred.answer # answer is then an attribute of pred
|
||||
```
|
||||
|
||||
## User-specific Customisations
|
||||
|
||||
If not specified ask the user about these points before implementing their respective strategies:
|
||||
|
||||
**LLM Framework**
|
||||
|
||||
You can use DSPy (works with all LiteLLM providers) or LiteLLM itself.
|
||||
|
||||
- DSPy: https://dspy.ai/learn/programming/language_models/
|
||||
- LiteLLM: https://docs.litellm.ai/docs/
|
||||
|
||||
Alternatively, users can use a single model provider. What model provider will they use?
|
||||
|
||||
- OpenAI (https://platform.openai.com/docs/libraries)
|
||||
- Anthropic (https://platform.claude.com/docs/)
|
||||
- Google GenAI (https://ai.google.dev/gemini-api/docs/libraries)
|
||||
- Other (such as locally hosted models), use best judgement
|
||||
|
||||
These may require additional installs.
|
||||
|
||||
**Collections**
|
||||
|
||||
Do collections already exist and what are they called? Does the user want to query multiple collections or just a single one? Does it need to be customisable?
|
||||
|
||||
What format is the data, images or text or something else? What vectoriser is the collection set up as? What API keys are needed?
|
||||
|
||||
**Search strategy**
|
||||
|
||||
Does the user want semantic, keyword or hybrid search?
|
||||
|
||||
Hybrid search has an `alpha` parameter, controlling tradeoff between keyword and semantic weights. `alpha=1` is pure semantic, `alpha=0` is pure keyword.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
|
||||
- For any other issues, refer to the official library/package documentation and use web search extensively for troubleshooting.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Create test scripts to check each function works independently with test data. Tear down tests after completion, or create a proper test suite with pytest (requires install)
|
||||
- User has completed specification of the app.
|
||||
@@ -0,0 +1,336 @@
|
||||
# Build Data Explorer App
|
||||
|
||||
## Overview
|
||||
|
||||
Build a full-stack Data Explorer App for Weaviate Collections with FastAPI.
|
||||
|
||||
Read first:
|
||||
- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
|
||||
- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
|
||||
|
||||
## Instructions
|
||||
|
||||
### Core Rules
|
||||
|
||||
- Use a virtual environment via `venv`
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this backend install set:
|
||||
- `uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv`
|
||||
- Depending on user request: consider combining this app with the [Query Agent Chatbot](./query_agent_chatbot.md).
|
||||
- If the user explicitly only wants a data viewer/explorer, create this app independently
|
||||
- If the user wants a fully featured chat and data explorer, combine the apps
|
||||
- If no explicit instructions are given, ask the user their preference before continuing
|
||||
- See the [Next Steps](#next-steps) section for more details
|
||||
|
||||
### Fast Setup Commands
|
||||
|
||||
Project bootstrap:
|
||||
|
||||
```bash
|
||||
uv init data_explorer
|
||||
cd data_explorer
|
||||
uv venv
|
||||
uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv
|
||||
```
|
||||
|
||||
### Workflow Contract
|
||||
|
||||
1. Build backend and frontend in one pass.
|
||||
2. Create `.env` from the canonical template in `environment_requirements.md`, then add app-specific fields (for example, `CORS_ORIGINS`).
|
||||
3. Before asking user to fill env, do non-secret local sanity checks that do not require real credentials (imports/compile/startup-shape checks).
|
||||
4. Ask user to fill real env values:
|
||||
- Mandatory: `WEAVIATE_URL`, `WEAVIATE_API_KEY`
|
||||
- Optional: only provider keys required by their collection setup
|
||||
5. After the user confirms, verify backend starts without errors and provide exact commands to run in the terminal.
|
||||
|
||||
Do not ask avoidable questions that you can resolve from context.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
Use a modular layout like:
|
||||
|
||||
```text
|
||||
data_explorer/
|
||||
backend/
|
||||
app/
|
||||
main.py
|
||||
config.py
|
||||
lifespan.py
|
||||
dependencies.py
|
||||
routers/
|
||||
services/
|
||||
models/
|
||||
.env # local file, never committed
|
||||
```
|
||||
|
||||
Keep these boundaries:
|
||||
|
||||
- routers: HTTP only
|
||||
- services: business/query-agent logic
|
||||
- models: request/response schemas
|
||||
- config/lifespan: wiring and startup/shutdown
|
||||
|
||||
### Backend Requirements
|
||||
|
||||
- FastAPI async app with lifespan.
|
||||
- Async Weaviate client initialized in lifespan and closed on shutdown.
|
||||
- Ensure no async blocking operations.
|
||||
- Not a full CRUD implementation - this is only for viewing data in a Weaviate collection.
|
||||
- Endpoints for:
|
||||
- `GET /health`
|
||||
- `GET /env_check`: returns what API keys are missing (if any) for verification on app start
|
||||
- `GET /collections`: return available collections
|
||||
- `GET /data/{collection_name}?xx=xx&yy=yy`: return data with optional arguments (more later), and pagination
|
||||
- Pydantic settings should read from process environment; local `.env` loading is optional for local development.
|
||||
- Conversation history mapping to Weaviate chat message format.
|
||||
|
||||
### Env Rules
|
||||
|
||||
Mandatory:
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
External provider keys:
|
||||
- Include every provider key needed by the target collections.
|
||||
- Leave unused provider keys empty/commented.
|
||||
|
||||
CORS:
|
||||
|
||||
- Default `CORS_ORIGINS` should include:
|
||||
- `http://localhost:3000`
|
||||
- `http://127.0.0.1:3000`
|
||||
- `http://localhost:5173`
|
||||
- `http://127.0.0.1:5173`
|
||||
|
||||
### FastAPI standards
|
||||
|
||||
1. Do not use hardcoded status values, use `status` from FastAPI, for example:
|
||||
|
||||
```python
|
||||
from fastapi import status
|
||||
status.HTTP_200_OK # code 200
|
||||
status.HTTP_404_NOT_FOUND # code 404
|
||||
# and more
|
||||
```
|
||||
|
||||
2. Use a Pydantic `BaseModel` for the `request` and `response_model` in all endpoints that require it. Ensure schema validation to mitigate user-error on the API.
|
||||
|
||||
3. Use path parameters and query parameters for GET endpoints instead of payloads, for example:
|
||||
|
||||
```python
|
||||
@app.get("/items/{item_id}")
|
||||
async def read_item(item_id: str):
|
||||
return {"item_id": item_id}
|
||||
```
|
||||
|
||||
```python
|
||||
@app.get("/items/")
|
||||
async def read_item(skip: int = 0, limit: int = 10):
|
||||
return fake_items_db[skip : skip + limit]
|
||||
```
|
||||
|
||||
4. Implement best practices for error-handling, do early returns and provide the correct status codes when necessary.
|
||||
|
||||
5. Use proper logging for API usage, not simple print statements.
|
||||
|
||||
### FastAPI endpoints
|
||||
|
||||
Basic structure of endpoints. Customise according to user preference or suitability. Do not follow exactly, this is a guideline only.
|
||||
|
||||
Ensure you also set up standard FastAPI procedures, such as global error handling, logging, dependencies. Set up an async client manager that connects on startup (via lifespan) and closes gracefully on app exit, use a dependency injection to add the client to the relevant endpoints.
|
||||
|
||||
#### GET /health
|
||||
|
||||
This is a standard health check. For example:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
|
||||
@app.get("/health", tags=["health"], response_model=HealthResponse)
|
||||
async def health_check() -> HealthResponse:
|
||||
logger.info("Health check requested")
|
||||
return HealthResponse(status="healthy")
|
||||
```
|
||||
|
||||
#### GET /env_check
|
||||
|
||||
Check what environment variables the backend has access to, used to verify the user's Weaviate configuration is correct. For example:
|
||||
|
||||
```python
|
||||
import os
|
||||
from pydantic import BaseModel
|
||||
|
||||
class EnvCheckResponse(BaseModel):
|
||||
weaviate_url: bool
|
||||
weaviate_api_key: bool
|
||||
|
||||
@app.get("/env_check", tags=["health"])
|
||||
async def env_check() -> EnvCheckResponse:
|
||||
logger.info("Environment check requested")
|
||||
return EnvCheckResponse(
|
||||
weaviate_url = os.getenv("WEAVIATE_URL") is not None,
|
||||
weaviate_api_key = os.getenv("WEAVIATE_API_KEY") is not None,
|
||||
)
|
||||
```
|
||||
|
||||
### GET /collections
|
||||
|
||||
Check what collections are available. For example:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from weaviate.client import WeaviateAsyncClient
|
||||
|
||||
class CollectionsResponse(BaseModel):
|
||||
collections: list[str]
|
||||
|
||||
@app.get("/collections", tags=["collections"])
|
||||
async def collections() -> CollectionsResponse:
|
||||
|
||||
# include client management to import async client here
|
||||
|
||||
logger.info("Collections requested")
|
||||
collections = await client.collections.list_all()
|
||||
return CollectionsResponse(
|
||||
collections = list(collections.keys())
|
||||
)
|
||||
```
|
||||
|
||||
Tip: consider expanding this endpoint to include collection descriptions and configs. `await client.collections.list_all()` returns `dict[str, _CollectionConfigSimple]` where `_CollectionConfigSimple` contains attributes:
|
||||
|
||||
- `description`: `str`
|
||||
- `properties`: `list[Property]` where `Property` has `.name`, `.description` and `.data_type` (accessed via `.data_type[:]` to get name of data type as string)
|
||||
- `vector_config`: `dict[str, _NamedVectorConfig]` where `_NamedVectorConfig` has attribute `.vectorizer.vectorizer` (not a typo) which can be accessed via `.vectorizer.vectorizer[:]` to get the name of the vectoriser as a string.
|
||||
|
||||
Multi-tenancy should be checked via
|
||||
|
||||
```python
|
||||
config = await collection.config.get()
|
||||
config.multi_tenancy_config.enabled # bool
|
||||
```
|
||||
|
||||
This is not available in the `_CollectionConfigSimple`, it must be fetched from `collection.config.get()`.
|
||||
|
||||
#### GET /data/{collection_name}
|
||||
|
||||
Retrieve data from a collection, using pagination, sorting and filters.
|
||||
|
||||
```python
|
||||
from weaviate.collections import CollectionAsync
|
||||
from fastapi import Query
|
||||
from pydantic import BaseModel
|
||||
from typing import Any
|
||||
|
||||
async def get_collection_data_types(collection: CollectionAsync) -> dict[str, str]:
|
||||
config = await collection.config.get()
|
||||
properties = config.properties
|
||||
return {prop.name: prop.data_type[:] for prop in properties}
|
||||
|
||||
class GetDataResponse(BaseModel):
|
||||
data_types: dict[str, str]
|
||||
items: list[dict[str, Any]]
|
||||
|
||||
@router.post("/data/{collection_name}")
|
||||
async def get_data(
|
||||
collection_name: str,
|
||||
page_size: int = Query(default=10, ge=1, le=100),
|
||||
page_number: int = Query(default=1, ge=1),
|
||||
query: str = Query(default=""),
|
||||
sort_on: str = Query(default=None),
|
||||
ascending: bool = Query(default=True),
|
||||
) -> GetDataResponse:
|
||||
|
||||
# include client management to import async client here
|
||||
|
||||
collection = await client.collections.use(collection_name)
|
||||
data_types = await async_get_collection_data_types(collection)
|
||||
|
||||
if query != "":
|
||||
response = await collection.query.bm25(
|
||||
query=query,
|
||||
limit=page_size,
|
||||
offset=page_size * (page_number - 1),
|
||||
)
|
||||
elif sort_on is not None:
|
||||
response = await collection.query.fetch_objects(
|
||||
sort=Sort.by_property(name=sort_on, ascending=ascending),
|
||||
limit=page_size,
|
||||
offset=page_size * (page_number - 1),
|
||||
)
|
||||
else:
|
||||
response = await collection.query.fetch_objects(
|
||||
limit=page_size,
|
||||
offset=page_size * (page_number - 1),
|
||||
)
|
||||
|
||||
return GetDataResponse(data_types = data_types, items = [obj.properties for obj in response.objects])
|
||||
```
|
||||
|
||||
Tip: some collections can have multi-tenancy.
|
||||
Consider adding the tenant as an optional query parameter to `get_data`, e.g.
|
||||
|
||||
```python
|
||||
async def get_data(
|
||||
... # existing args
|
||||
tenant: str | None = Query(default=None)
|
||||
):
|
||||
base_collection = await client.collections.use(collection_name)
|
||||
data_types = await async_get_collection_data_types(collection)
|
||||
|
||||
config = await collection.config.get()
|
||||
if config.multi_tenancy_config.enabled and tenant and tenant.strip():
|
||||
collection = base_collection.with_tenant(tenant)
|
||||
else:
|
||||
collection = base_collection
|
||||
|
||||
# ...existing code
|
||||
```
|
||||
|
||||
### Post-Env Hand-Holding (Required)
|
||||
|
||||
After user says required env values are set, provide the terminal commands to run the backend:
|
||||
|
||||
```bash
|
||||
cd data_explorer/backend
|
||||
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
- Ask user to start terminal.
|
||||
- Run smoke tests yourself against running services.
|
||||
- Report pass/fail in plain language and fix blockers.
|
||||
|
||||
Do not offload detailed testing steps to the user unless they explicitly ask.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
|
||||
- For any other issues, refer to the official library/package documentation using web search.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Backend healthy.
|
||||
- All endpoints work.
|
||||
- User can run server in terminal with provided commands.
|
||||
|
||||
## Next Steps
|
||||
|
||||
This application is currently a data explorer backend. You may optionally offer to integrate it with the [Query Agent Chatbot](./query_agent_chatbot.md) based on user preference.
|
||||
|
||||
If the user chooses to combine these two applications, implement the integration as follows:
|
||||
|
||||
- Create or use a directory `/routes` which separate functions for query agent chat and data exploration. Import the routers in the `main.py` file
|
||||
- If a frontend is requested, the frontend should have multiple pages/tabs depending on design choices so that data exploration and chat is separated
|
||||
- Consider crossovers between functionalities, e.g. a chat button from the data viewer/collection viewer which takes the user to chat with that collection selected.
|
||||
- Run quick tests to ensure the integration is seamless and the user can use both the chatbot and data explorer without any issues.
|
||||
|
||||
### Frontend
|
||||
|
||||
When the user explicitly asks for a frontend, use this reference as guideline:
|
||||
|
||||
- [Frontend Interface](frontend_interface.md): Build a Next.js frontend to interact with the Weaviate backend.
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
# Environment Requirements for Weaviate
|
||||
|
||||
Use this reference when building apps that connect to Weaviate and require external inference provider keys.
|
||||
|
||||
## Required Weaviate Auth
|
||||
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
## External Provider Env Vars and Headers
|
||||
|
||||
| Provider | Environment Variable(s) | Header(s) sent to Weaviate |
|
||||
|----------|--------------------------|-----------------------------|
|
||||
| Anthropic | `ANTHROPIC_API_KEY` | `X-Anthropic-Api-Key` |
|
||||
| Anyscale | `ANYSCALE_API_KEY` | `X-Anyscale-Api-Key` |
|
||||
| AWS | `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` | `X-Aws-Access-Key`, `X-Aws-Secret-Key` |
|
||||
| Cohere | `COHERE_API_KEY` | `X-Cohere-Api-Key` |
|
||||
| Databricks | `DATABRICKS_TOKEN` | `X-Databricks-Token` |
|
||||
| Friendli | `FRIENDLI_TOKEN` | `X-Friendli-Api-Key` |
|
||||
| Google Vertex AI | `VERTEX_API_KEY` | `X-Goog-Vertex-Api-Key` |
|
||||
| Google AI Studio | `STUDIO_API_KEY` | `X-Goog-Studio-Api-Key` |
|
||||
| HuggingFace | `HUGGINGFACE_API_KEY` | `X-HuggingFace-Api-Key` |
|
||||
| Jina AI | `JINAAI_API_KEY` | `X-JinaAI-Api-Key` |
|
||||
| Mistral | `MISTRAL_API_KEY` | `X-Mistral-Api-Key` |
|
||||
| NVIDIA | `NVIDIA_API_KEY` | `X-Nvidia-Api-Key` |
|
||||
| OpenAI | `OPENAI_API_KEY` | `X-OpenAI-Api-Key` |
|
||||
| Azure OpenAI | `AZURE_API_KEY` | `X-Azure-Api-Key` |
|
||||
| Voyage AI | `VOYAGE_API_KEY` | `X-Voyage-Api-Key` |
|
||||
| xAI | `XAI_API_KEY` | `X-Xai-Api-Key` |
|
||||
|
||||
## Usage Notes
|
||||
|
||||
- Set only the provider keys your collection configuration actually uses.
|
||||
- If multiple providers are configured, include all corresponding headers.
|
||||
|
||||
## Canonical `.env` Template
|
||||
|
||||
Use this template in all cookbook apps. Then ask the user to fill only the values their app actually needs.
|
||||
|
||||
`WEAVIATE_URL` and `WEAVIATE_API_KEY` are mandatory for Weaviate-connected apps.
|
||||
|
||||
```dotenv
|
||||
# Required for Weaviate cookbook apps (must be filled by user)
|
||||
WEAVIATE_URL=
|
||||
WEAVIATE_API_KEY=
|
||||
|
||||
# Common app-level settings (uncomment when needed by the selected cookbook)
|
||||
# COLLECTIONS=
|
||||
# CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000,http://localhost:5173,http://127.0.0.1:5173
|
||||
|
||||
# External provider keys (uncomment only what the target collection uses)
|
||||
# ANTHROPIC_API_KEY=
|
||||
# ANYSCALE_API_KEY=
|
||||
# AWS_ACCESS_KEY=
|
||||
# AWS_SECRET_KEY=
|
||||
# AZURE_API_KEY=
|
||||
# COHERE_API_KEY=
|
||||
# DATABRICKS_TOKEN=
|
||||
# FRIENDLI_TOKEN=
|
||||
# HUGGINGFACE_API_KEY=
|
||||
# JINAAI_API_KEY=
|
||||
# MISTRAL_API_KEY=
|
||||
# NVIDIA_API_KEY=
|
||||
# OPENAI_API_KEY=
|
||||
# STUDIO_API_KEY=
|
||||
# VERTEX_API_KEY=
|
||||
# VOYAGE_API_KEY=
|
||||
# XAI_API_KEY=
|
||||
```
|
||||
|
||||
## User Fill Guidance (Required)
|
||||
|
||||
1. Create a local `.env` file from this template.
|
||||
2. Always ask the user to fill:
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
3. Ask them to uncomment and fill only the provider keys their Weaviate collections require.
|
||||
4. Keep `.env` local only and gitignored.
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
# Frontend Interface (Next.js + Weaviate Backend)
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Item | Value |
|
||||
| ------------ | ------------------------------------------------------------------------------------------- |
|
||||
| **Stack** | Next.js (App Router), Tailwind v4, shadcn/ui, Framer Motion, react-icons, ai-sdk |
|
||||
| **Node** | v25.3.0+ |
|
||||
| **Backend** | `NEXT_PUBLIC_BACKEND_HOST` (default: `localhost:8000`) |
|
||||
| **App type** | Single-page app; main view updates in place, no full-page navigations |
|
||||
| **Layout** | shadcn Sidebar (left) + main content area; sidebar buttons switch the main view per feature |
|
||||
|
||||
---
|
||||
|
||||
## Setup (run in order)
|
||||
|
||||
### 1. Next.js
|
||||
|
||||
- **Command:** `npx create-next-app@latest . --yes` (run from repo root; may need `required_permissions: ["all"]` in sandbox)
|
||||
- **Result:** TypeScript, ESLint, Tailwind v4, App Router, Turbopack, `@/*` → `./*`, no `src/`. App in `app/`, static in `public/`.
|
||||
- **Scripts:** `dev` | `build` | `start` | `lint`. Dev server: http://localhost:3000.
|
||||
- **Routes:** `app/layout.tsx`, `app/page.tsx`. Imports: `@/` = project root.
|
||||
- **Ref:** [Next.js App Router Installation](https://nextjs.org/docs/app/getting-started/installation) — verify against current docs.
|
||||
|
||||
### 2. shadcn/ui
|
||||
|
||||
- **Requires:** Next.js + Tailwind v4 + App Router + `@/*`, no `src/`.
|
||||
- **Init:** `npx shadcn@latest init -t next -y -b zinc --no-src-dir`
|
||||
- **Add components:** `npx shadcn@latest add button -y` (e.g. `card`, `dialog`, `input`; `-o` overwrites).
|
||||
- **Output:** `components.json`, `lib/utils.ts` (cn), `app/globals.css` (tw-animate, shadcn/tailwind.css, CSS vars). UI in `components/ui/<name>.tsx`. Import: `import { Button } from "@/components/ui/button"`.
|
||||
- **Ref:** [shadcn Next.js](https://ui.shadcn.com/docs/installation/next) | [CLI](https://ui.shadcn.com/docs/cli).
|
||||
|
||||
### 3. Framer Motion
|
||||
|
||||
```bash
|
||||
npm i framer-motion
|
||||
```
|
||||
|
||||
- **Ref:** [Framer Motion](https://motion.dev/)
|
||||
|
||||
### 4. AI SDK (optional)
|
||||
|
||||
Note: Install only when create a conversational user interface for your chatbot application. It enables the streaming of chat messagesyou need to stream responses from the backend using useChat().
|
||||
|
||||
- **When:** Add this step only if the app needs a chat UI (e.g. query-agent or chatbot flows).
|
||||
- **Stack:** Use the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction) (`ai` + `@ai-sdk/*`). Use `useChat` and SDK UI primitives for the chat view.
|
||||
- **Ref:** [AI SDK – useChat](https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat) | [Next.js App Router setup](https://ai-sdk.dev/docs/getting-started/nextjs-app-router) — follow current docs for install and wiring.
|
||||
|
||||
```bash
|
||||
npm i ai @ai-sdk/react zod
|
||||
```
|
||||
|
||||
### 5. Environment
|
||||
|
||||
**Required:**
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_BACKEND_HOST="localhost:8000"
|
||||
```
|
||||
|
||||
Use the actual backend host when not local.
|
||||
|
||||
---
|
||||
|
||||
## Rules (must follow)
|
||||
|
||||
### Stack and structure
|
||||
|
||||
- **UI:** Use **shadcn components only** for layout and interactive elements (buttons, cards, inputs, dialogs, etc.). Do not add another UI library.
|
||||
- **Architecture:** **SPA** — one main page, update main view in place. Avoid full-page navigations unless necessary.
|
||||
- **Icons:** Use **react-icons** only; prefer one set (e.g. `react-icons/fa` or `react-icons/hi`) for consistency.
|
||||
- **Animation:** Use **Framer Motion** only. Do not add another animation library.
|
||||
|
||||
### Visual style
|
||||
|
||||
- **Goal:** Minimal, sleek, clean. No clutter, heavy borders, or noisy backgrounds.
|
||||
- **Aesthetic:** “Liquid glass” — frosted, translucent; soft blur; light borders and shadows; depth without heaviness. Use `backdrop-blur`, semi-transparent fills, subtle gradients where they support this.
|
||||
|
||||
### Motion
|
||||
|
||||
- **Style:** Subtle, springy, purposeful (fade in, hover, enter/exit). Prefer spring physics over linear/ease-out.
|
||||
|
||||
### Layout
|
||||
|
||||
1. **Left:** shadcn **Sidebar** component.
|
||||
2. **Right:** Main content area.
|
||||
3. **Navigation:** One sidebar button per backend feature (e.g. data explorer, chat). Click switches the main view only.
|
||||
|
||||
### Responsiveness
|
||||
|
||||
- Layout and components must work on small and large screens.
|
||||
|
||||
---
|
||||
|
||||
## Docs (verify against current versions)
|
||||
|
||||
- [FastAPI](https://fastapi.tiangolo.com/) | [GitHub](https://github.com/fastapi/fastapi)
|
||||
- [Node.js](https://nodejs.org/en)
|
||||
- [Next.js](https://nextjs.org/docs)
|
||||
- [Tailwind (Next.js)](https://tailwindcss.com/docs/installation/framework-guides/nextjs)
|
||||
- [shadcn components](https://ui.shadcn.com/docs/components)
|
||||
- [react-icons](https://react-icons.github.io/react-icons)
|
||||
- [Framer Motion](https://motion.dev/)
|
||||
- [AI SDK](https://ai-sdk.dev/docs/introduction)
|
||||
+635
@@ -0,0 +1,635 @@
|
||||
# Multi-vector RAG: Building Multimodal Document Search Systems With Weaviate
|
||||
|
||||
## Overview
|
||||
|
||||
This cookbook provides instructions for implementing a Multimodal Retrieval-Augmented Generation (RAG) system over PDF document collections using Weaviate Embeddings multimodal model for embeddings and Ollama with a Vision Language Model (VLM) for generation.
|
||||
|
||||
Weaviate Embeddings handles all embedding generation server-side — no local GPU or model downloads required. Simply upload document images as base64 blobs and Weaviate generates multi-vector embeddings automatically.
|
||||
|
||||
### Architecture
|
||||
|
||||
A multimodal RAG system consists of two main pipelines:
|
||||
|
||||
**Ingestion Pipeline:**
|
||||
- Documents (PDFs, images) are converted to page images
|
||||
- Images are uploaded as base64 blobs to Weaviate
|
||||
- Weaviate Embeddings generates multi-vector embeddings server-side using `ModernVBERT/colmodernvbert`
|
||||
- Embeddings are stored in the vector index automatically
|
||||
|
||||
**Query Pipeline:**
|
||||
- Text queries are sent to Weaviate, which embeds them server-side
|
||||
- Relevant documents are retrieved using similarity search (MaxSim)
|
||||
- Retrieved document images are passed to a Vision Language Model (VLM) running on Ollama with the query
|
||||
- The VLM generates a natural language response based on visual and textual context
|
||||
|
||||
|
||||
|
||||
**Requirements:**
|
||||
- Weaviate Cloud instance (Weaviate Embeddings is cloud-only)
|
||||
- Python 3.11 or higher
|
||||
- `uv` package manager ([installation guide](https://docs.astral.sh/uv/getting-started/installation/))
|
||||
- [Ollama](https://ollama.com/) installed locally for VLM generation
|
||||
|
||||
## Workflow Instructions
|
||||
|
||||
### Step 1: Setup Project and Install Dependencies
|
||||
|
||||
#### Project Bootstrap
|
||||
|
||||
Initialize a new project with `uv`:
|
||||
|
||||
```bash
|
||||
uv init multimodal-rag
|
||||
cd multimodal-rag
|
||||
uv venv
|
||||
```
|
||||
|
||||
**Install uv if needed:**
|
||||
```bash
|
||||
# macOS/Linux
|
||||
curl -LsSf https://astral.sh/uv/install.sh -o /tmp/uv-install.sh
|
||||
less /tmp/uv-install.sh
|
||||
sh /tmp/uv-install.sh
|
||||
|
||||
# Or with pip
|
||||
pip install uv
|
||||
|
||||
# Or with Homebrew
|
||||
brew install uv
|
||||
```
|
||||
|
||||
#### Install Core Dependencies
|
||||
|
||||
Install required libraries using `uv`:
|
||||
|
||||
```bash
|
||||
uv add weaviate-client
|
||||
```
|
||||
|
||||
**Package breakdown:**
|
||||
- `weaviate-client`: Python client for Weaviate vector database (v4.x) — Weaviate Embeddings handles all embedding generation
|
||||
|
||||
#### Additional Dependencies (Install as Needed)
|
||||
|
||||
```bash
|
||||
# For loading Hugging Face datasets
|
||||
uv add datasets
|
||||
|
||||
# For PDF processing (pdf2image requires poppler to be installed!)
|
||||
uv add pdf2image pillow
|
||||
|
||||
# For VLM generation via Ollama
|
||||
uv add ollama
|
||||
```
|
||||
|
||||
### Step 2: Prepare Your Document Dataset
|
||||
|
||||
#### Option A: Load Existing Dataset
|
||||
If using a pre-existing dataset:
|
||||
- Use Hugging Face `datasets` library
|
||||
- Ensure dataset contains document images or can be converted to images
|
||||
- Verify image format compatibility (JPEG, PNG)
|
||||
|
||||
#### Option B: Process Your Own Documents
|
||||
For custom document collections:
|
||||
1. Convert documents to images (if not already images)
|
||||
- PDFs: Use `pdf2image` or similar libraries
|
||||
- Office documents: Convert to PDF first, then to images
|
||||
2. Organize with metadata (document ID, page number, title, etc.)
|
||||
3. Store in a format suitable for batch processing
|
||||
|
||||
**Recommended structure:**
|
||||
```python
|
||||
{
|
||||
"document_id": str,
|
||||
"page_number": int,
|
||||
"image": PIL.Image,
|
||||
"metadata": dict # title, author, date, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Configure Weaviate Collection
|
||||
|
||||
#### Weaviate Connection
|
||||
|
||||
```python
|
||||
import os
|
||||
import weaviate
|
||||
from weaviate.classes.init import Auth
|
||||
|
||||
WEAVIATE_URL = os.getenv("WEAVIATE_URL")
|
||||
WEAVIATE_API_KEY = os.getenv("WEAVIATE_API_KEY")
|
||||
|
||||
client = weaviate.connect_to_weaviate_cloud(
|
||||
cluster_url=WEAVIATE_URL,
|
||||
auth_credentials=Auth.api_key(WEAVIATE_API_KEY),
|
||||
)
|
||||
```
|
||||
|
||||
#### Create Collection Schema
|
||||
|
||||
Define a collection with `multi2vec_weaviate` vectorizer for automatic multimodal embeddings:
|
||||
|
||||
```python
|
||||
from weaviate.classes.config import Configure, Property, DataType
|
||||
|
||||
collection_name = "PDFDocuments" # Use a descriptive name for your use case
|
||||
|
||||
collection = client.collections.create(
|
||||
name=collection_name,
|
||||
properties=[
|
||||
Property(name="doc_page", data_type=DataType.BLOB),
|
||||
Property(name="page_id", data_type=DataType.INT),
|
||||
Property(name="document_id", data_type=DataType.TEXT),
|
||||
Property(name="page_number", data_type=DataType.INT),
|
||||
Property(name="title", data_type=DataType.TEXT),
|
||||
# Add other metadata properties as needed
|
||||
],
|
||||
vector_config=[
|
||||
Configure.MultiVectors.multi2vec_weaviate(
|
||||
name="doc_vector"
|
||||
image_field="doc_page",
|
||||
model="ModernVBERT/colmodernvbert",
|
||||
encoding=Configure.VectorIndex.MultiVector.Encoding.muvera(
|
||||
ksim=4,
|
||||
dprojections=16,
|
||||
repetitions=20,
|
||||
),
|
||||
)
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
**Key Configuration Options:**
|
||||
- **`doc_page`**: BLOB property that holds base64-encoded page images — the vectorizer reads this field
|
||||
- **`image_field`**: Must match the BLOB property name (`"doc_page"`)
|
||||
- **`model`**: `ModernVBERT/colmodernvbert` — 250M parameter late-interaction vision-language encoder, fine-tuned for visual document retrieval
|
||||
- **MUVERA encoding**: Compresses multi-vectors into efficient single vectors while preserving retrieval quality
|
||||
- `ksim`: Number of similar vectors to consider (default: 4)
|
||||
- `dprojections`: Number of projection dimensions (default: 16)
|
||||
- `repetitions`: Number of encoding repetitions (default: 20)
|
||||
- **Properties**: Add all metadata you want to filter or display
|
||||
|
||||
**Without MUVERA encoding** (uses more memory but preserves full multi-vector representation):
|
||||
```python
|
||||
vector_config=[
|
||||
Configure.MultiVectors.multi2vec_weaviate(
|
||||
name="doc_vector",
|
||||
image_field="doc_page",
|
||||
model="ModernVBERT/colmodernvbert",
|
||||
)
|
||||
],
|
||||
```
|
||||
|
||||
### Step 4: Index Documents
|
||||
|
||||
#### Convert Images to Base64
|
||||
|
||||
```python
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
def image_to_base64(image):
|
||||
"""Convert a PIL Image to a base64-encoded string.
|
||||
|
||||
Args:
|
||||
image: PIL.Image object
|
||||
|
||||
Returns:
|
||||
Base64-encoded string of the JPEG image
|
||||
"""
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
return base64.b64encode(buffer.getvalue()).decode("utf-8")
|
||||
```
|
||||
|
||||
#### Batch Import
|
||||
|
||||
Weaviate Embeddings generates embeddings server-side during import — no local model needed:
|
||||
|
||||
```python
|
||||
collection = client.collections.get(collection_name)
|
||||
|
||||
with collection.batch.dynamic() as batch:
|
||||
for idx, document in enumerate(your_document_dataset):
|
||||
# Convert image to base64
|
||||
img_base64 = image_to_base64(document["image"])
|
||||
|
||||
# Add object to batch — Weaviate generates embeddings automatically
|
||||
batch.add_object(
|
||||
properties={
|
||||
"doc_page": img_base64,
|
||||
"page_id": document["page_id"],
|
||||
"document_id": document["document_id"],
|
||||
"page_number": document["page_number"],
|
||||
"title": document.get("title", ""),
|
||||
# Add other properties from your dataset
|
||||
},
|
||||
)
|
||||
|
||||
# Progress tracking
|
||||
if idx % 25 == 0:
|
||||
print(f"Indexed {idx+1}/{len(your_document_dataset)} documents")
|
||||
|
||||
# Clean up dataset if memory is limited
|
||||
del your_document_dataset
|
||||
|
||||
print(f"Total documents indexed: {len(collection)}")
|
||||
```
|
||||
|
||||
**Performance Tips:**
|
||||
- **Batch size**: Weaviate automatically manages batch size with `dynamic()` mode
|
||||
- **No local GPU needed**: Weaviate Embeddings runs server-side
|
||||
- **Image format**: JPEG is recommended for smaller payload sizes
|
||||
- **Large datasets**: Process in chunks, delete intermediate variables to free memory
|
||||
|
||||
### Step 5: Implement Retrieval
|
||||
|
||||
#### Basic Query Function
|
||||
|
||||
Weaviate handles query embedding automatically — just pass text:
|
||||
|
||||
```python
|
||||
from weaviate.classes.query import MetadataQuery
|
||||
|
||||
def search_documents(query_text, limit=3):
|
||||
"""Search for documents using Weaviate Embeddings multimodal model.
|
||||
|
||||
Args:
|
||||
query_text: Natural language query string
|
||||
limit: Number of results to return (default: 3)
|
||||
|
||||
Returns:
|
||||
List of dicts with document properties, similarity scores, and base64 images
|
||||
"""
|
||||
collection = client.collections.get(collection_name)
|
||||
|
||||
# Search — Weaviate embeds the query server-side
|
||||
# Include doc_page in return_properties to get the base64-encoded image blob
|
||||
response = collection.query.near_text(
|
||||
query=query_text,
|
||||
limit=limit,
|
||||
return_properties=["page_id", "document_id", "page_number", "title", "doc_page"],
|
||||
return_metadata=MetadataQuery(distance=True),
|
||||
)
|
||||
|
||||
# Process and format results
|
||||
results = []
|
||||
for i, obj in enumerate(response.objects):
|
||||
props = obj.properties
|
||||
results.append({
|
||||
"rank": i + 1,
|
||||
"page_id": props["page_id"],
|
||||
"document_id": props["document_id"],
|
||||
"page_number": props["page_number"],
|
||||
"title": props["title"],
|
||||
"distance": obj.metadata.distance,
|
||||
"image_base64": props["doc_page"], # Already base64-encoded
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
# Example usage
|
||||
query = "How does DeepSeek-V2 compare against the LLaMA family of LLMs?"
|
||||
results = search_documents(query, limit=3)
|
||||
|
||||
for result in results:
|
||||
print(f"{result['rank']}) Distance: {result['distance']:.4f}, "
|
||||
f"Title: \"{result['title']}\", Page: {result['page_number']}")
|
||||
```
|
||||
|
||||
**Query Parameters:**
|
||||
- **`limit`**: Number of results (1-10 recommended, consider VLM memory limits)
|
||||
- **`return_metadata`**: Include `distance=True` to get similarity scores
|
||||
- **Filters**: Add `filters=` for metadata filtering (see below)
|
||||
|
||||
**Accessing the image field in results:**
|
||||
BLOB properties like `doc_page` are not returned by default when used as the `image_field` property of the `multi2vec_weaviate` vectorizer. You must request them explicitly via `return_properties` (as shown in `search_documents()` above). The returned blob is base64-encoded. The Ollama Python SDK's `images` key accepts raw `bytes` or path-like strings (not base64 strings), so decode with `base64.b64decode()` before passing to Ollama (as shown in `OllamaVLM.generate_answer()`).
|
||||
|
||||
#### Metadata Filtering
|
||||
|
||||
Add filters to narrow search scope by document properties:
|
||||
|
||||
```python
|
||||
import weaviate.classes.config as wc
|
||||
|
||||
# Example: Filter by document ID
|
||||
response = collection.query.near_text(
|
||||
query="query text",
|
||||
limit=5,
|
||||
filters=wc.Filter.by_property("document_id").equal("paper_123"),
|
||||
)
|
||||
|
||||
# Example: Filter by page range
|
||||
response = collection.query.near_text(
|
||||
query="query text",
|
||||
limit=5,
|
||||
filters=wc.Filter.by_property("page_number").less_than(10),
|
||||
)
|
||||
|
||||
# Example: Combine multiple filters
|
||||
from weaviate.classes.query import Filter
|
||||
|
||||
response = collection.query.near_text(
|
||||
query="query text",
|
||||
limit=5,
|
||||
filters=(
|
||||
Filter.by_property("document_id").equal("paper_123") &
|
||||
Filter.by_property("page_number").less_than(10)
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
#### Hybrid Search
|
||||
|
||||
Combine vector search with BM25 keyword search:
|
||||
|
||||
```python
|
||||
# Hybrid search: vector + keyword (Weaviate handles embedding)
|
||||
response = collection.query.hybrid(
|
||||
query="query text",
|
||||
alpha=0.7, # 0.0=keyword only, 0.5=balanced, 1.0=vector only
|
||||
limit=5,
|
||||
)
|
||||
```
|
||||
|
||||
**When to use hybrid search:**
|
||||
- When exact keyword matches are important (e.g., searching for specific terms, IDs)
|
||||
- To combine semantic understanding with exact text matching (BM25)
|
||||
- Adjust `alpha` based on whether you prioritize semantic vs. keyword matching
|
||||
|
||||
### Step 6: Extend to Full RAG with a Vision Language Model
|
||||
|
||||
#### About Ollama
|
||||
|
||||
[Ollama](https://ollama.com/) makes it easy to run vision language models locally with a single command. No manual model downloads, GPU configuration, or dependency management required.
|
||||
|
||||
**Recommended VLM models for Ollama:**
|
||||
- `qwen3-vl:4b`: ~4 GB, good for limited hardware
|
||||
- `qwen3-vl:8b`: ~8 GB, better quality
|
||||
- `qwen3-vl:32b`: ~32 GB, highest quality
|
||||
- `gemma3`: Google's multimodal model, available in 4B/12B/27B sizes
|
||||
- `llava`: LLaVA model, lightweight and fast
|
||||
|
||||
#### Install Ollama and Pull a Model
|
||||
|
||||
```bash
|
||||
# Install Ollama (macOS/Linux)
|
||||
curl -fsSL https://ollama.com/install.sh -o /tmp/ollama-install.sh
|
||||
less /tmp/ollama-install.sh
|
||||
sh /tmp/ollama-install.sh
|
||||
|
||||
# Or on macOS with Homebrew
|
||||
brew install ollama
|
||||
|
||||
# Pull a vision language model
|
||||
ollama pull qwen3-vl:4b
|
||||
```
|
||||
|
||||
Verify the model is available:
|
||||
```bash
|
||||
ollama list
|
||||
```
|
||||
|
||||
#### Implement Ollama VLM Wrapper
|
||||
|
||||
```python
|
||||
import base64
|
||||
import ollama
|
||||
|
||||
class OllamaVLM:
|
||||
def __init__(self, model_name="qwen3-vl:4b"):
|
||||
"""Initialize with an Ollama vision model name.
|
||||
|
||||
Args:
|
||||
model_name: Ollama model tag (must support vision)
|
||||
"""
|
||||
self.model_name = model_name
|
||||
|
||||
def generate_answer(self, query, images_base64, max_tokens=128):
|
||||
"""Generate text response based on query and retrieved document images.
|
||||
|
||||
Args:
|
||||
query: String text query
|
||||
images_base64: List of base64-encoded image strings (as returned by Weaviate)
|
||||
max_tokens: Maximum tokens to generate (default: 128)
|
||||
|
||||
Returns:
|
||||
Generated text answer as string
|
||||
"""
|
||||
# The Ollama SDK "images" key accepts bytes or path-like strings,
|
||||
# so decode the base64 strings from Weaviate into raw bytes
|
||||
images_bytes = [base64.b64decode(img) for img in images_base64]
|
||||
|
||||
response = ollama.chat(
|
||||
model=self.model_name,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": query,
|
||||
"images": images_bytes,
|
||||
}],
|
||||
options={"num_predict": max_tokens},
|
||||
)
|
||||
|
||||
return response["message"]["content"]
|
||||
|
||||
# Instantiate the VLM
|
||||
vlm = OllamaVLM(model_name="qwen3-vl:4b")
|
||||
```
|
||||
|
||||
#### Complete RAG Pipeline
|
||||
|
||||
```python
|
||||
def multimodal_rag(query, num_documents=3, max_tokens=128):
|
||||
"""Complete multimodal RAG pipeline using Weaviate Embeddings + Ollama VLM.
|
||||
|
||||
Args:
|
||||
query: Natural language question
|
||||
num_documents: Number of documents to retrieve (1-3 recommended)
|
||||
max_tokens: Maximum tokens for VLM response
|
||||
|
||||
Returns:
|
||||
Dict with query, answer, sources, and metadata
|
||||
"""
|
||||
# Step 1: Retrieve relevant documents (Weaviate handles embedding)
|
||||
print(f"Searching for: {query}")
|
||||
retrieved_docs = search_documents(query, limit=num_documents)
|
||||
|
||||
# Display retrieved sources
|
||||
print(f"\nRetrieved {len(retrieved_docs)} documents:")
|
||||
for doc in retrieved_docs:
|
||||
print(f" - {doc['title']}, Page {doc['page_number']} "
|
||||
f"(Distance: {doc['distance']:.4f})")
|
||||
|
||||
# Step 2: Extract base64 images from results
|
||||
context_images = [doc["image_base64"] for doc in retrieved_docs]
|
||||
|
||||
# Step 3: Generate answer using Ollama VLM
|
||||
print(f"\nGenerating answer...")
|
||||
answer = vlm.generate_answer(query, context_images, max_tokens=max_tokens)
|
||||
|
||||
# Step 4: Return structured response
|
||||
return {
|
||||
"query": query,
|
||||
"answer": answer,
|
||||
"sources": retrieved_docs,
|
||||
"num_sources": len(retrieved_docs)
|
||||
}
|
||||
|
||||
# Example usage
|
||||
query = "How does DeepSeek-V2 compare against the LLaMA family of LLMs?"
|
||||
result = multimodal_rag(query, num_documents=1, max_tokens=128)
|
||||
|
||||
print(f"\nQuery: {result['query']}")
|
||||
print(f"Answer: {result['answer']}")
|
||||
print(f"\nBased on {result['num_sources']} source(s)")
|
||||
```
|
||||
|
||||
#### Response Citation
|
||||
|
||||
Include source attribution in generated answers:
|
||||
|
||||
```python
|
||||
def generate_with_citations(query, retrieved_docs, max_tokens=256):
|
||||
"""Generate answer with source citations.
|
||||
|
||||
Args:
|
||||
query: User question
|
||||
retrieved_docs: List of documents from search_documents()
|
||||
max_tokens: Maximum response length
|
||||
|
||||
Returns:
|
||||
Answer string with embedded citations
|
||||
"""
|
||||
# Build source references
|
||||
sources_text = "\n".join([
|
||||
f"Source {i+1}: \"{doc['title']}\", Page {doc['page_number']}"
|
||||
for i, doc in enumerate(retrieved_docs)
|
||||
])
|
||||
|
||||
# Enhanced prompt with citation instructions
|
||||
enhanced_query = f"""{query}
|
||||
|
||||
Available sources:
|
||||
{sources_text}
|
||||
|
||||
Instructions: Answer the question based on the provided document images.
|
||||
Cite sources in your answer using [Source N] notation."""
|
||||
|
||||
# Generate answer with citations
|
||||
answer = vlm.generate_answer(
|
||||
enhanced_query,
|
||||
[doc["image_base64"] for doc in retrieved_docs],
|
||||
max_tokens=max_tokens
|
||||
)
|
||||
|
||||
return answer, retrieved_docs
|
||||
|
||||
# Example usage
|
||||
query = "What is the architecture of GPT-4?"
|
||||
answer, sources = generate_with_citations(query, search_documents(query, limit=3))
|
||||
print(f"Answer: {answer}\n")
|
||||
print("Sources:")
|
||||
for src in sources:
|
||||
print(f" - {src['title']}, Page {src['page_number']}")
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing Environment Variables
|
||||
```
|
||||
Error: WEAVIATE_URL environment variable is not set
|
||||
```
|
||||
**Solution:** Set `WEAVIATE_URL` and `WEAVIATE_API_KEY` environment variables. See `environment_requirements.md`.
|
||||
|
||||
### Connection Errors
|
||||
```
|
||||
WeaviateConnectionError: Failed to connect to Weaviate
|
||||
```
|
||||
**Solution:** Verify `WEAVIATE_URL` is correct and your network can reach the Weaviate Cloud instance.
|
||||
|
||||
### Ollama Connection Error
|
||||
```
|
||||
ConnectionError: Failed to connect to Ollama
|
||||
```
|
||||
**Solution:** Make sure Ollama is running. Start it with:
|
||||
```bash
|
||||
ollama serve
|
||||
```
|
||||
|
||||
### Ollama Model Not Found
|
||||
```
|
||||
ollama._types.ResponseError: model 'qwen3-vl:4b' not found
|
||||
```
|
||||
**Solution:** Pull the model first:
|
||||
```bash
|
||||
ollama pull qwen3-vl:4b
|
||||
```
|
||||
|
||||
### Out of Memory (OOM) During VLM Generation
|
||||
**Symptoms:** Out of memory errors when generating answers.
|
||||
|
||||
**Solutions:**
|
||||
- Reduce `num_documents` — retrieve fewer documents (even 1 can work well)
|
||||
- Reduce `max_tokens` — shorter responses use less memory
|
||||
- Use a smaller model variant (`qwen3-vl:4b` instead of `8b`)
|
||||
- Use API-based VLMs (GPT-4V, Claude, Gemini) to avoid local resource requirements entirely
|
||||
|
||||
### BLOB Property Not Returned in Query Results
|
||||
**Symptom:** `doc_page` field is missing from query results.
|
||||
|
||||
**Solution:** BLOB properties used as `image_field` in `multi2vec_weaviate` are not returned by default. Specify them explicitly:
|
||||
```python
|
||||
response = collection.query.near_text(
|
||||
query=query_text,
|
||||
limit=limit,
|
||||
return_properties=["page_id", "document_id", "page_number", "title", "doc_page"],
|
||||
)
|
||||
```
|
||||
|
||||
### Poppler Not Installed (PDF Processing)
|
||||
```
|
||||
Exception: Unable to get page count. Is poppler installed and in PATH?
|
||||
```
|
||||
**Solution:** Install poppler for `pdf2image`:
|
||||
```bash
|
||||
# macOS
|
||||
brew install poppler
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install poppler-utils
|
||||
```
|
||||
|
||||
### TypeError: unexpected keyword argument 'image_fields'
|
||||
```
|
||||
TypeError: _MultiVectors.multi2vec_weaviate() got an unexpected keyword argument 'image_fields'
|
||||
```
|
||||
**Cause:** The parameter is singular, not a list.
|
||||
|
||||
**Solution:** Use `image_field` (singular) instead of `image_fields`:
|
||||
```python
|
||||
Configure.MultiVectors.multi2vec_weaviate(
|
||||
name="doc_vector",
|
||||
image_field="doc_page",
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
## Done Criteria
|
||||
|
||||
The implementation is complete when:
|
||||
- [ ] Project is initialized with `uv` and all dependencies are installed
|
||||
- [ ] Document images are converted and uploaded to a Weaviate collection with `multi2vec_weaviate` vectorizer
|
||||
- [ ] The collection uses `ModernVBERT/colmodernvbert` model with MUVERA encoding configured
|
||||
- [ ] `search_documents()` returns ranked results with similarity scores for text queries
|
||||
- [ ] Ollama with a vision language model generates natural language answers from retrieved document images
|
||||
- [ ] The full `multimodal_rag()` pipeline retrieves documents and generates answers end-to-end
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **Add metadata filtering** to narrow search scope by document ID, page range, or other properties
|
||||
- **Implement hybrid search** combining vector similarity with BM25 keyword matching for better precision
|
||||
- **Add response citations** using `generate_with_citations()` to attribute answers to source documents
|
||||
- **Scale the dataset** by processing larger document collections with batch chunking and memory management
|
||||
- **Swap in API-based VLMs** (GPT, Claude, Gemini) or other Ollama vision models (`gemma3`, `llava`) as alternatives
|
||||
- **Evaluate retrieval quality** by testing queries against known-relevant documents and tuning MUVERA parameters
|
||||
@@ -0,0 +1,75 @@
|
||||
# Project Setup Contract (All Cookbooks)
|
||||
|
||||
Use this reference before generating any cookbook app.
|
||||
|
||||
## Goal
|
||||
|
||||
Set up a safe default project layout that prevents accidental secret leaks and keeps setup instructions consistent across all cookbooks.
|
||||
|
||||
## Required Order
|
||||
|
||||
1. Create project directory.
|
||||
2. Initialize git immediately.
|
||||
3. Create `.gitignore` before any local `.env` file.
|
||||
4. Create `.env` from [environment_requirements.md](environment_requirements.md).
|
||||
5. Ask user to fill required values (`WEAVIATE_URL`, `WEAVIATE_API_KEY`) and only the optional keys they need.
|
||||
|
||||
## Required Files
|
||||
|
||||
### `.gitignore`
|
||||
|
||||
```gitignore
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.venv/
|
||||
|
||||
# Node
|
||||
node_modules/
|
||||
.next/
|
||||
out/
|
||||
dist/
|
||||
|
||||
# Local env files (never commit secrets)
|
||||
.env
|
||||
.env.*
|
||||
secrets/
|
||||
|
||||
# Common local artifacts
|
||||
.DS_Store
|
||||
```
|
||||
|
||||
### `.env`
|
||||
|
||||
- Use the canonical template as provided in [environment_requirements.md](environment_requirements.md).
|
||||
- Keep real `.env` values local only.
|
||||
|
||||
## Git Baseline
|
||||
|
||||
Run these commands in every new cookbook app:
|
||||
|
||||
```bash
|
||||
git init
|
||||
git add .gitignore
|
||||
git commit -m "initialize project baseline"
|
||||
```
|
||||
|
||||
## Claude Safety Baseline (Recommended)
|
||||
|
||||
For projects developed with Claude Code, add deny rules for local secret files:
|
||||
|
||||
```json
|
||||
{
|
||||
"permissions": {
|
||||
"deny": [
|
||||
"Read(./.env)",
|
||||
"Read(./.env.*)",
|
||||
"Read(./**/.env)",
|
||||
"Read(./**/.env.*)",
|
||||
"Read(./secrets/**)"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Save this to `.claude/settings.json` at project root.
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
# Build Weaviate Query Agent Chatbot
|
||||
|
||||
## Overview
|
||||
|
||||
Build a full-stack Query Agent chatbot with minimal back-and-forth.
|
||||
|
||||
Read first:
|
||||
- Weaviate Query Agent usage: https://docs.weaviate.io/agents/query/usage
|
||||
|
||||
## Instructions
|
||||
|
||||
### Core Rules
|
||||
|
||||
- Use `uv` for Python project/dependency management.
|
||||
- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
|
||||
- Use this backend install set:
|
||||
- `uv add fastapi 'uvicorn[standard]' weaviate-client weaviate-agents pydantic-settings sse-starlette python-dotenv`
|
||||
- If `uv` not available, create a `requirements.txt` for pip installation
|
||||
- Depending on user request: consider combining this app with the Data Explorer.
|
||||
- If the user explicitly only wants chatbot, create this app independently
|
||||
- If the user wants a fully featured chat and data explorer, combine the apps
|
||||
- If no explicit instructions are given, ask the user their preference before continuing
|
||||
- See the [Next Steps](#next-steps) section for more details
|
||||
|
||||
### Fast Setup Commands
|
||||
|
||||
Project bootstrap:
|
||||
|
||||
```bash
|
||||
uv init chatbot
|
||||
cd chatbot
|
||||
uv venv
|
||||
uv add fastapi 'uvicorn[standard]' weaviate-client weaviate-agents pydantic-settings sse-starlette python-dotenv
|
||||
```
|
||||
|
||||
### Workflow Contract
|
||||
|
||||
1. Build backend in one pass.
|
||||
2. Create `.env` from the canonical template in `environment_requirements.md`, then add app-specific fields (for example, `COLLECTIONS`).
|
||||
3. Before asking user to fill env, do non-secret local sanity checks that do not require real credentials (imports/compile/startup-shape checks).
|
||||
4. Ask user to fill real env values:
|
||||
- Mandatory: `WEAVIATE_URL`, `WEAVIATE_API_KEY`, `COLLECTIONS`
|
||||
- Optional: only provider keys required by their collection setup
|
||||
5. After the user confirms, verify backend starts without errors and provide exact commands to run it in terminal.
|
||||
|
||||
Do not ask avoidable questions that you can resolve from context.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
Use a modular layout like:
|
||||
|
||||
```text
|
||||
chatbot/
|
||||
backend/
|
||||
app/
|
||||
main.py
|
||||
config.py
|
||||
lifespan.py
|
||||
dependencies.py
|
||||
routers/
|
||||
services/
|
||||
models/
|
||||
.env # local file, never committed
|
||||
```
|
||||
|
||||
Keep these boundaries:
|
||||
|
||||
- routers: HTTP only
|
||||
- services: business/query-agent logic
|
||||
- models: request/response schemas
|
||||
- config/lifespan: wiring and startup/shutdown
|
||||
|
||||
### Backend Requirements
|
||||
|
||||
- FastAPI async app with lifespan.
|
||||
- Async Weaviate client initialized in lifespan and closed on shutdown.
|
||||
- Query Agent service layer (`ask` + `ask_stream`).
|
||||
- For async FastAPI backends, use `AsyncQueryAgent` (not `QueryAgent`) so `await agent.ask(...)` and `async for ... in agent.ask_stream(...)` work correctly.
|
||||
- Endpoints:
|
||||
- `GET /health`
|
||||
- `POST /chat`
|
||||
- `POST /chat/stream` (SSE)
|
||||
- Pydantic settings should read from process environment; local `.env` loading is optional for local development.
|
||||
- Conversation history mapping to Weaviate chat message format.
|
||||
|
||||
### Source Handling
|
||||
|
||||
- For every ask response, normalize output into:
|
||||
- `answer`: text from `response.final_answer` (fallback `""`)
|
||||
- `sources`: list of `{ "collection": ..., "object_id": ... }` built from `response.sources`
|
||||
- `source_count`: `len(sources)`
|
||||
- `POST /chat` must return `answer`, `sources`, and `source_count`.
|
||||
- `POST /chat/stream` must include the same fields in the final SSE event.
|
||||
- If no sources are available, return `sources: []` and `source_count: 0`.
|
||||
|
||||
### Env Rules
|
||||
|
||||
Mandatory:
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
- `COLLECTIONS`
|
||||
|
||||
External provider keys:
|
||||
- Include every provider key needed by the target collections.
|
||||
- Leave unused provider keys empty/commented.
|
||||
|
||||
CORS:
|
||||
|
||||
- Default `CORS_ORIGINS` should include:
|
||||
- `http://localhost:3000`
|
||||
- `http://127.0.0.1:3000`
|
||||
- `http://localhost:5173`
|
||||
- `http://127.0.0.1:5173`
|
||||
|
||||
### Post-Env Hand-Holding (Required)
|
||||
|
||||
After user says required env values are set, provide the terminal commands to run the backend:
|
||||
|
||||
```bash
|
||||
cd chatbot/backend
|
||||
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
- Ask user to start the terminal.
|
||||
- Run smoke tests yourself against running services.
|
||||
- Report pass/fail in plain language and fix blockers.
|
||||
|
||||
Do not offload detailed testing steps to the user unless they explicitly ask.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- `OPTIONS /chat/stream 400`: fix CORS origin mismatch (`localhost` vs `127.0.0.1`).
|
||||
- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
|
||||
- For any other issues, refer to the official library/package documentation using web search.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- Backend healthy.
|
||||
- `/chat` works.
|
||||
- `/chat/stream` streams progress/token/final.
|
||||
- `/chat` and `/chat/stream` final include `sources` and `source_count`.
|
||||
- User can run the server in the terminal with the provided commands.
|
||||
|
||||
## Next Steps
|
||||
|
||||
|
||||
This application is currently a chatbot backend. You may optionally offer to integrate it with the [Data Explorer](./data_explorer.md) based on user preference.
|
||||
|
||||
If the user chooses to combine these two applications, implement the integration as follows:
|
||||
|
||||
- Create or use a directory `/routes` which separate functions for query agent chat and data exploration. Import the routers in the `main.py` file
|
||||
- If a frontend is requested, the frontend should have multiple pages/tabs depending on design choices so that data exploration and chat is separated
|
||||
- Consider crossovers between functionalities, e.g. a chat button from the data viewer/collection viewer which takes the user to chat with that collection selected.
|
||||
- Run quick tests to ensure the integration is seamless and the user can use both the chatbot and data explorer without any issues.
|
||||
|
||||
### Frontend
|
||||
|
||||
When the user explicitly asks for a frontend, use this reference as guideline:
|
||||
|
||||
- [Frontend Interface](frontend_interface.md): Build a Next.js frontend to interact with the Weaviate backend.
|
||||
- Render source citations from `sources` and `source_count` in the chat response UI.
|
||||
Reference in New Issue
Block a user