📦 deps(thirdparty): update snapshots
This commit is contained in:
+36
@@ -0,0 +1,36 @@
|
||||
# Query Agent - Ask Mode
|
||||
|
||||
Generate AI-powered answers with source citations using the Weaviate Query Agent.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/ask.py --query "USER_QUESTION" --collections "Collection1,Collection2" [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--query` | `-q` | Yes | — | Natural language question |
|
||||
| `--collections` | `-c` | Yes | — | Comma-separated collection names to query across |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown-formatted answer with a sources table
|
||||
- **JSON**: Structured response with `answer` and `sources` fields
|
||||
|
||||
## Examples
|
||||
|
||||
Ask a question across multiple collections:
|
||||
|
||||
```bash
|
||||
uv run scripts/ask.py --query "What are the main topics in the dataset?" --collections "Articles,Reports"
|
||||
```
|
||||
|
||||
JSON output:
|
||||
|
||||
```bash
|
||||
uv run scripts/ask.py --query "Summarize recent findings" --collections "Research" --json
|
||||
```
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
# Create Collection
|
||||
|
||||
Create a new Weaviate collection with a custom schema, optional vectorizer, and multi-tenancy support.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/create_collection.py CollectionName --properties '[...]' [--description "..."] [--vectorizer "..."] [--replication-factor N] [--multi-tenancy] [--auto-tenant-creation] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `name` | — | Yes (positional) | — | Collection name (auto-capitalized per GraphQL convention) |
|
||||
| `--properties` | `-p` | Yes | — | JSON array of property definitions |
|
||||
| `--description` | `-d` | No | — | Collection description — **strongly recommended**. Weaviate agents (Query Agent, Personalization Agent) read this to understand what the collection contains and decide which collection to query |
|
||||
| `--vectorizer` | `-v` | No | `text2vec_weaviate` | Vectorizer module to use |
|
||||
| `--replication-factor` | `-r` | No | — | Replication factor (defers to server default when not set) |
|
||||
| `--multi-tenancy` | `-m` | No | `false` | Enable multi-tenancy for data isolation |
|
||||
| `--auto-tenant-creation` | `-a` | No | `false` | Auto-create tenants on insert (requires `--multi-tenancy`) |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Property Definition Format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "property_name",
|
||||
"data_type": "text",
|
||||
"description": "Optional description",
|
||||
"tokenization": "word",
|
||||
"index_filterable": true,
|
||||
"index_searchable": true,
|
||||
"index_range_filters": false,
|
||||
"nested_properties": []
|
||||
}
|
||||
```
|
||||
|
||||
- `name` (required): Property name
|
||||
- `data_type` (required): One of the supported data types below
|
||||
- `description` (optional): Human-readable description — **strongly recommended**. The Query Agent reads property descriptions to understand your schema, choose the right collection, and construct accurate queries. Good descriptions include units, formats, and valid values (e.g., `"Price in US dollars (USD)"`, `"ISO two-character country code"`, `"Date the paper was published on arXiv"`)
|
||||
- `tokenization` (optional): For text types — `word`, `lowercase`, `whitespace`, or `field`
|
||||
- `index_filterable` (optional): Enable roaring-bitmap filter index for `where` clauses. Default `true` for all types except `blob`, `geoCoordinates`, `object`, `object[]`, `phoneNumber`
|
||||
- `index_searchable` (optional): Enable BM25/inverted index for keyword and hybrid search. Only applies to `text` and `text[]`. Default `true`
|
||||
- `index_range_filters` (optional): Enable range-comparison index (`>`, `<`, `>=`, `<=`, `between`) for `int`, `int[]`, `number`, `number[]`, `date`, `date[]`. Default `false` — **set to `true` for any numeric or date field you plan to range-filter**
|
||||
- `nested_properties` (optional): For `object` / `object[]` types — array of nested property definitions
|
||||
|
||||
## Supported Data Types
|
||||
|
||||
`text`, `text[]`, `boolean`, `boolean[]`, `int`, `int[]`, `number`, `number[]`, `date`, `date[]`, `uuid`, `uuid[]`, `geoCoordinates`, `phoneNumber`, `blob`, `object`, `object[]`
|
||||
|
||||
Aliases: `bool` → `boolean`, `bool[]` → `boolean[]`
|
||||
|
||||
## Supported Vectorizers
|
||||
|
||||
`text2vec_weaviate`, `text2vec_openai`, `text2vec_cohere`, `text2vec_huggingface`, `text2vec_palm`, `text2vec_jinaai`, `text2vec_voyageai`, `text2vec_contextionary`, `text2vec_transformers`, `text2vec_gpt4all`, `text2vec_ollama`, `multi2vec_clip`, `multi2vec_bind`, `multi2vec_palm`, `img2vec_neural`, `ref2vec_centroid`, `none`
|
||||
|
||||
## Inferring Schema from Data Files
|
||||
|
||||
Before creating a collection, inspect a few rows from the source file to understand field names and value types. Use the commands below — they read only the first 3 objects and are safe on large files.
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
python3 -c "
|
||||
import csv, json
|
||||
with open('data.csv') as f:
|
||||
rows = list(csv.DictReader(f))[:3]
|
||||
print(json.dumps(rows, indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
python3 -c "
|
||||
import json
|
||||
print(json.dumps(json.load(open('data.json'))[:3], indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
**JSONL:**
|
||||
```bash
|
||||
python3 -c "
|
||||
import json
|
||||
lines = []
|
||||
with open('data.jsonl') as f:
|
||||
for line in f:
|
||||
if len(lines) >= 3: break
|
||||
if line.strip(): lines.append(json.loads(line))
|
||||
print(json.dumps(lines, indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
From the sample, map each field to a Weaviate data type:
|
||||
|
||||
| Value looks like | data_type |
|
||||
|---|---|
|
||||
| `"hello"`, any text | `text` |
|
||||
| `123`, `"123"` | `int` |
|
||||
| `1.5`, `"1.5"` | `number` |
|
||||
| `true`/`false` | `boolean` |
|
||||
| `"2024-01-15"`, `"2024-01-15T10:30:00Z"` | `date` |
|
||||
| UUID-shaped string | `uuid` |
|
||||
| List of strings | `text[]` |
|
||||
| List of numbers | `int[]` or `number[]` |
|
||||
| Nested object | `object` |
|
||||
|
||||
**Important:** `id`, `_id`, and `_additional` are reserved by Weaviate — never use them as property names. If they appear in your data, use `--skip-fields` or `--mapping` in `import.py` to handle them.
|
||||
|
||||
## Examples
|
||||
|
||||
Basic collection:
|
||||
|
||||
```bash
|
||||
uv run scripts/create_collection.py Article \
|
||||
--description "News articles with title and full body text." \
|
||||
--properties '[
|
||||
{"name": "title", "data_type": "text", "description": "Title of the article"},
|
||||
{"name": "body", "data_type": "text", "description": "Full text body of the article"}
|
||||
]'
|
||||
```
|
||||
|
||||
Collection with various data types, descriptions, and recommended index flags:
|
||||
|
||||
```bash
|
||||
uv run scripts/create_collection.py Product \
|
||||
--description "E-commerce product catalog with pricing, brand, stock status, and tags." \
|
||||
--properties '[
|
||||
{"name": "name", "data_type": "text", "description": "Name or title of the product"},
|
||||
{"name": "sku", "data_type": "text", "index_searchable": false, "description": "Stock-keeping unit identifier"},
|
||||
{"name": "price", "data_type": "number", "index_range_filters": true, "description": "Product price in US dollars (USD)"},
|
||||
{"name": "created_at", "data_type": "date", "index_range_filters": true, "description": "Date the product was added to the catalog"},
|
||||
{"name": "in_stock", "data_type": "boolean", "description": "Whether the product is currently in stock"},
|
||||
{"name": "tags", "data_type": "text[]", "description": "List of descriptive tags for the product"}
|
||||
]'
|
||||
```
|
||||
|
||||
With explicit vectorizer:
|
||||
|
||||
```bash
|
||||
uv run scripts/create_collection.py Article \
|
||||
--description "News articles with title and full body text." \
|
||||
--properties '[{"name": "title", "data_type": "text", "description": "Title of the article"}]' \
|
||||
--vectorizer "text2vec_openai"
|
||||
```
|
||||
|
||||
With multi-tenancy:
|
||||
|
||||
```bash
|
||||
uv run scripts/create_collection.py Workspace \
|
||||
--properties '[{"name": "content", "data_type": "text"}]' \
|
||||
--multi-tenancy --auto-tenant-creation
|
||||
```
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
# Environment Requirements for Weaviate
|
||||
|
||||
Use this reference when building apps that connect to Weaviate and require external inference provider keys.
|
||||
|
||||
## Required Weaviate Auth
|
||||
|
||||
- `WEAVIATE_URL`
|
||||
- `WEAVIATE_API_KEY`
|
||||
|
||||
## External Provider Env Vars and Headers
|
||||
|
||||
| Provider | Environment Variable(s) | Header(s) sent to Weaviate |
|
||||
|----------|--------------------------|-----------------------------|
|
||||
| Anthropic | `ANTHROPIC_API_KEY` | `X-Anthropic-Api-Key` |
|
||||
| Anyscale | `ANYSCALE_API_KEY` | `X-Anyscale-Api-Key` |
|
||||
| AWS | `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` | `X-Aws-Access-Key`, `X-Aws-Secret-Key` |
|
||||
| Cohere | `COHERE_API_KEY` | `X-Cohere-Api-Key` |
|
||||
| Databricks | `DATABRICKS_TOKEN` | `X-Databricks-Token` |
|
||||
| Friendli | `FRIENDLI_TOKEN` | `X-Friendli-Api-Key` |
|
||||
| Google Vertex AI | `VERTEX_API_KEY` | `X-Goog-Vertex-Api-Key` |
|
||||
| Google AI Studio | `STUDIO_API_KEY` | `X-Goog-Studio-Api-Key` |
|
||||
| HuggingFace | `HUGGINGFACE_API_KEY` | `X-HuggingFace-Api-Key` |
|
||||
| Jina AI | `JINAAI_API_KEY` | `X-JinaAI-Api-Key` |
|
||||
| Mistral | `MISTRAL_API_KEY` | `X-Mistral-Api-Key` |
|
||||
| NVIDIA | `NVIDIA_API_KEY` | `X-Nvidia-Api-Key` |
|
||||
| OpenAI | `OPENAI_API_KEY` | `X-OpenAI-Api-Key` |
|
||||
| Azure OpenAI | `AZURE_API_KEY` | `X-Azure-Api-Key` |
|
||||
| Voyage AI | `VOYAGE_API_KEY` | `X-Voyage-Api-Key` |
|
||||
| xAI | `XAI_API_KEY` | `X-Xai-Api-Key` |
|
||||
|
||||
## Usage Notes
|
||||
|
||||
- Set only the provider keys your collection configuration actually uses.
|
||||
- If multiple providers are configured, include all corresponding headers.
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
# Example Data
|
||||
|
||||
Add example data to a Weaviate collection for users without their own data or wanting a quick example. Downloads data from the huggingface hub.
|
||||
|
||||
```bash
|
||||
uv run scripts/example_data.py --domain "DOMAIN_NAME" [--vectorizer "..."] [--nrows X]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--domain` | `-d` | No | `academic` | Defines which dataset is being used. One of 'academic', 'finance', 'ecommerce', 'medical', or 'customer_support'. |
|
||||
| `--vectorizer` | `-v` | No | `text2vec_weaviate` | Optional vectorizer (e.g., `text2vec_openai`, `text2vec_cohere`, `none`) |
|
||||
| `--nrows` | `-n` | No | `None` | Optionally subset the data. If not supplied uses full dataset. |
|
||||
|
||||
**When to use:** Creating example data for immediate use of other skills, if no data is available or user requests some toy data.
|
||||
|
||||
**Domain Datasets:**
|
||||
- `academic` is the `jamescalam/ai-arxiv2` dataset, contains a selection of chunked papers from Arxiv on the topic of AI/ML. Creates the `AI_Arxiv` collection in the Weaviate instance
|
||||
- `finance` is the `AgamiAI/Indian-Income-Tax-Returns` dataset, fully synthetic Indian Income Tax Return forms. Creates the `Income_Tax_Returns` collection in the Weaviate instance
|
||||
- `ecommerce` is the `pkghf/ecom-product-catalog` dataset, containing structured e-commerce product information including product details, pricing, categorization. Creates the `Product_Catalog` collection in the Weaviate instance
|
||||
- `medical` is the `Amod/hair_medical_sit`, containing information about common hair related diseases. Creates the `Hair_Medical` collection in the Weaviate instance
|
||||
- `customer_support` is the `Console-AI/IT-helpdesk-synthetic-tickets`, synthetic customer support tickets from IT. Creates the `IT_Support_Tickets` collection in the Weaviate instance
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
# Explore Collection
|
||||
|
||||
Get statistical insights, aggregation metrics, and sample data from a collection.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/explore_collection.py "CollectionName" [--limit 5] [--no-metrics] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `name` | — | Yes (positional) | — | Collection name |
|
||||
| `--limit` | `-l` | No | `5` | Number of sample objects to show |
|
||||
| `--no-metrics` | — | No | `false` | Skip calculating individual property metrics (faster) |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Metrics by Data Type
|
||||
|
||||
The script calculates aggregation metrics based on property data types:
|
||||
|
||||
| Data Type | Metrics |
|
||||
|-----------|---------|
|
||||
| **Text** | count, top_occurrences (top 5 values with counts) |
|
||||
| **Int / Number** | count, min, max, mean, median, mode, sum |
|
||||
| **Boolean** | count, percentage_true, percentage_false, total_true, total_false |
|
||||
| **Date** | count, min, max, median, mode |
|
||||
|
||||
Use `--no-metrics` to skip metric calculation for faster results when you only need sample objects.
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown-formatted report with total count, per-property metrics tables, and sample objects
|
||||
- **JSON**: Structured metrics and sample data
|
||||
|
||||
## Examples
|
||||
|
||||
Explore with default settings:
|
||||
|
||||
```bash
|
||||
uv run scripts/explore_collection.py "Articles"
|
||||
```
|
||||
|
||||
More samples, skip metrics:
|
||||
|
||||
```bash
|
||||
uv run scripts/explore_collection.py "Products" --limit 20 --no-metrics
|
||||
```
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
# Fetch and Filter
|
||||
|
||||
Fetch objects from a collection by UUID, with filters, or as a random sample. Supports complex nested filter logic (AND, OR).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/fetch_filter.py "CollectionName" [--id "UUID"] [--filters 'JSON'] [--limit 10] [--properties "prop1,prop2"] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `collection_name` | — | Yes (positional) | — | Collection name |
|
||||
| `--id` | — | No | — | Fetch a specific object by UUID |
|
||||
| `--filters` | `-f` | No | — | JSON string defining filters (see filter syntax below) |
|
||||
| `--limit` | `-l` | No | `10` | Number of objects to fetch |
|
||||
| `--properties` | `-p` | No | all | Comma-separated properties to include in output |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Modes
|
||||
|
||||
1. **Fetch by UUID**: Use `--id` to retrieve a specific object
|
||||
2. **Fetch with filters**: Use `--filters` to retrieve filtered subsets
|
||||
3. **Fetch random sample**: Omit both `--id` and `--filters` for unfiltered results
|
||||
|
||||
## Filter Syntax
|
||||
|
||||
### Simple property filter
|
||||
|
||||
```json
|
||||
{"property": "category", "operator": "equal", "value": "Science"}
|
||||
```
|
||||
|
||||
### Logical operators (AND / OR)
|
||||
|
||||
```json
|
||||
{"operator": "and", "filters": [
|
||||
{"property": "category", "operator": "equal", "value": "Science"},
|
||||
{"property": "year", "operator": "greater_than", "value": 2020}
|
||||
]}
|
||||
```
|
||||
|
||||
### List of filters (implicit AND)
|
||||
|
||||
```json
|
||||
[
|
||||
{"property": "category", "operator": "equal", "value": "Science"},
|
||||
{"property": "year", "operator": "greater_than", "value": 2020}
|
||||
]
|
||||
```
|
||||
|
||||
### Supported operators
|
||||
|
||||
`equal`, `not_equal`, `less_than`, `less_or_equal`, `greater_than`, `greater_or_equal`, `like`, `contains_any`, `contains_all`, `is_none`
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with object UUIDs and properties
|
||||
- **JSON**: Array of objects with full metadata
|
||||
|
||||
## Examples
|
||||
|
||||
Fetch by UUID:
|
||||
|
||||
```bash
|
||||
uv run scripts/fetch_filter.py "Articles" --id "550e8400-e29b-41d4-a716-446655440000"
|
||||
```
|
||||
|
||||
Filter by property:
|
||||
|
||||
```bash
|
||||
uv run scripts/fetch_filter.py "Products" --filters '{"property": "price", "operator": "less_than", "value": 50}'
|
||||
```
|
||||
|
||||
Complex filter with AND/OR:
|
||||
|
||||
```bash
|
||||
uv run scripts/fetch_filter.py "Articles" --filters '{"operator": "or", "filters": [{"property": "category", "operator": "equal", "value": "Science"}, {"property": "category", "operator": "equal", "value": "Tech"}]}'
|
||||
```
|
||||
|
||||
Select specific properties:
|
||||
|
||||
```bash
|
||||
uv run scripts/fetch_filter.py "Products" --properties "name,price" --limit 5
|
||||
```
|
||||
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
# Get Collection Details
|
||||
|
||||
Get detailed configuration of a specific collection including vectorizer, properties, replication, and multi-tenancy settings.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/get_collection.py --name "CollectionName" [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--name` | `-n` | Yes | — | Collection name |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown-formatted collection details with property table
|
||||
- **JSON**: Full collection configuration object
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
uv run scripts/get_collection.py --name "Articles"
|
||||
```
|
||||
|
||||
```bash
|
||||
uv run scripts/get_collection.py --name "Products" --json
|
||||
```
|
||||
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
# Hybrid Search
|
||||
|
||||
Combines vector similarity and keyword (BM25) matching for balanced search results on a single collection.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/hybrid_search.py --query "USER_QUERY" --collection "CollectionName" [--alpha 0.7] [--limit 10] [--properties "prop1,prop2"] [--target-vector "vector_name"] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--query` | `-q` | Yes | — | Search query text |
|
||||
| `--collection` | `-c` | Yes | — | Collection name |
|
||||
| `--alpha` | `-a` | No | `0.7` | Balance between vector (1.0) and keyword (0.0) |
|
||||
| `--limit` | `-l` | No | `10` | Maximum number of results |
|
||||
| `--properties` | `-p` | No | all | Comma-separated properties to search |
|
||||
| `--target-vector` | `-t` | No | — | Target vector name for named vector collections |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with object properties and score
|
||||
- **JSON**: Array of objects with properties and search metadata
|
||||
|
||||
## Examples
|
||||
|
||||
Basic hybrid search:
|
||||
|
||||
```bash
|
||||
uv run scripts/hybrid_search.py --query "climate change effects" --collection "Articles"
|
||||
```
|
||||
|
||||
Keyword-heavy search (lower alpha):
|
||||
|
||||
```bash
|
||||
uv run scripts/hybrid_search.py --query "product SKU-1234" --collection "Products" --alpha 0.3
|
||||
```
|
||||
|
||||
Search specific properties with named vector:
|
||||
|
||||
```bash
|
||||
uv run scripts/hybrid_search.py --query "renewable energy" --collection "Papers" --properties "title,abstract" --target-vector "title_vector"
|
||||
```
|
||||
|
||||
+160
@@ -0,0 +1,160 @@
|
||||
# Import Data
|
||||
|
||||
Import one or more CSV, JSON, JSONL, or PDF files into a Weaviate collection with automatic type conversion and column mapping. Multiple files of the same format can be passed in a single invocation — all objects are appended to the same collection. PDF files are converted page-by-page to base64-encoded JPEG images; the collection is created automatically on first import and reused on subsequent runs.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# CSV/JSON/JSONL — collection must already exist
|
||||
uv run scripts/import.py "data.csv" --collection "CollectionName" [--mapping '{}'] [--tenant "name"] [--batch-size 100] [--json]
|
||||
|
||||
# Multiple files of the same format
|
||||
uv run scripts/import.py a.csv b.csv c.csv --collection "CollectionName"
|
||||
|
||||
# PDF — collection is created automatically on first run; appended to on subsequent runs
|
||||
uv run scripts/import.py "document.pdf" --collection "CollectionName" [--image-field "doc_page"] [--batch-size 100] [--json]
|
||||
|
||||
# Multiple PDFs into the same collection
|
||||
uv run scripts/import.py page1.pdf page2.pdf page3.pdf --collection "PDFDocuments"
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `files` | — | Yes (positional, one or more) | — | One or more CSV, JSON, JSONL, or PDF files (all must be the same format) |
|
||||
| `--collection` | `-c` | Yes | — | Target collection name (must already exist for CSV/JSON/JSONL; created automatically for PDF if absent, otherwise appended to) |
|
||||
| `--mapping` | `-m` | No | — | JSON object mapping file columns/keys to collection properties (CSV/JSON/JSONL only) |
|
||||
| `--tenant` | `-t` | No | — | Tenant name for multi-tenant collections (required if collection has multi-tenancy enabled) |
|
||||
| `--batch-size` | `-b` | No | `100` | Number of objects per batch |
|
||||
| `--image-field` | `-i` | No | `doc_page` | BLOB property name to store base64 page images (PDF imports only) |
|
||||
| `--skip-fields` | — | No | — | Comma-separated field names to exclude from import (e.g. `vector`) |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## File Formats
|
||||
|
||||
### CSV
|
||||
|
||||
- First row must be a header — column names must match collection property names (case-sensitive)
|
||||
- Delimiter and quoting auto-detected via `csv.Sniffer`
|
||||
- Files without a header row are rejected with a clear error
|
||||
|
||||
### JSON
|
||||
|
||||
- Must be an array of objects: `[{"prop1": "value1"}, {"prop2": "value2"}]`
|
||||
- Keys must match collection property names
|
||||
- The entire file is loaded into memory — for large datasets, always prefer JSONL
|
||||
|
||||
### JSONL
|
||||
|
||||
- One JSON object per line
|
||||
- Each object's keys must match collection property names
|
||||
- Streamed line-by-line — preferred format for large datasets
|
||||
|
||||
### PDF
|
||||
|
||||
- Each page is converted to a JPEG image and base64-encoded
|
||||
- Each page becomes one Weaviate object with these properties:
|
||||
- `doc_page` (or `--image-field` value): base64-encoded JPEG image of the page
|
||||
- `page_number`: 1-indexed page number (int)
|
||||
- `file_name`: PDF filename without extension (text)
|
||||
- The collection is **created automatically** with `multi2vec_weaviate` (`ModernVBERT/colmodernvbert` + MUVERA encoding) if it does not already exist. If the collection already exists, pages are appended to it — allowing multiple PDFs to be loaded into the same collection across multiple runs.
|
||||
- Requires `poppler` to be installed on the system (for Mac, simply run `brew install poppler`)
|
||||
|
||||
## Type Conversion
|
||||
|
||||
For CSV, JSON, and JSONL imports the script uses the collection schema to guide conversion. Non-string values (JSON/JSONL native types) pass through unchanged. String values are cast based on the declared property type:
|
||||
|
||||
| Schema type | Conversion |
|
||||
|---|---|
|
||||
| `int` / `int[]` | `int(value)` — falls back to string if it fails |
|
||||
| `number` / `number[]` | `float(value)` — falls back to string if it fails |
|
||||
| `boolean` / `boolean[]` | `"true"`/`"false"` → bool — falls back to string |
|
||||
| `date` / `date[]` | `"YYYY-MM-DD"` → `"YYYY-MM-DDT00:00:00Z"`, `"YYYY-MM-DD HH:MM:SS"` → RFC3339 with `Z` |
|
||||
| `text[]`, `int[]`, `number[]`, `boolean[]`, `date[]`, `uuid[]`, `object`, `object[]`, `geoCoordinates`, `phoneNumber` | JSON/JSONL: native lists/dicts pass through unchanged. CSV: cell is parsed with `json.loads()` — falls back to string if it fails |
|
||||
| `text`, `uuid` | kept as string |
|
||||
| `blob` | kept as string — must already be base64-encoded in the source data |
|
||||
| field not in schema | kept as string |
|
||||
|
||||
`None` and empty strings are always skipped.
|
||||
|
||||
## Reserved Fields
|
||||
|
||||
`id` and `_additional` are reserved by Weaviate and cannot be used as property names (even for nested properties). If your data contains these keys/columns the import will fail. Use `--skip-fields` to drop them or `--mapping` to rename them.
|
||||
|
||||
**IMPORTANT NOTE:** Renaming must **always** be preferred over dropping when the field contains meaningful data. e.g. renaming `id` to `object_id` or `product_id` (based on the data).
|
||||
|
||||
`--mapping` and `--skip-fields` support dot notation for nested object fields (e.g. `author.id`).
|
||||
|
||||
```bash
|
||||
# Drop the top-level id field entirely
|
||||
uv run scripts/import.py data.json --collection "Articles" --skip-fields "id"
|
||||
|
||||
# Rename top-level id to source_id
|
||||
uv run scripts/import.py data.json --collection "Articles" --mapping '{"id": "source_id"}'
|
||||
|
||||
# Rename a nested id field inside an object property (e.g. author.id → author.author_id)
|
||||
uv run scripts/import.py data.json --collection "Articles" --mapping '{"author.id": "author.author_id"}'
|
||||
|
||||
# Drop a nested id field
|
||||
uv run scripts/import.py data.json --collection "Articles" --skip-fields "author.id"
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Import summary with total, imported, and failed counts (plus sample errors if any)
|
||||
- **JSON**: Structured import stats
|
||||
|
||||
Returns exit code `1` if any imports fail.
|
||||
|
||||
## Examples
|
||||
|
||||
Import from CSV:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py data.csv --collection "Articles"
|
||||
```
|
||||
|
||||
Import with column mapping:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py data.csv --collection "Articles" \
|
||||
--mapping '{"title_col": "title", "body_col": "content"}'
|
||||
```
|
||||
|
||||
Import to multi-tenant collection:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py data.jsonl --collection "Workspace" --tenant "tenant1"
|
||||
```
|
||||
|
||||
Import JSON with custom batch size:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py products.json --collection "Products" --batch-size 500
|
||||
```
|
||||
|
||||
Import a PDF (collection is created automatically on first run):
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py paper.pdf --collection "PDFDocuments"
|
||||
```
|
||||
|
||||
Import multiple PDFs into the same collection:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py chapter1.pdf chapter2.pdf chapter3.pdf --collection "PDFDocuments"
|
||||
```
|
||||
|
||||
Import a PDF with a custom image field name:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py paper.pdf --collection "PDFDocuments" --image-field "page_image"
|
||||
```
|
||||
|
||||
Import multiple CSV files into the same collection:
|
||||
|
||||
```bash
|
||||
uv run scripts/import.py jan.csv feb.csv mar.csv --collection "Articles"
|
||||
```
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
# Keyword Search
|
||||
|
||||
BM25 keyword matching search on a single collection.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/keyword_search.py --query "USER_QUERY" --collection "CollectionName" [--limit 10] [--properties "title^2,content"] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--query` | `-q` | Yes | — | Keyword search query |
|
||||
| `--collection` | `-c` | Yes | — | Collection name |
|
||||
| `--limit` | `-l` | No | `10` | Maximum number of results |
|
||||
| `--properties` | `-p` | No | all | Properties to search with optional boost (e.g., `title^2,content`) |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with object properties and BM25 scores
|
||||
- **JSON**: Array of objects with properties and score metadata
|
||||
|
||||
## Examples
|
||||
|
||||
Basic keyword search:
|
||||
|
||||
```bash
|
||||
uv run scripts/keyword_search.py --query "Python tutorial" --collection "Articles"
|
||||
```
|
||||
|
||||
Search with property boosting:
|
||||
|
||||
```bash
|
||||
uv run scripts/keyword_search.py --query "authentication" --collection "Docs" --properties "title^2,body"
|
||||
```
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# List Collections
|
||||
|
||||
Show all available Weaviate collections with their properties.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/list_collections.py [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with collection names, descriptions, and property lists
|
||||
- **JSON**: Array of collection objects with full property details
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
uv run scripts/list_collections.py
|
||||
```
|
||||
|
||||
```bash
|
||||
uv run scripts/list_collections.py --json
|
||||
```
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
# Query Agent - Search Mode
|
||||
|
||||
Retrieve raw objects using natural language queries across multiple collections via the Weaviate Query Agent.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/query_search.py --query "USER_QUERY" --collections "Collection1,Collection2" [--limit 10] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--query` | `-q` | Yes | — | Natural language search query |
|
||||
| `--collections` | `-c` | Yes | — | Comma-separated collection names to search across |
|
||||
| `--limit` | `-l` | No | `10` | Maximum number of results to return |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with UUIDs, collection names, and all object properties (columns generated dynamically)
|
||||
- **JSON**: Array of objects with `uuid`, `collection`, and `properties`
|
||||
|
||||
## Examples
|
||||
|
||||
Search across collections:
|
||||
|
||||
```bash
|
||||
uv run scripts/query_search.py --query "machine learning papers" --collections "Articles,Research" --limit 5
|
||||
```
|
||||
|
||||
JSON output:
|
||||
|
||||
```bash
|
||||
uv run scripts/query_search.py --query "products under $50" --collections "Products" --json
|
||||
```
|
||||
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
# Semantic Search
|
||||
|
||||
Pure vector similarity search using embeddings on a single collection.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
uv run scripts/semantic_search.py --query "USER_QUERY" --collection "CollectionName" [--limit 10] [--distance 0.5] [--target-vector "vector_name"] [--json]
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Flag | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `--query` | `-q` | Yes | — | Search query text |
|
||||
| `--collection` | `-c` | Yes | — | Collection name |
|
||||
| `--limit` | `-l` | No | `10` | Maximum number of results |
|
||||
| `--distance` | `-d` | No | — | Maximum distance threshold (filters out less similar results) |
|
||||
| `--target-vector` | `-t` | No | — | Target vector name for named vector collections |
|
||||
| `--json` | — | No | `false` | Output in JSON format |
|
||||
|
||||
## Output
|
||||
|
||||
- **Default**: Markdown table with object properties and distance scores
|
||||
- **JSON**: Array of objects with properties and distance metadata
|
||||
|
||||
## Examples
|
||||
|
||||
Basic semantic search:
|
||||
|
||||
```bash
|
||||
uv run scripts/semantic_search.py --query "environmental impact of urbanization" --collection "Research"
|
||||
```
|
||||
|
||||
With distance threshold:
|
||||
|
||||
```bash
|
||||
uv run scripts/semantic_search.py --query "machine learning" --collection "Papers" --distance 0.3 --limit 5
|
||||
```
|
||||
|
||||
With named vector:
|
||||
|
||||
```bash
|
||||
uv run scripts/semantic_search.py --query "abstract art" --collection "Artworks" --target-vector "description_vector"
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user