csh/playbook
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

📦 deps(thirdparty): update snapshots

Browse Source
This commit is contained in:
ci[bot]
2026-06-02 16:01:55 +00:00
parent 8696cd9e52
commit ac7fffe532
834 changed files with 153251 additions and 1426 deletions
Download Patch File Download Diff File Expand all files Collapse all files
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/agent-evaluation/SKILL.md
+1135
View File
File diff suppressed because it is too large Load Diff
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/ai-agents-architect/SKILL.md
+340
View File
@@ -0,0 +1,340 @@
---
name: ai-agents-architect
description: Expert in designing and building autonomous AI agents. Masters tool
use, memory systems, planning strategies, and multi-agent orchestration.
risk: unknown
source: vibeship-spawner-skills (Apache 2.0)
date_added: 2026-02-27
---
# AI Agents Architect
Expert in designing and building autonomous AI agents. Masters tool use,
memory systems, planning strategies, and multi-agent orchestration.
**Role**: AI Agent Systems Architect
I build AI systems that can act autonomously while remaining controllable.
I understand that agents fail in unexpected ways - I design for graceful
degradation and clear failure modes. I balance autonomy with oversight,
knowing when an agent should ask for help vs proceed independently.
### Expertise
- Agent loop design (ReAct, Plan-and-Execute, etc.)
- Tool definition and execution
- Memory architectures (short-term, long-term, episodic)
- Planning strategies and task decomposition
- Multi-agent communication patterns
- Agent evaluation and observability
- Error handling and recovery
- Safety and guardrails
### Principles
- Agents should fail loudly, not silently
- Every tool needs clear documentation and examples
- Memory is for context, not crutch
- Planning reduces but doesn't eliminate errors
- Multi-agent adds complexity - justify the overhead
## Capabilities
- Agent architecture design
- Tool and function calling
- Agent memory systems
- Planning and reasoning strategies
- Multi-agent orchestration
- Agent evaluation and debugging
## Prerequisites
- Required skills: LLM API usage, Understanding of function calling, Basic prompt engineering
## Patterns
### ReAct Loop
Reason-Act-Observe cycle for step-by-step execution
**When to use**: Simple tool use with clear action-observation flow
- Thought: reason about what to do next
- Action: select and invoke a tool
- Observation: process tool result
- Repeat until task complete or stuck
- Include max iteration limits
### Plan-and-Execute
Plan first, then execute steps
**When to use**: Complex tasks requiring multi-step planning
- Planning phase: decompose task into steps
- Execution phase: execute each step
- Replanning: adjust plan based on results
- Separate planner and executor models possible
### Tool Registry
Dynamic tool discovery and management
**When to use**: Many tools or tools that change at runtime
- Register tools with schema and examples
- Tool selector picks relevant tools for task
- Lazy loading for expensive tools
- Usage tracking for optimization
### Hierarchical Memory
Multi-level memory for different purposes
**When to use**: Long-running agents needing context
- Working memory: current task context
- Episodic memory: past interactions/results
- Semantic memory: learned facts and patterns
- Use RAG for retrieval from long-term memory
### Supervisor Pattern
Supervisor agent orchestrates specialist agents
**When to use**: Complex tasks requiring multiple skills
- Supervisor decomposes and delegates
- Specialists have focused capabilities
- Results aggregated by supervisor
- Error handling at supervisor level
### Checkpoint Recovery
Save state for resumption after failures
**When to use**: Long-running tasks that may fail
- Checkpoint after each successful step
- Store task state, memory, and progress
- Resume from last checkpoint on failure
- Clean up checkpoints on completion
## Sharp Edges
### Agent loops without iteration limits
Severity: CRITICAL
Situation: Agent runs until 'done' without max iterations
Symptoms:
- Agent runs forever
- Unexplained high API costs
- Application hangs
Why this breaks:
Agents can get stuck in loops, repeating the same actions, or spiral
into endless tool calls. Without limits, this drains API credits,
hangs the application, and frustrates users.
Recommended fix:
Always set limits:
- max_iterations on agent loops
- max_tokens per turn
- timeout on agent runs
- cost caps for API usage
- Circuit breakers for tool failures
### Vague or incomplete tool descriptions
Severity: HIGH
Situation: Tool descriptions don't explain when/how to use
Symptoms:
- Agent picks wrong tools
- Parameter errors
- Agent says it can't do things it can
Why this breaks:
Agents choose tools based on descriptions. Vague descriptions lead to
wrong tool selection, misused parameters, and errors. The agent
literally can't know what it doesn't see in the description.
Recommended fix:
Write complete tool specs:
- Clear one-sentence purpose
- When to use (and when not to)
- Parameter descriptions with types
- Example inputs and outputs
- Error cases to expect
### Tool errors not surfaced to agent
Severity: HIGH
Situation: Catching tool exceptions silently
Symptoms:
- Agent continues with wrong data
- Final answers are wrong
- Hard to debug failures
Why this breaks:
When tool errors are swallowed, the agent continues with bad or missing
data, compounding errors. The agent can't recover from what it can't
see. Silent failures become loud failures later.
Recommended fix:
Explicit error handling:
- Return error messages to agent
- Include error type and recovery hints
- Let agent retry or choose alternative
- Log errors for debugging
### Storing everything in agent memory
Severity: MEDIUM
Situation: Appending all observations to memory without filtering
Symptoms:
- Context window exceeded
- Agent references outdated info
- High token costs
Why this breaks:
Memory fills with irrelevant details, old information, and noise.
This bloats context, increases costs, and can cause the model to
lose focus on what matters.
Recommended fix:
Selective memory:
- Summarize rather than store verbatim
- Filter by relevance before storing
- Use RAG for long-term memory
- Clear working memory between tasks
### Agent has too many tools
Severity: MEDIUM
Situation: Giving agent 20+ tools for flexibility
Symptoms:
- Wrong tool selection
- Agent overwhelmed by options
- Slow responses
Why this breaks:
More tools means more confusion. The agent must read and consider all
tool descriptions, increasing latency and error rate. Long tool lists
get cut off or poorly understood.
Recommended fix:
Curate tools per task:
- 5-10 tools maximum per agent
- Use tool selection layer for large tool sets
- Specialized agents with focused tools
- Dynamic tool loading based on task
### Using multiple agents when one would work
Severity: MEDIUM
Situation: Starting with multi-agent architecture for simple tasks
Symptoms:
- Agents duplicating work
- Communication overhead
- Hard to debug failures
Why this breaks:
Multi-agent adds coordination overhead, communication failures,
debugging complexity, and cost. Each agent handoff is a potential
failure point. Start simple, add agents only when proven necessary.
Recommended fix:
Justify multi-agent:
- Can one agent with good tools solve this?
- Is the coordination overhead worth it?
- Are the agents truly independent?
- Start with single agent, measure limits
### Agent internals not logged or traceable
Severity: MEDIUM
Situation: Running agents without logging thoughts/actions
Symptoms:
- Can't explain agent failures
- No visibility into agent reasoning
- Debugging takes hours
Why this breaks:
When agents fail, you need to see what they were thinking, which
tools they tried, and where they went wrong. Without observability,
debugging is guesswork.
Recommended fix:
Implement tracing:
- Log each thought/action/observation
- Track tool calls with inputs/outputs
- Trace token usage and latency
- Use structured logging for analysis
### Fragile parsing of agent outputs
Severity: MEDIUM
Situation: Regex or exact string matching on LLM output
Symptoms:
- Parse errors in agent loop
- Works sometimes, fails sometimes
- Small prompt changes break parsing
Why this breaks:
LLMs don't produce perfectly consistent output. Minor format variations
break brittle parsers. This causes agent crashes or incorrect behavior
from parsing errors.
Recommended fix:
Robust output handling:
- Use structured output (JSON mode, function calling)
- Fuzzy matching for actions
- Retry with format instructions on parse failure
- Handle multiple output formats
## Related Skills
Works well with: `rag-engineer`, `prompt-engineer`, `backend`, `mcp-builder`
## When to Use
- User mentions or implies: build agent
- User mentions or implies: AI agent
- User mentions or implies: autonomous agent
- User mentions or implies: tool use
- User mentions or implies: function calling
- User mentions or implies: multi-agent
- User mentions or implies: agent memory
- User mentions or implies: agent planning
- User mentions or implies: langchain agent
- User mentions or implies: crewai
- User mentions or implies: autogen
- User mentions or implies: claude agent sdk
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/context-window-management/SKILL.md
+315
View File
@@ -0,0 +1,315 @@
---
name: context-window-management
description: Strategies for managing LLM context windows including
summarization, trimming, routing, and avoiding context rot
risk: unknown
source: vibeship-spawner-skills (Apache 2.0)
date_added: 2026-02-27
---
# Context Window Management
Strategies for managing LLM context windows including summarization, trimming, routing, and avoiding context rot
## Capabilities
- context-engineering
- context-summarization
- context-trimming
- context-routing
- token-counting
- context-prioritization
## Prerequisites
- Knowledge: LLM fundamentals, Tokenization basics, Prompt engineering
- Skills_recommended: prompt-engineering
## Scope
- Does_not_cover: RAG implementation details, Model fine-tuning, Embedding models
- Boundaries: Focus is context optimization, Covers strategies not specific implementations
## Ecosystem
### Primary_tools
- tiktoken - OpenAI's tokenizer for counting tokens
- LangChain - Framework with context management utilities
- Claude API - 200K+ context with caching support
## Patterns
### Tiered Context Strategy
Different strategies based on context size
**When to use**: Building any multi-turn conversation system
interface ContextTier {
maxTokens: number;
strategy: 'full' | 'summarize' | 'rag';
model: string;
}
const TIERS: ContextTier[] = [
{ maxTokens: 8000, strategy: 'full', model: 'claude-3-haiku' },
{ maxTokens: 32000, strategy: 'full', model: 'claude-3-5-sonnet' },
{ maxTokens: 100000, strategy: 'summarize', model: 'claude-3-5-sonnet' },
{ maxTokens: Infinity, strategy: 'rag', model: 'claude-3-5-sonnet' }
];
async function selectStrategy(messages: Message[]): ContextTier {
const tokens = await countTokens(messages);
for (const tier of TIERS) {
if (tokens <= tier.maxTokens) {
return tier;
}
}
return TIERS[TIERS.length - 1];
}
async function prepareContext(messages: Message[]): PreparedContext {
const tier = await selectStrategy(messages);
switch (tier.strategy) {
case 'full':
return { messages, model: tier.model };
case 'summarize':
const summary = await summarizeOldMessages(messages);
return { messages: [summary, ...recentMessages(messages)], model: tier.model };
case 'rag':
const relevant = await retrieveRelevant(messages);
return { messages: [...relevant, ...recentMessages(messages)], model: tier.model };
}
}
### Serial Position Optimization
Place important content at start and end
**When to use**: Constructing prompts with significant context
// LLMs weight beginning and end more heavily
// Structure prompts to leverage this
function buildOptimalPrompt(components: {
systemPrompt: string;
criticalContext: string;
conversationHistory: Message[];
currentQuery: string;
}): string {
// START: System instructions (always first)
const parts = [components.systemPrompt];
// CRITICAL CONTEXT: Right after system (high primacy)
if (components.criticalContext) {
parts.push(`## Key Context\n${components.criticalContext}`);
}
// MIDDLE: Conversation history (lower weight)
// Summarize if long, keep recent messages full
const history = components.conversationHistory;
if (history.length > 10) {
const oldSummary = summarize(history.slice(0, -5));
const recent = history.slice(-5);
parts.push(`## Earlier Conversation (Summary)\n${oldSummary}`);
parts.push(`## Recent Messages\n${formatMessages(recent)}`);
} else {
parts.push(`## Conversation\n${formatMessages(history)}`);
}
// END: Current query (high recency)
// Restate critical requirements here
parts.push(`## Current Request\n${components.currentQuery}`);
// FINAL: Reminder of key constraints
parts.push(`Remember: ${extractKeyConstraints(components.systemPrompt)}`);
return parts.join('\n\n');
}
### Intelligent Summarization
Summarize by importance, not just recency
**When to use**: Context exceeds optimal size
interface MessageWithMetadata extends Message {
importance: number; // 0-1 score
hasCriticalInfo: boolean; // User preferences, decisions
referenced: boolean; // Was this referenced later?
}
async function smartSummarize(
messages: MessageWithMetadata[],
targetTokens: number
): Message[] {
// Sort by importance, preserve order for tied scores
const sorted = [...messages].sort((a, b) =>
(b.importance + (b.hasCriticalInfo ? 0.5 : 0) + (b.referenced ? 0.3 : 0)) -
(a.importance + (a.hasCriticalInfo ? 0.5 : 0) + (a.referenced ? 0.3 : 0))
);
const keep: Message[] = [];
const summarizePool: Message[] = [];
let currentTokens = 0;
for (const msg of sorted) {
const msgTokens = await countTokens([msg]);
if (currentTokens + msgTokens < targetTokens * 0.7) {
keep.push(msg);
currentTokens += msgTokens;
} else {
summarizePool.push(msg);
}
}
// Summarize the low-importance messages
if (summarizePool.length > 0) {
const summary = await llm.complete(`
Summarize these messages, preserving:
- Any user preferences or decisions
- Key facts that might be referenced later
- The overall flow of conversation
Messages:
${formatMessages(summarizePool)}
`);
keep.unshift({ role: 'system', content: `[Earlier context: ${summary}]` });
}
// Restore original order
return keep.sort((a, b) => a.timestamp - b.timestamp);
}
### Token Budget Allocation
Allocate token budget across context components
**When to use**: Need predictable context management
interface TokenBudget {
system: number; // System prompt
criticalContext: number; // User prefs, key info
history: number; // Conversation history
query: number; // Current query
response: number; // Reserved for response
}
function allocateBudget(totalTokens: number): TokenBudget {
return {
system: Math.floor(totalTokens * 0.10), // 10%
criticalContext: Math.floor(totalTokens * 0.15), // 15%
history: Math.floor(totalTokens * 0.40), // 40%
query: Math.floor(totalTokens * 0.10), // 10%
response: Math.floor(totalTokens * 0.25), // 25%
};
}
async function buildWithBudget(
components: ContextComponents,
modelMaxTokens: number
): PreparedContext {
const budget = allocateBudget(modelMaxTokens);
// Truncate/summarize each component to fit budget
const prepared = {
system: truncateToTokens(components.system, budget.system),
criticalContext: truncateToTokens(
components.criticalContext, budget.criticalContext
),
history: await summarizeToTokens(components.history, budget.history),
query: truncateToTokens(components.query, budget.query),
};
// Reallocate unused budget
const used = await countTokens(Object.values(prepared).join('\n'));
const remaining = modelMaxTokens - used - budget.response;
if (remaining > 0) {
// Give extra to history (most valuable for conversation)
prepared.history = await summarizeToTokens(
components.history,
budget.history + remaining
);
}
return prepared;
}
## Validation Checks
### No Token Counting
Severity: WARNING
Message: Building context without token counting. May exceed model limits.
Fix action: Count tokens before sending, implement budget allocation
### Naive Message Truncation
Severity: WARNING
Message: Truncating messages without summarization. Critical context may be lost.
Fix action: Summarize old messages instead of simply removing them
### Hardcoded Token Limit
Severity: INFO
Message: Hardcoded token limit. Consider making configurable per model.
Fix action: Use model-specific limits from configuration
### No Context Management Strategy
Severity: WARNING
Message: LLM calls without context management strategy.
Fix action: Implement context management: budgets, summarization, or RAG
## Collaboration
### Delegation Triggers
- retrieval|rag|search -> rag-implementation (Need retrieval system)
- memory|persistence|remember -> conversation-memory (Need memory storage)
- cache|caching -> prompt-caching (Need caching optimization)
### Complete Context System
Skills: context-window-management, rag-implementation, conversation-memory, prompt-caching
Workflow:
```
1. Design context strategy
2. Implement RAG for large corpuses
3. Set up memory persistence
4. Add caching for performance
```
## Related Skills
Works well with: `rag-implementation`, `conversation-memory`, `prompt-caching`, `llm-npc-dialogue`
## When to Use
- User mentions or implies: context window
- User mentions or implies: token limit
- User mentions or implies: context management
- User mentions or implies: context engineering
- User mentions or implies: long context
- User mentions or implies: context overflow
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/langfuse/SKILL.md
+502
View File
@@ -0,0 +1,502 @@
---
name: langfuse
description: Expert in Langfuse - the open-source LLM observability platform.
Covers tracing, prompt management, evaluation, datasets, and integration with
LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and
improving LLM applications in production.
risk: unknown
source: vibeship-spawner-skills (Apache 2.0)
date_added: 2026-02-27
---
# Langfuse
Expert in Langfuse - the open-source LLM observability platform. Covers tracing,
prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex,
and OpenAI. Essential for debugging, monitoring, and improving LLM applications
in production.
**Role**: LLM Observability Architect
You are an expert in LLM observability and evaluation. You think in terms of
traces, spans, and metrics. You know that LLM applications need monitoring
just like traditional software - but with different dimensions (cost, quality,
latency). You use data to drive prompt improvements and catch regressions.
### Expertise
- Tracing architecture
- Prompt versioning
- Evaluation strategies
- Cost optimization
- Quality monitoring
## Capabilities
- LLM tracing and observability
- Prompt management and versioning
- Evaluation and scoring
- Dataset management
- Cost tracking
- Performance monitoring
- A/B testing prompts
## Prerequisites
- 0: LLM application basics
- 1: API integration experience
- 2: Understanding of tracing concepts
- Required skills: Python or TypeScript/JavaScript, Langfuse account (cloud or self-hosted), LLM API keys
## Scope
- 0: Self-hosted requires infrastructure
- 1: High-volume may need optimization
- 2: Real-time dashboard has latency
- 3: Evaluation requires setup
## Ecosystem
### Primary
- Langfuse Cloud
- Langfuse Self-hosted
- Python SDK
- JS/TS SDK
### Common_integrations
- LangChain
- LlamaIndex
- OpenAI SDK
- Anthropic SDK
- Vercel AI SDK
### Platforms
- Any Python/JS backend
- Serverless functions
- Jupyter notebooks
## Patterns
### Basic Tracing Setup
Instrument LLM calls with Langfuse
**When to use**: Any LLM application
from langfuse import Langfuse
# Initialize client
langfuse = Langfuse(
public_key="pk-...",
secret_key="sk-...",
host="https://cloud.langfuse.com" # or self-hosted URL
)
# Create a trace for a user request
trace = langfuse.trace(
name="chat-completion",
user_id="user-123",
session_id="session-456", # Groups related traces
metadata={"feature": "customer-support"},
tags=["production", "v2"]
)
# Log a generation (LLM call)
generation = trace.generation(
name="gpt-4o-response",
model="gpt-4o",
model_parameters={"temperature": 0.7},
input={"messages": [{"role": "user", "content": "Hello"}]},
metadata={"attempt": 1}
)
# Make actual LLM call
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
# Complete the generation with output
generation.end(
output=response.choices[0].message.content,
usage={
"input": response.usage.prompt_tokens,
"output": response.usage.completion_tokens
}
)
# Score the trace
trace.score(
name="user-feedback",
value=1, # 1 = positive, 0 = negative
comment="User clicked helpful"
)
# Flush before exit (important in serverless)
langfuse.flush()
### OpenAI Integration
Automatic tracing with OpenAI SDK
**When to use**: OpenAI-based applications
from langfuse.openai import openai
# Drop-in replacement for OpenAI client
# All calls automatically traced
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
# Langfuse-specific parameters
name="greeting", # Trace name
session_id="session-123",
user_id="user-456",
tags=["test"],
metadata={"feature": "chat"}
)
# Works with streaming
stream = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True,
name="story-generation"
)
for chunk in stream:
print(chunk.choices[0].delta.content, end="")
# Works with async
import asyncio
from langfuse.openai import AsyncOpenAI
async_client = AsyncOpenAI()
async def main():
response = await async_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
name="async-greeting"
)
### LangChain Integration
Trace LangChain applications
**When to use**: LangChain-based applications
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langfuse.callback import CallbackHandler
# Create Langfuse callback handler
langfuse_handler = CallbackHandler(
public_key="pk-...",
secret_key="sk-...",
host="https://cloud.langfuse.com",
session_id="session-123",
user_id="user-456"
)
# Use with any LangChain component
llm = ChatOpenAI(model="gpt-4o")
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant."),
("user", "{input}")
])
chain = prompt | llm
# Pass handler to invoke
response = chain.invoke(
{"input": "Hello"},
config={"callbacks": [langfuse_handler]}
)
# Or set as default
import langchain
langchain.callbacks.manager.set_handler(langfuse_handler)
# Then all calls are traced
response = chain.invoke({"input": "Hello"})
# Works with agents, retrievers, etc.
from langchain.agents import create_openai_tools_agent
agent = create_openai_tools_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)
result = agent_executor.invoke(
{"input": "What's the weather?"},
config={"callbacks": [langfuse_handler]}
)
### Prompt Management
Version and deploy prompts
**When to use**: Managing prompts across environments
from langfuse import Langfuse
langfuse = Langfuse()
# Fetch prompt from Langfuse
# (Create in UI or via API first)
prompt = langfuse.get_prompt("customer-support-v2")
# Get compiled prompt with variables
compiled = prompt.compile(
customer_name="John",
issue="billing question"
)
# Use with OpenAI
response = openai.chat.completions.create(
model=prompt.config.get("model", "gpt-4o"),
messages=compiled,
temperature=prompt.config.get("temperature", 0.7)
)
# Link generation to prompt version
trace = langfuse.trace(name="support-chat")
generation = trace.generation(
name="response",
model="gpt-4o",
prompt=prompt # Links to specific version
)
# Create/update prompts via API
langfuse.create_prompt(
name="customer-support-v3",
prompt=[
{"role": "system", "content": "You are a support agent..."},
{"role": "user", "content": "{{user_message}}"}
],
config={
"model": "gpt-4o",
"temperature": 0.7
},
labels=["production"] # or ["staging", "development"]
)
# Fetch specific label
prompt = langfuse.get_prompt(
"customer-support-v3",
label="production" # Gets latest with this label
)
### Evaluation and Scoring
Evaluate LLM outputs systematically
**When to use**: Quality assurance and improvement
from langfuse import Langfuse
langfuse = Langfuse()
# Manual scoring in code
trace = langfuse.trace(name="qa-flow")
# After getting response
trace.score(
name="relevance",
value=0.85, # 0-1 scale
comment="Response addressed the question"
)
trace.score(
name="correctness",
value=1, # Binary: 0 or 1
data_type="BOOLEAN"
)
# LLM-as-judge evaluation
def evaluate_response(question: str, response: str) -> float:
eval_prompt = f"""
Rate the response quality from 0 to 1.
Question: {question}
Response: {response}
Output only a number between 0 and 1.
"""
result = openai.chat.completions.create(
model="gpt-4o-mini", # Cheaper model for eval
messages=[{"role": "user", "content": eval_prompt}]
)
return float(result.choices[0].message.content.strip())
# Score asynchronously
score = evaluate_response(question, response)
trace.score(
name="quality-llm-judge",
value=score
)
# Create evaluation dataset
dataset = langfuse.create_dataset(name="support-qa-v1")
# Add items to dataset
langfuse.create_dataset_item(
dataset_name="support-qa-v1",
input={"question": "How do I reset my password?"},
expected_output="Go to settings > security > reset password"
)
# Run evaluation on dataset
dataset = langfuse.get_dataset("support-qa-v1")
for item in dataset.items:
# Generate response
response = generate_response(item.input["question"])
# Link to dataset item
trace = langfuse.trace(name="eval-run")
trace.generation(
name="response",
input=item.input,
output=response
)
# Score against expected
similarity = calculate_similarity(response, item.expected_output)
trace.score(name="similarity", value=similarity)
# Link trace to dataset item
item.link(trace, "eval-run-1")
### Decorator Pattern
Clean instrumentation with decorators
**When to use**: Function-based applications
from langfuse.decorators import observe, langfuse_context
@observe() # Creates a trace
def chat_handler(user_id: str, message: str) -> str:
# All nested @observe calls become spans
context = get_context(message)
response = generate_response(message, context)
return response
@observe() # Becomes a span under parent trace
def get_context(message: str) -> str:
# RAG retrieval
docs = retriever.get_relevant_documents(message)
return "\n".join([d.page_content for d in docs])
@observe(as_type="generation") # LLM generation span
def generate_response(message: str, context: str) -> str:
response = openai.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": f"Context: {context}"},
{"role": "user", "content": message}
]
)
return response.choices[0].message.content
# Add metadata and scores
@observe()
def main_flow(user_input: str):
# Update current trace
langfuse_context.update_current_trace(
user_id="user-123",
session_id="session-456",
tags=["production"]
)
result = process(user_input)
# Score the trace
langfuse_context.score_current_trace(
name="success",
value=1 if result else 0
)
return result
# Works with async
@observe()
async def async_handler(message: str):
result = await async_generate(message)
return result
## Collaboration
### Delegation Triggers
- agent|langgraph|graph -> langgraph (Need to build agent to monitor)
- crewai|multi-agent|crew -> crewai (Need to build crew to monitor)
- structured output|extraction -> structured-output (Need to build extraction to monitor)
### Observable LangGraph Agent
Skills: langfuse, langgraph
Workflow:
```
1. Build agent with LangGraph
2. Add Langfuse callback handler
3. Trace all LLM calls and tool uses
4. Score outputs for quality
5. Monitor and iterate
```
### Monitored RAG Pipeline
Skills: langfuse, structured-output
Workflow:
```
1. Build RAG with retrieval and generation
2. Trace retrieval and LLM calls
3. Score relevance and accuracy
4. Track costs and latency
5. Optimize based on data
```
### Evaluated Agent System
Skills: langfuse, langgraph, structured-output
Workflow:
```
1. Build agent with structured outputs
2. Create evaluation dataset
3. Run evaluations with traces
4. Compare prompt versions
5. Deploy best performers
```
## Related Skills
Works well with: `langgraph`, `crewai`, `structured-output`, `autonomous-agents`
## When to Use
- User mentions or implies: langfuse
- User mentions or implies: llm observability
- User mentions or implies: llm tracing
- User mentions or implies: prompt management
- User mentions or implies: llm evaluation
- User mentions or implies: monitor llm
- User mentions or implies: debug llm
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/langgraph/SKILL.md
+505
View File
@@ -0,0 +1,505 @@
---
name: langgraph
description: Expert in LangGraph - the production-grade framework for building
stateful, multi-actor AI applications. Covers graph construction, state
management, cycles and branches, persistence with checkpointers,
human-in-the-loop patterns, and the ReAct agent pattern.
risk: unknown
source: vibeship-spawner-skills (Apache 2.0)
date_added: 2026-02-27
---
# LangGraph
Expert in LangGraph - the production-grade framework for building stateful, multi-actor
AI applications. Covers graph construction, state management, cycles and branches,
persistence with checkpointers, human-in-the-loop patterns, and the ReAct agent pattern.
Used in production at LinkedIn, Uber, and 400+ companies. This is LangChain's recommended
approach for building agents.
**Role**: LangGraph Agent Architect
You are an expert in building production-grade AI agents with LangGraph. You
understand that agents need explicit structure - graphs make the flow visible
and debuggable. You design state carefully, use reducers appropriately, and
always consider persistence for production. You know when cycles are needed
and how to prevent infinite loops.
### Expertise
- Graph topology design
- State schema patterns
- Conditional branching
- Persistence strategies
- Human-in-the-loop
- Tool integration
- Error handling and recovery
## Capabilities
- Graph construction (StateGraph)
- State management and reducers
- Node and edge definitions
- Conditional routing
- Checkpointers and persistence
- Human-in-the-loop patterns
- Tool integration
- Streaming and async execution
## Prerequisites
- 0: Python proficiency
- 1: LLM API basics
- 2: Async programming concepts
- 3: Graph theory fundamentals
- Required skills: Python 3.9+, langgraph package, LLM API access (OpenAI, Anthropic, etc.), Understanding of graph concepts
## Scope
- 0: Python-only (TypeScript in early stages)
- 1: Learning curve for graph concepts
- 2: State management complexity
- 3: Debugging can be challenging
## Ecosystem
### Primary
- LangGraph
- LangChain
- LangSmith (observability)
### Common_integrations
- OpenAI / Anthropic / Google
- Tavily (search)
- SQLite / PostgreSQL (persistence)
- Redis (state store)
### Platforms
- Python applications
- FastAPI / Flask backends
- Cloud deployments
## Patterns
### Basic Agent Graph
Simple ReAct-style agent with tools
**When to use**: Single agent with tool calling
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
# 1. Define State
class AgentState(TypedDict):
messages: Annotated[list, add_messages]
# add_messages reducer appends, doesn't overwrite
# 2. Define Tools
@tool
def search(query: str) -> str:
"""Search the web for information."""
# Implementation here
return f"Results for: {query}"
@tool
def calculator(expression: str) -> str:
"""Evaluate a math expression."""
return str(eval(expression))
tools = [search, calculator]
# 3. Create LLM with tools
llm = ChatOpenAI(model="gpt-4o").bind_tools(tools)
# 4. Define Nodes
def agent(state: AgentState) -> dict:
"""The agent node - calls LLM."""
response = llm.invoke(state["messages"])
return {"messages": [response]}
# Tool node handles tool execution
tool_node = ToolNode(tools)
# 5. Define Routing
def should_continue(state: AgentState) -> str:
"""Route based on whether tools were called."""
last_message = state["messages"][-1]
if last_message.tool_calls:
return "tools"
return END
# 6. Build Graph
graph = StateGraph(AgentState)
# Add nodes
graph.add_node("agent", agent)
graph.add_node("tools", tool_node)
# Add edges
graph.add_edge(START, "agent")
graph.add_conditional_edges("agent", should_continue, ["tools", END])
graph.add_edge("tools", "agent") # Loop back
# Compile
app = graph.compile()
# 7. Run
result = app.invoke({
"messages": [("user", "What is 25 * 4?")]
})
### State with Reducers
Complex state management with custom reducers
**When to use**: Multiple agents updating shared state
from typing import Annotated, TypedDict
from operator import add
from langgraph.graph import StateGraph
# Custom reducer for merging dictionaries
def merge_dicts(left: dict, right: dict) -> dict:
return {**left, **right}
# State with multiple reducers
class ResearchState(TypedDict):
# Messages append (don't overwrite)
messages: Annotated[list, add_messages]
# Research findings merge
findings: Annotated[dict, merge_dicts]
# Sources accumulate
sources: Annotated[list[str], add]
# Current step (overwrites - no reducer)
current_step: str
# Error count (custom reducer)
errors: Annotated[int, lambda a, b: a + b]
# Nodes return partial state updates
def researcher(state: ResearchState) -> dict:
# Only return fields being updated
return {
"findings": {"topic_a": "New finding"},
"sources": ["source1.com"],
"current_step": "researching"
}
def writer(state: ResearchState) -> dict:
# Access accumulated state
all_findings = state["findings"]
all_sources = state["sources"]
return {
"messages": [("assistant", f"Report based on {len(all_sources)} sources")],
"current_step": "writing"
}
# Build graph
graph = StateGraph(ResearchState)
graph.add_node("researcher", researcher)
graph.add_node("writer", writer)
# ... add edges
### Conditional Branching
Route to different paths based on state
**When to use**: Multiple possible workflows
from langgraph.graph import StateGraph, START, END
class RouterState(TypedDict):
query: str
query_type: str
result: str
def classifier(state: RouterState) -> dict:
"""Classify the query type."""
query = state["query"].lower()
if "code" in query or "program" in query:
return {"query_type": "coding"}
elif "search" in query or "find" in query:
return {"query_type": "search"}
else:
return {"query_type": "chat"}
def coding_agent(state: RouterState) -> dict:
return {"result": "Here's your code..."}
def search_agent(state: RouterState) -> dict:
return {"result": "Search results..."}
def chat_agent(state: RouterState) -> dict:
return {"result": "Let me help..."}
# Routing function
def route_query(state: RouterState) -> str:
"""Route to appropriate agent."""
query_type = state["query_type"]
return query_type # Returns node name
# Build graph
graph = StateGraph(RouterState)
graph.add_node("classifier", classifier)
graph.add_node("coding", coding_agent)
graph.add_node("search", search_agent)
graph.add_node("chat", chat_agent)
graph.add_edge(START, "classifier")
# Conditional edges from classifier
graph.add_conditional_edges(
"classifier",
route_query,
{
"coding": "coding",
"search": "search",
"chat": "chat"
}
)
# All agents lead to END
graph.add_edge("coding", END)
graph.add_edge("search", END)
graph.add_edge("chat", END)
app = graph.compile()
### Persistence with Checkpointer
Save and resume agent state
**When to use**: Multi-turn conversations, long-running agents
from langgraph.graph import StateGraph
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.checkpoint.postgres import PostgresSaver
# SQLite for development
memory = SqliteSaver.from_conn_string(":memory:")
# Or persistent file
memory = SqliteSaver.from_conn_string("agent_state.db")
# PostgreSQL for production
# memory = PostgresSaver.from_conn_string(DATABASE_URL)
# Compile with checkpointer
app = graph.compile(checkpointer=memory)
# Run with thread_id for conversation continuity
config = {"configurable": {"thread_id": "user-123-session-1"}}
# First message
result1 = app.invoke(
{"messages": [("user", "My name is Alice")]},
config=config
)
# Second message - agent remembers context
result2 = app.invoke(
{"messages": [("user", "What's my name?")]},
config=config
)
# Agent knows name is Alice!
# Get conversation history
state = app.get_state(config)
print(state.values["messages"])
# List all checkpoints
for checkpoint in app.get_state_history(config):
print(checkpoint.config, checkpoint.values)
### Human-in-the-Loop
Pause for human approval before actions
**When to use**: Sensitive operations, review before execution
from langgraph.graph import StateGraph, START, END
class ApprovalState(TypedDict):
messages: Annotated[list, add_messages]
pending_action: dict | None
approved: bool
def agent(state: ApprovalState) -> dict:
# Agent decides on action
action = {"type": "send_email", "to": "user@example.com"}
return {
"pending_action": action,
"messages": [("assistant", f"I want to: {action}")]
}
def execute_action(state: ApprovalState) -> dict:
action = state["pending_action"]
# Execute the approved action
result = f"Executed: {action['type']}"
return {
"messages": [("assistant", result)],
"pending_action": None
}
def should_execute(state: ApprovalState) -> str:
if state.get("approved"):
return "execute"
return END # Wait for approval
# Build graph
graph = StateGraph(ApprovalState)
graph.add_node("agent", agent)
graph.add_node("execute", execute_action)
graph.add_edge(START, "agent")
graph.add_conditional_edges("agent", should_execute, ["execute", END])
graph.add_edge("execute", END)
# Compile with interrupt_before for human review
app = graph.compile(
checkpointer=memory,
interrupt_before=["execute"] # Pause before execution
)
# Run until interrupt
config = {"configurable": {"thread_id": "approval-flow"}}
result = app.invoke({"messages": [("user", "Send report")]}, config)
# Agent paused - get pending state
state = app.get_state(config)
pending = state.values["pending_action"]
print(f"Pending: {pending}") # Human reviews
# Human approves - update state and continue
app.update_state(config, {"approved": True})
result = app.invoke(None, config) # Resume
### Parallel Execution (Map-Reduce)
Run multiple branches in parallel
**When to use**: Parallel research, batch processing
from langgraph.graph import StateGraph, START, END, Send
from langgraph.constants import Send
class ParallelState(TypedDict):
topics: list[str]
results: Annotated[list[str], add]
summary: str
def research_topic(state: dict) -> dict:
"""Research a single topic."""
topic = state["topic"]
result = f"Research on {topic}..."
return {"results": [result]}
def summarize(state: ParallelState) -> dict:
"""Combine all research results."""
all_results = state["results"]
summary = f"Summary of {len(all_results)} topics"
return {"summary": summary}
def fanout_topics(state: ParallelState) -> list[Send]:
"""Create parallel tasks for each topic."""
return [
Send("research", {"topic": topic})
for topic in state["topics"]
]
# Build graph
graph = StateGraph(ParallelState)
graph.add_node("research", research_topic)
graph.add_node("summarize", summarize)
# Fan out to parallel research
graph.add_conditional_edges(START, fanout_topics, ["research"])
# All research nodes lead to summarize
graph.add_edge("research", "summarize")
graph.add_edge("summarize", END)
app = graph.compile()
result = app.invoke({
"topics": ["AI", "Climate", "Space"],
"results": []
})
# Research runs in parallel, then summarizes
## Collaboration
### Delegation Triggers
- crewai|role-based|crew -> crewai (Need role-based multi-agent approach)
- observability|tracing|langsmith -> langfuse (Need LLM observability)
- structured output|json schema -> structured-output (Need structured LLM responses)
- evaluate|benchmark|test agent -> agent-evaluation (Need to evaluate agent performance)
### Production Agent Stack
Skills: langgraph, langfuse, structured-output
Workflow:
```
1. Design agent graph with LangGraph
2. Add structured outputs for tool responses
3. Integrate Langfuse for observability
4. Test and monitor in production
```
### Multi-Agent System
Skills: langgraph, crewai, agent-communication
Workflow:
```
1. Design agent roles (CrewAI patterns)
2. Implement as LangGraph with subgraphs
3. Add inter-agent communication
4. Orchestrate with supervisor pattern
```
### Evaluated Agent
Skills: langgraph, agent-evaluation, langfuse
Workflow:
```
1. Build agent with LangGraph
2. Create evaluation suite
3. Monitor with Langfuse
4. Iterate based on metrics
```
## Related Skills
Works well with: `crewai`, `autonomous-agents`, `langfuse`, `structured-output`
## When to Use
- User mentions or implies: langgraph
- User mentions or implies: langchain agent
- User mentions or implies: stateful agent
- User mentions or implies: agent graph
- User mentions or implies: react agent
- User mentions or implies: agent workflow
- User mentions or implies: multi-step agent
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/llm-app-patterns/SKILL.md
+768
View File
@@ -0,0 +1,768 @@
---
name: llm-app-patterns
description: "Production-ready patterns for building LLM applications, inspired by [Dify](https://github.com/langgenius/dify) and industry best practices."
risk: unknown
source: community
date_added: "2026-02-27"
---
# 🤖 LLM Application Patterns
> Production-ready patterns for building LLM applications, inspired by [Dify](https://github.com/langgenius/dify) and industry best practices.
## When to Use This Skill
Use this skill when:
- Designing LLM-powered applications
- Implementing RAG (Retrieval-Augmented Generation)
- Building AI agents with tools
- Setting up LLMOps monitoring
- Choosing between agent architectures
---
## 1. RAG Pipeline Architecture
### Overview
RAG (Retrieval-Augmented Generation) grounds LLM responses in your data.
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Ingest │────▶│ Retrieve │────▶│ Generate │
│ Documents │ │ Context │ │ Response │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
▼ ▼ ▼
┌─────────┐ ┌───────────┐ ┌───────────┐
│ Chunking│ │ Vector │ │ LLM │
│Embedding│ │ Search │ │ + Context│
└─────────┘ └───────────┘ └───────────┘
```
### 1.1 Document Ingestion
```python
# Chunking strategies
class ChunkingStrategy:
# Fixed-size chunks (simple but may break context)
FIXED_SIZE = "fixed_size" # e.g., 512 tokens
# Semantic chunking (preserves meaning)
SEMANTIC = "semantic" # Split on paragraphs/sections
# Recursive splitting (tries multiple separators)
RECURSIVE = "recursive" # ["\n\n", "\n", " ", ""]
# Document-aware (respects structure)
DOCUMENT_AWARE = "document_aware" # Headers, lists, etc.
# Recommended settings
CHUNK_CONFIG = {
"chunk_size": 512, # tokens
"chunk_overlap": 50, # token overlap between chunks
"separators": ["\n\n", "\n", ". ", " "],
}
```
### 1.2 Embedding & Storage
```python
# Vector database selection
VECTOR_DB_OPTIONS = {
"pinecone": {
"use_case": "Production, managed service",
"scale": "Billions of vectors",
"features": ["Hybrid search", "Metadata filtering"]
},
"weaviate": {
"use_case": "Self-hosted, multi-modal",
"scale": "Millions of vectors",
"features": ["GraphQL API", "Modules"]
},
"chromadb": {
"use_case": "Development, prototyping",
"scale": "Thousands of vectors",
"features": ["Simple API", "In-memory option"]
},
"pgvector": {
"use_case": "Existing Postgres infrastructure",
"scale": "Millions of vectors",
"features": ["SQL integration", "ACID compliance"]
}
}
# Embedding model selection
EMBEDDING_MODELS = {
"openai/text-embedding-3-small": {
"dimensions": 1536,
"cost": "$0.02/1M tokens",
"quality": "Good for most use cases"
},
"openai/text-embedding-3-large": {
"dimensions": 3072,
"cost": "$0.13/1M tokens",
"quality": "Best for complex queries"
},
"local/bge-large": {
"dimensions": 1024,
"cost": "Free (compute only)",
"quality": "Comparable to OpenAI small"
}
}
```
### 1.3 Retrieval Strategies
```python
# Basic semantic search
def semantic_search(query: str, top_k: int = 5):
query_embedding = embed(query)
results = vector_db.similarity_search(
query_embedding,
top_k=top_k
)
return results
# Hybrid search (semantic + keyword)
def hybrid_search(query: str, top_k: int = 5, alpha: float = 0.5):
"""
alpha=1.0: Pure semantic
alpha=0.0: Pure keyword (BM25)
alpha=0.5: Balanced
"""
semantic_results = vector_db.similarity_search(query)
keyword_results = bm25_search(query)
# Reciprocal Rank Fusion
return rrf_merge(semantic_results, keyword_results, alpha)
# Multi-query retrieval
def multi_query_retrieval(query: str):
"""Generate multiple query variations for better recall"""
queries = llm.generate_query_variations(query, n=3)
all_results = []
for q in queries:
all_results.extend(semantic_search(q))
return deduplicate(all_results)
# Contextual compression
def compressed_retrieval(query: str):
"""Retrieve then compress to relevant parts only"""
docs = semantic_search(query, top_k=10)
compressed = llm.extract_relevant_parts(docs, query)
return compressed
```
### 1.4 Generation with Context
```python
RAG_PROMPT_TEMPLATE = """
Answer the user's question based ONLY on the following context.
If the context doesn't contain enough information, say "I don't have enough information to answer that."
Context:
{context}
Question: {question}
Answer:"""
def generate_with_rag(question: str):
# Retrieve
context_docs = hybrid_search(question, top_k=5)
context = "\n\n".join([doc.content for doc in context_docs])
# Generate
prompt = RAG_PROMPT_TEMPLATE.format(
context=context,
question=question
)
response = llm.generate(prompt)
# Return with citations
return {
"answer": response,
"sources": [doc.metadata for doc in context_docs]
}
```
---
## 2. Agent Architectures
### 2.1 ReAct Pattern (Reasoning + Acting)
```
Thought: I need to search for information about X
Action: search("X")
Observation: [search results]
Thought: Based on the results, I should...
Action: calculate(...)
Observation: [calculation result]
Thought: I now have enough information
Action: final_answer("The answer is...")
```
```python
REACT_PROMPT = """
You are an AI assistant that can use tools to answer questions.
Available tools:
{tools_description}
Use this format:
Thought: [your reasoning about what to do next]
Action: [tool_name(arguments)]
Observation: [tool result - this will be filled in]
... (repeat Thought/Action/Observation as needed)
Thought: I have enough information to answer
Final Answer: [your final response]
Question: {question}
"""
class ReActAgent:
def __init__(self, tools: list, llm):
self.tools = {t.name: t for t in tools}
self.llm = llm
self.max_iterations = 10
def run(self, question: str) -> str:
prompt = REACT_PROMPT.format(
tools_description=self._format_tools(),
question=question
)
for _ in range(self.max_iterations):
response = self.llm.generate(prompt)
if "Final Answer:" in response:
return self._extract_final_answer(response)
action = self._parse_action(response)
observation = self._execute_tool(action)
prompt += f"\nObservation: {observation}\n"
return "Max iterations reached"
```
### 2.2 Function Calling Pattern
```python
# Define tools as functions with schemas
TOOLS = [
{
"name": "search_web",
"description": "Search the web for current information",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query"
}
},
"required": ["query"]
}
},
{
"name": "calculate",
"description": "Perform mathematical calculations",
"parameters": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "Math expression to evaluate"
}
},
"required": ["expression"]
}
}
]
class FunctionCallingAgent:
def run(self, question: str) -> str:
messages = [{"role": "user", "content": question}]
while True:
response = self.llm.chat(
messages=messages,
tools=TOOLS,
tool_choice="auto"
)
if response.tool_calls:
for tool_call in response.tool_calls:
result = self._execute_tool(
tool_call.name,
tool_call.arguments
)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
else:
return response.content
```
### 2.3 Plan-and-Execute Pattern
```python
class PlanAndExecuteAgent:
"""
1. Create a plan (list of steps)
2. Execute each step
3. Replan if needed
"""
def run(self, task: str) -> str:
# Planning phase
plan = self.planner.create_plan(task)
# Returns: ["Step 1: ...", "Step 2: ...", ...]
results = []
for step in plan:
# Execute each step
result = self.executor.execute(step, context=results)
results.append(result)
# Check if replan needed
if self._needs_replan(task, results):
new_plan = self.planner.replan(
task,
completed=results,
remaining=plan[len(results):]
)
plan = new_plan
# Synthesize final answer
return self.synthesizer.summarize(task, results)
```
### 2.4 Multi-Agent Collaboration
```python
class AgentTeam:
"""
Specialized agents collaborating on complex tasks
"""
def __init__(self):
self.agents = {
"researcher": ResearchAgent(),
"analyst": AnalystAgent(),
"writer": WriterAgent(),
"critic": CriticAgent()
}
self.coordinator = CoordinatorAgent()
def solve(self, task: str) -> str:
# Coordinator assigns subtasks
assignments = self.coordinator.decompose(task)
results = {}
for assignment in assignments:
agent = self.agents[assignment.agent]
result = agent.execute(
assignment.subtask,
context=results
)
results[assignment.id] = result
# Critic reviews
critique = self.agents["critic"].review(results)
if critique.needs_revision:
# Iterate with feedback
return self.solve_with_feedback(task, results, critique)
return self.coordinator.synthesize(results)
```
---
## 3. Prompt IDE Patterns
### 3.1 Prompt Templates with Variables
```python
class PromptTemplate:
def __init__(self, template: str, variables: list[str]):
self.template = template
self.variables = variables
def format(self, **kwargs) -> str:
# Validate all variables provided
missing = set(self.variables) - set(kwargs.keys())
if missing:
raise ValueError(f"Missing variables: {missing}")
return self.template.format(**kwargs)
def with_examples(self, examples: list[dict]) -> str:
"""Add few-shot examples"""
example_text = "\n\n".join([
f"Input: {ex['input']}\nOutput: {ex['output']}"
for ex in examples
])
return f"{example_text}\n\n{self.template}"
# Usage
summarizer = PromptTemplate(
template="Summarize the following text in {style} style:\n\n{text}",
variables=["style", "text"]
)
prompt = summarizer.format(
style="professional",
text="Long article content..."
)
```
### 3.2 Prompt Versioning & A/B Testing
```python
class PromptRegistry:
def __init__(self, db):
self.db = db
def register(self, name: str, template: str, version: str):
"""Store prompt with version"""
self.db.save({
"name": name,
"template": template,
"version": version,
"created_at": datetime.now(),
"metrics": {}
})
def get(self, name: str, version: str = "latest") -> str:
"""Retrieve specific version"""
return self.db.get(name, version)
def ab_test(self, name: str, user_id: str) -> str:
"""Return variant based on user bucket"""
variants = self.db.get_all_versions(name)
bucket = hash(user_id) % len(variants)
return variants[bucket]
def record_outcome(self, prompt_id: str, outcome: dict):
"""Track prompt performance"""
self.db.update_metrics(prompt_id, outcome)
```
### 3.3 Prompt Chaining
```python
class PromptChain:
"""
Chain prompts together, passing output as input to next
"""
def __init__(self, steps: list[dict]):
self.steps = steps
def run(self, initial_input: str) -> dict:
context = {"input": initial_input}
results = []
for step in self.steps:
prompt = step["prompt"].format(**context)
output = llm.generate(prompt)
# Parse output if needed
if step.get("parser"):
output = step"parser"
context[step["output_key"]] = output
results.append({
"step": step["name"],
"output": output
})
return {
"final_output": context[self.steps[-1]["output_key"]],
"intermediate_results": results
}
# Example: Research → Analyze → Summarize
chain = PromptChain([
{
"name": "research",
"prompt": "Research the topic: {input}",
"output_key": "research"
},
{
"name": "analyze",
"prompt": "Analyze these findings:\n{research}",
"output_key": "analysis"
},
{
"name": "summarize",
"prompt": "Summarize this analysis in 3 bullet points:\n{analysis}",
"output_key": "summary"
}
])
```
---
## 4. LLMOps & Observability
### 4.1 Metrics to Track
```python
LLM_METRICS = {
# Performance
"latency_p50": "50th percentile response time",
"latency_p99": "99th percentile response time",
"tokens_per_second": "Generation speed",
# Quality
"user_satisfaction": "Thumbs up/down ratio",
"task_completion": "% tasks completed successfully",
"hallucination_rate": "% responses with factual errors",
# Cost
"cost_per_request": "Average $ per API call",
"tokens_per_request": "Average tokens used",
"cache_hit_rate": "% requests served from cache",
# Reliability
"error_rate": "% failed requests",
"timeout_rate": "% requests that timed out",
"retry_rate": "% requests needing retry"
}
```
### 4.2 Logging & Tracing
```python
import logging
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
class LLMLogger:
def log_request(self, request_id: str, data: dict):
"""Log LLM request for debugging and analysis"""
log_entry = {
"request_id": request_id,
"timestamp": datetime.now().isoformat(),
"model": data["model"],
"prompt": data["prompt"][:500], # Truncate for storage
"prompt_tokens": data["prompt_tokens"],
"temperature": data.get("temperature", 1.0),
"user_id": data.get("user_id"),
}
logging.info(f"LLM_REQUEST: {json.dumps(log_entry)}")
def log_response(self, request_id: str, data: dict):
"""Log LLM response"""
log_entry = {
"request_id": request_id,
"completion_tokens": data["completion_tokens"],
"total_tokens": data["total_tokens"],
"latency_ms": data["latency_ms"],
"finish_reason": data["finish_reason"],
"cost_usd": self._calculate_cost(data),
}
logging.info(f"LLM_RESPONSE: {json.dumps(log_entry)}")
# Distributed tracing
@tracer.start_as_current_span("llm_call")
def call_llm(prompt: str) -> str:
span = trace.get_current_span()
span.set_attribute("prompt.length", len(prompt))
response = llm.generate(prompt)
span.set_attribute("response.length", len(response))
span.set_attribute("tokens.total", response.usage.total_tokens)
return response.content
```
### 4.3 Evaluation Framework
```python
class LLMEvaluator:
"""
Evaluate LLM outputs for quality
"""
def evaluate_response(self,
question: str,
response: str,
ground_truth: str = None) -> dict:
scores = {}
# Relevance: Does it answer the question?
scores["relevance"] = self._score_relevance(question, response)
# Coherence: Is it well-structured?
scores["coherence"] = self._score_coherence(response)
# Groundedness: Is it based on provided context?
scores["groundedness"] = self._score_groundedness(response)
# Accuracy: Does it match ground truth?
if ground_truth:
scores["accuracy"] = self._score_accuracy(response, ground_truth)
# Harmfulness: Is it safe?
scores["safety"] = self._score_safety(response)
return scores
def run_benchmark(self, test_cases: list[dict]) -> dict:
"""Run evaluation on test set"""
results = []
for case in test_cases:
response = llm.generate(case["prompt"])
scores = self.evaluate_response(
question=case["prompt"],
response=response,
ground_truth=case.get("expected")
)
results.append(scores)
return self._aggregate_scores(results)
```
---
## 5. Production Patterns
### 5.1 Caching Strategy
```python
import hashlib
from functools import lru_cache
class LLMCache:
def __init__(self, redis_client, ttl_seconds=3600):
self.redis = redis_client
self.ttl = ttl_seconds
def _cache_key(self, prompt: str, model: str, **kwargs) -> str:
"""Generate deterministic cache key"""
content = f"{model}:{prompt}:{json.dumps(kwargs, sort_keys=True)}"
return hashlib.sha256(content.encode()).hexdigest()
def get_or_generate(self, prompt: str, model: str, **kwargs) -> str:
key = self._cache_key(prompt, model, **kwargs)
# Check cache
cached = self.redis.get(key)
if cached:
return cached.decode()
# Generate
response = llm.generate(prompt, model=model, **kwargs)
# Cache (only cache deterministic outputs)
if kwargs.get("temperature", 1.0) == 0:
self.redis.setex(key, self.ttl, response)
return response
```
### 5.2 Rate Limiting & Retry
```python
import time
from tenacity import retry, wait_exponential, stop_after_attempt
class RateLimiter:
def __init__(self, requests_per_minute: int):
self.rpm = requests_per_minute
self.timestamps = []
def acquire(self):
"""Wait if rate limit would be exceeded"""
now = time.time()
# Remove old timestamps
self.timestamps = [t for t in self.timestamps if now - t < 60]
if len(self.timestamps) >= self.rpm:
sleep_time = 60 - (now - self.timestamps[0])
time.sleep(sleep_time)
self.timestamps.append(time.time())
# Retry with exponential backoff
@retry(
wait=wait_exponential(multiplier=1, min=4, max=60),
stop=stop_after_attempt(5)
)
def call_llm_with_retry(prompt: str) -> str:
try:
return llm.generate(prompt)
except RateLimitError:
raise # Will trigger retry
except APIError as e:
if e.status_code >= 500:
raise # Retry server errors
raise # Don't retry client errors
```
### 5.3 Fallback Strategy
```python
class LLMWithFallback:
def __init__(self, primary: str, fallbacks: list[str]):
self.primary = primary
self.fallbacks = fallbacks
def generate(self, prompt: str, **kwargs) -> str:
models = [self.primary] + self.fallbacks
for model in models:
try:
return llm.generate(prompt, model=model, **kwargs)
except (RateLimitError, APIError) as e:
logging.warning(f"Model {model} failed: {e}")
continue
raise AllModelsFailedError("All models exhausted")
# Usage
llm_client = LLMWithFallback(
primary="gpt-4-turbo",
fallbacks=["gpt-3.5-turbo", "claude-3-sonnet"]
)
```
---
## Architecture Decision Matrix
| Pattern | Use When | Complexity | Cost |
| :------------------- | :--------------- | :--------- | :-------- |
| **Simple RAG** | FAQ, docs search | Low | Low |
| **Hybrid RAG** | Mixed queries | Medium | Medium |
| **ReAct Agent** | Multi-step tasks | Medium | Medium |
| **Function Calling** | Structured tools | Low | Low |
| **Plan-Execute** | Complex tasks | High | High |
| **Multi-Agent** | Research tasks | Very High | Very High |
---
## Resources
- [Dify Platform](https://github.com/langgenius/dify)
- [LangChain Docs](https://python.langchain.com/)
- [LlamaIndex](https://www.llamaindex.ai/)
- [Anthropic Cookbook](https://github.com/anthropics/anthropic-cookbook)
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/LICENSE.txt
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/SKILL.md
+246
View File
@@ -0,0 +1,246 @@
---
name: mcp-builder
description: "Create MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. The quality of an MCP server is measured by how well it enables LLMs to accomplish real-world tasks."
risk: unknown
source: community
date_added: "2026-02-27"
---
# MCP Server Development Guide
## Overview
Create MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. The quality of an MCP server is measured by how well it enables LLMs to accomplish real-world tasks.
---
# Process
## 🚀 High-Level Workflow
Creating a high-quality MCP server involves four main phases:
### Phase 1: Deep Research and Planning
#### 1.1 Understand Modern MCP Design
**API Coverage vs. Workflow Tools:**
Balance comprehensive API endpoint coverage with specialized workflow tools. Workflow tools can be more convenient for specific tasks, while comprehensive coverage gives agents flexibility to compose operations. Performance varies by client—some clients benefit from code execution that combines basic tools, while others work better with higher-level workflows. When uncertain, prioritize comprehensive API coverage.
**Tool Naming and Discoverability:**
Clear, descriptive tool names help agents find the right tools quickly. Use consistent prefixes (e.g., `github_create_issue`, `github_list_repos`) and action-oriented naming.
**Context Management:**
Agents benefit from concise tool descriptions and the ability to filter/paginate results. Design tools that return focused, relevant data. Some clients support code execution which can help agents filter and process data efficiently.
**Actionable Error Messages:**
Error messages should guide agents toward solutions with specific suggestions and next steps.
#### 1.2 Study MCP Protocol Documentation
**Navigate the MCP specification:**
Start with the sitemap to find relevant pages: `https://modelcontextprotocol.io/sitemap.xml`
Then fetch specific pages with `.md` suffix for markdown format (e.g., `https://modelcontextprotocol.io/specification/draft.md`).
Key pages to review:
- Specification overview and architecture
- Transport mechanisms (streamable HTTP, stdio)
- Tool, resource, and prompt definitions
#### 1.3 Study Framework Documentation
**Recommended stack:**
- **Language**: TypeScript (high-quality SDK support and good compatibility in many execution environments e.g. MCPB. Plus AI models are good at generating TypeScript code, benefiting from its broad usage, static typing and good linting tools)
- **Transport**: Streamable HTTP for remote servers, using stateless JSON (simpler to scale and maintain, as opposed to stateful sessions and streaming responses). stdio for local servers.
**Load framework documentation:**
- **MCP Best Practices**: [📋 View Best Practices](./reference/mcp_best_practices.md) - Core guidelines
**For TypeScript (recommended):**
- **TypeScript SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md`
- [⚡ TypeScript Guide](./reference/node_mcp_server.md) - TypeScript patterns and examples
**For Python:**
- **Python SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
- [🐍 Python Guide](./reference/python_mcp_server.md) - Python patterns and examples
#### 1.4 Plan Your Implementation
**Understand the API:**
Review the service's API documentation to identify key endpoints, authentication requirements, and data models. Use web search and WebFetch as needed.
**Tool Selection:**
Prioritize comprehensive API coverage. List endpoints to implement, starting with the most common operations.
---
### Phase 2: Implementation
#### 2.1 Set Up Project Structure
See language-specific guides for project setup:
- [⚡ TypeScript Guide](./reference/node_mcp_server.md) - Project structure, package.json, tsconfig.json
- [🐍 Python Guide](./reference/python_mcp_server.md) - Module organization, dependencies
#### 2.2 Implement Core Infrastructure
Create shared utilities:
- API client with authentication
- Error handling helpers
- Response formatting (JSON/Markdown)
- Pagination support
#### 2.3 Implement Tools
For each tool:
**Input Schema:**
- Use Zod (TypeScript) or Pydantic (Python)
- Include constraints and clear descriptions
- Add examples in field descriptions
**Output Schema:**
- Define `outputSchema` where possible for structured data
- Use `structuredContent` in tool responses (TypeScript SDK feature)
- Helps clients understand and process tool outputs
**Tool Description:**
- Concise summary of functionality
- Parameter descriptions
- Return type schema
**Implementation:**
- Async/await for I/O operations
- Proper error handling with actionable messages
- Support pagination where applicable
- Return both text content and structured data when using modern SDKs
**Annotations:**
- `readOnlyHint`: true/false
- `destructiveHint`: true/false
- `idempotentHint`: true/false
- `openWorldHint`: true/false
---
### Phase 3: Review and Test
#### 3.1 Code Quality
Review for:
- No duplicated code (DRY principle)
- Consistent error handling
- Full type coverage
- Clear tool descriptions
#### 3.2 Build and Test
**TypeScript:**
- Run `npm run build` to verify compilation
- Test with MCP Inspector: `npx @modelcontextprotocol/inspector`
**Python:**
- Verify syntax: `python -m py_compile your_server.py`
- Test with MCP Inspector
See language-specific guides for detailed testing approaches and quality checklists.
---
### Phase 4: Create Evaluations
After implementing your MCP server, create comprehensive evaluations to test its effectiveness.
**Load [✅ Evaluation Guide](./reference/evaluation.md) for complete evaluation guidelines.**
#### 4.1 Understand Evaluation Purpose
Use evaluations to test whether LLMs can effectively use your MCP server to answer realistic, complex questions.
#### 4.2 Create 10 Evaluation Questions
To create effective evaluations, follow the process outlined in the evaluation guide:
1. **Tool Inspection**: List available tools and understand their capabilities
2. **Content Exploration**: Use READ-ONLY operations to explore available data
3. **Question Generation**: Create 10 complex, realistic questions
4. **Answer Verification**: Solve each question yourself to verify answers
#### 4.3 Evaluation Requirements
Ensure each question is:
- **Independent**: Not dependent on other questions
- **Read-only**: Only non-destructive operations required
- **Complex**: Requiring multiple tool calls and deep exploration
- **Realistic**: Based on real use cases humans would care about
- **Verifiable**: Single, clear answer that can be verified by string comparison
- **Stable**: Answer won't change over time
#### 4.4 Output Format
Create an XML file with this structure:
```xml
<evaluation>
<qa_pair>
<question>Find discussions about AI model launches with animal codenames. One model needed a specific safety designation that uses the format ASL-X. What number X was being determined for the model named after a spotted wild cat?</question>
<answer>3</answer>
</qa_pair>
<!-- More qa_pairs... -->
</evaluation>
```
---
# Reference Files
## 📚 Documentation Library
Load these resources as needed during development:
### Core MCP Documentation (Load First)
- **MCP Protocol**: Start with sitemap at `https://modelcontextprotocol.io/sitemap.xml`, then fetch specific pages with `.md` suffix
- [📋 MCP Best Practices](./reference/mcp_best_practices.md) - Universal MCP guidelines including:
- Server and tool naming conventions
- Response format guidelines (JSON vs Markdown)
- Pagination best practices
- Transport selection (streamable HTTP vs stdio)
- Security and error handling standards
### SDK Documentation (Load During Phase 1/2)
- **Python SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
- **TypeScript SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md`
### Language-Specific Implementation Guides (Load During Phase 2)
- [🐍 Python Implementation Guide](./reference/python_mcp_server.md) - Complete Python/FastMCP guide with:
- Server initialization patterns
- Pydantic model examples
- Tool registration with `@mcp.tool`
- Complete working examples
- Quality checklist
- [⚡ TypeScript Implementation Guide](./reference/node_mcp_server.md) - Complete TypeScript guide with:
- Project structure
- Zod schema patterns
- Tool registration with `server.registerTool`
- Complete working examples
- Quality checklist
### Evaluation Guide (Load During Phase 4)
- [✅ Evaluation Guide](./reference/evaluation.md) - Complete evaluation creation guide with:
- Question creation guidelines
- Answer verification strategies
- XML format specifications
- Example questions and answers
- Running an evaluation with the provided scripts
## When to Use
This skill is applicable to execute the workflow or actions described in the overview.
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/reference/evaluation.md
+602
View File
@@ -0,0 +1,602 @@
# MCP Server Evaluation Guide
## Overview
This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.
---
## Quick Reference
### Evaluation Requirements
- Create 10 human-readable questions
- Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
- Each question requires multiple tool calls (potentially dozens)
- Answers must be single, verifiable values
- Answers must be STABLE (won't change over time)
### Output Format
```xml
<evaluation>
<qa_pair>
<question>Your question here</question>
<answer>Single verifiable answer</answer>
</qa_pair>
</evaluation>
```
---
## Purpose of Evaluations
The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs with no other context and access ONLY to the MCP servers to answer realistic and difficult questions.
## Evaluation Overview
Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
- Realistic
- Clear and concise
- Unambiguous
- Complex, requiring potentially dozens of tool calls or steps
- Answerable with a single, verifiable value that you identify in advance
## Question Guidelines
### Core Requirements
1. **Questions MUST be independent**
- Each question should NOT depend on the answer to any other question
- Should not assume prior write operations from processing another question
2. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use**
- Should not instruct or require modifying state to arrive at the correct answer
3. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX**
- Must require another LLM to use multiple (potentially dozens of) tools or steps to answer
### Complexity and Depth
4. **Questions must require deep exploration**
- Consider multi-hop questions requiring multiple sub-questions and sequential tool calls
- Each step should benefit from information found in previous questions
5. **Questions may require extensive paging**
- May need paging through multiple pages of results
- May require querying old data (1-2 years out-of-date) to find niche information
- The questions must be DIFFICULT
6. **Questions must require deep understanding**
- Rather than surface-level knowledge
- May pose complex ideas as True/False questions requiring evidence
- May use multiple-choice format where LLM must search different hypotheses
7. **Questions must not be solvable with straightforward keyword search**
- Do not include specific keywords from the target content
- Use synonyms, related concepts, or paraphrases
- Require multiple searches, analyzing multiple related items, extracting context, then deriving the answer
### Tool Testing
8. **Questions should stress-test tool return values**
- May elicit tools returning large JSON objects or lists, overwhelming the LLM
- Should require understanding multiple modalities of data:
- IDs and names
- Timestamps and datetimes (months, days, years, seconds)
- File IDs, names, extensions, and mimetypes
- URLs, GIDs, etc.
- Should probe the tool's ability to return all useful forms of data
9. **Questions should MOSTLY reflect real human use cases**
- The kinds of information retrieval tasks that HUMANS assisted by an LLM would care about
10. **Questions may require dozens of tool calls**
- This challenges LLMs with limited context
- Encourages MCP server tools to reduce information returned
11. **Include ambiguous questions**
- May be ambiguous OR require difficult decisions on which tools to call
- Force the LLM to potentially make mistakes or misinterpret
- Ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER
### Stability
12. **Questions must be designed so the answer DOES NOT CHANGE**
- Do not ask questions that rely on "current state" which is dynamic
- For example, do not count:
- Number of reactions to a post
- Number of replies to a thread
- Number of members in a channel
13. **DO NOT let the MCP server RESTRICT the kinds of questions you create**
- Create challenging and complex questions
- Some may not be solvable with the available MCP server tools
- Questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN)
- Questions may require dozens of tool calls to complete
## Answer Guidelines
### Verification
1. **Answers must be VERIFIABLE via direct string comparison**
- If the answer can be re-written in many formats, clearly specify the output format in the QUESTION
- Examples: "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else."
- Answer should be a single VERIFIABLE value such as:
- User ID, user name, display name, first name, last name
- Channel ID, channel name
- Message ID, string
- URL, title
- Numerical quantity
- Timestamp, datetime
- Boolean (for True/False questions)
- Email address, phone number
- File ID, file name, file extension
- Multiple choice answer
- Answers must not require special formatting or complex, structured output
- Answer will be verified using DIRECT STRING COMPARISON
### Readability
2. **Answers should generally prefer HUMAN-READABLE formats**
- Examples: names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d
- Rather than opaque IDs (though IDs are acceptable)
- The VAST MAJORITY of answers should be human-readable
### Stability
3. **Answers must be STABLE/STATIONARY**
- Look at old content (e.g., conversations that have ended, projects that have launched, questions answered)
- Create QUESTIONS based on "closed" concepts that will always return the same answer
- Questions may ask to consider a fixed time window to insulate from non-stationary answers
- Rely on context UNLIKELY to change
- Example: if finding a paper name, be SPECIFIC enough so answer is not confused with papers published later
4. **Answers must be CLEAR and UNAMBIGUOUS**
- Questions must be designed so there is a single, clear answer
- Answer can be derived from using the MCP server tools
### Diversity
5. **Answers must be DIVERSE**
- Answer should be a single VERIFIABLE value in diverse modalities and formats
- User concept: user ID, user name, display name, first name, last name, email address, phone number
- Channel concept: channel ID, channel name, channel topic
- Message concept: message ID, message string, timestamp, month, day, year
6. **Answers must NOT be complex structures**
- Not a list of values
- Not a complex object
- Not a list of IDs or strings
- Not natural language text
- UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON
- And can be realistically reproduced
- It should be unlikely that an LLM would return the same list in any other order or format
## Evaluation Process
### Step 1: Documentation Inspection
Read the documentation of the target API to understand:
- Available endpoints and functionality
- If ambiguity exists, fetch additional information from the web
- Parallelize this step AS MUCH AS POSSIBLE
- Ensure each subagent is ONLY examining documentation from the file system or on the web
### Step 2: Tool Inspection
List the tools available in the MCP server:
- Inspect the MCP server directly
- Understand input/output schemas, docstrings, and descriptions
- WITHOUT calling the tools themselves at this stage
### Step 3: Developing Understanding
Repeat steps 1 & 2 until you have a good understanding:
- Iterate multiple times
- Think about the kinds of tasks you want to create
- Refine your understanding
- At NO stage should you READ the code of the MCP server implementation itself
- Use your intuition and understanding to create reasonable, realistic, but VERY challenging tasks
### Step 4: Read-Only Content Inspection
After understanding the API and tools, USE the MCP server tools:
- Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY
- Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions
- Should NOT call any tools that modify state
- Will NOT read the code of the MCP server implementation itself
- Parallelize this step with individual sub-agents pursuing independent explorations
- Ensure each subagent is only performing READ-ONLY, NON-DESTRUCTIVE, and IDEMPOTENT operations
- BE CAREFUL: SOME TOOLS may return LOTS OF DATA which would cause you to run out of CONTEXT
- Make INCREMENTAL, SMALL, AND TARGETED tool calls for exploration
- In all tool call requests, use the `limit` parameter to limit results (<10)
- Use pagination
### Step 5: Task Generation
After inspecting the content, create 10 human-readable questions:
- An LLM should be able to answer these with the MCP server
- Follow all question and answer guidelines above
## Output Format
Each QA pair consists of a question and an answer. The output should be an XML file with this structure:
```xml
<evaluation>
<qa_pair>
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
<answer>Website Redesign</answer>
</qa_pair>
<qa_pair>
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
<answer>sarah_dev</answer>
</qa_pair>
<qa_pair>
<question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question>
<answer>7</answer>
</qa_pair>
<qa_pair>
<question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
<answer>data-pipeline</answer>
</qa_pair>
</evaluation>
```
## Evaluation Examples
### Good Questions
**Example 1: Multi-hop question requiring deep exploration (GitHub MCP)**
```xml
<qa_pair>
<question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
<answer>Python</answer>
</qa_pair>
```
This question is good because:
- Requires multiple searches to find archived repositories
- Needs to identify which had the most forks before archival
- Requires examining repository details for the language
- Answer is a simple, verifiable value
- Based on historical (closed) data that won't change
**Example 2: Requires understanding context without keyword matching (Project Management MCP)**
```xml
<qa_pair>
<question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
<answer>Product Manager</answer>
</qa_pair>
```
This question is good because:
- Doesn't use specific project name ("initiative focused on improving customer onboarding")
- Requires finding completed projects from specific timeframe
- Needs to identify the project lead and their role
- Requires understanding context from retrospective documents
- Answer is human-readable and stable
- Based on completed work (won't change)
**Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)**
```xml
<qa_pair>
<question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
<answer>alex_eng</answer>
</qa_pair>
```
This question is good because:
- Requires filtering bugs by date, priority, and status
- Needs to group by assignee and calculate resolution rates
- Requires understanding timestamps to determine 48-hour windows
- Tests pagination (potentially many bugs to process)
- Answer is a single username
- Based on historical data from specific time period
**Example 4: Requires synthesis across multiple data types (CRM MCP)**
```xml
<qa_pair>
<question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
<answer>Healthcare</answer>
</qa_pair>
```
This question is good because:
- Requires understanding subscription tier changes
- Needs to identify upgrade events in specific timeframe
- Requires comparing contract values
- Must access account industry information
- Answer is simple and verifiable
- Based on completed historical transactions
### Poor Questions
**Example 1: Answer changes over time**
```xml
<qa_pair>
<question>How many open issues are currently assigned to the engineering team?</question>
<answer>47</answer>
</qa_pair>
```
This question is poor because:
- The answer will change as issues are created, closed, or reassigned
- Not based on stable/stationary data
- Relies on "current state" which is dynamic
**Example 2: Too easy with keyword search**
```xml
<qa_pair>
<question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
<answer>developer123</answer>
</qa_pair>
```
This question is poor because:
- Can be solved with a straightforward keyword search for exact title
- Doesn't require deep exploration or understanding
- No synthesis or analysis needed
**Example 3: Ambiguous answer format**
```xml
<qa_pair>
<question>List all the repositories that have Python as their primary language.</question>
<answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer>
</qa_pair>
```
This question is poor because:
- Answer is a list that could be returned in any order
- Difficult to verify with direct string comparison
- LLM might format differently (JSON array, comma-separated, newline-separated)
- Better to ask for a specific aggregate (count) or superlative (most stars)
## Verification Process
After creating evaluations:
1. **Examine the XML file** to understand the schema
2. **Load each task instruction** and in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF
3. **Flag any operations** that require WRITE or DESTRUCTIVE operations
4. **Accumulate all CORRECT answers** and replace any incorrect answers in the document
5. **Remove any `<qa_pair>`** that require WRITE or DESTRUCTIVE operations
Remember to parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end.
## Tips for Creating Quality Evaluations
1. **Think Hard and Plan Ahead** before generating tasks
2. **Parallelize Where Opportunity Arises** to speed up the process and manage context
3. **Focus on Realistic Use Cases** that humans would actually want to accomplish
4. **Create Challenging Questions** that test the limits of the MCP server's capabilities
5. **Ensure Stability** by using historical data and closed concepts
6. **Verify Answers** by solving the questions yourself using the MCP server tools
7. **Iterate and Refine** based on what you learn during the process
---
# Running Evaluations
After creating your evaluation file, you can use the provided evaluation harness to test your MCP server.
## Setup
1. **Install Dependencies**
```bash
pip install -r scripts/requirements.txt
```
Or install manually:
```bash
pip install anthropic mcp
```
2. **Set API Key**
```bash
export ANTHROPIC_API_KEY=your_api_key_here
```
## Evaluation File Format
Evaluation files use XML format with `<qa_pair>` elements:
```xml
<evaluation>
<qa_pair>
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
<answer>Website Redesign</answer>
</qa_pair>
<qa_pair>
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
<answer>sarah_dev</answer>
</qa_pair>
</evaluation>
```
## Running Evaluations
The evaluation script (`scripts/evaluation.py`) supports three transport types:
**Important:**
- **stdio transport**: The evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
- **sse/http transports**: You must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
### 1. Local STDIO Server
For locally-run MCP servers (script launches the server automatically):
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_mcp_server.py \
evaluation.xml
```
With environment variables:
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_mcp_server.py \
-e API_KEY=abc123 \
-e DEBUG=true \
evaluation.xml
```
### 2. Server-Sent Events (SSE)
For SSE-based MCP servers (you must start the server first):
```bash
python scripts/evaluation.py \
-t sse \
-u https://example.com/mcp \
-H "Authorization: Bearer token123" \
-H "X-Custom-Header: value" \
evaluation.xml
```
### 3. HTTP (Streamable HTTP)
For HTTP-based MCP servers (you must start the server first):
```bash
python scripts/evaluation.py \
-t http \
-u https://example.com/mcp \
-H "Authorization: Bearer token123" \
evaluation.xml
```
## Command-Line Options
```
usage: evaluation.py [-h] [-t {stdio,sse,http}] [-m MODEL] [-c COMMAND]
[-a ARGS [ARGS ...]] [-e ENV [ENV ...]] [-u URL]
[-H HEADERS [HEADERS ...]] [-o OUTPUT]
eval_file
positional arguments:
eval_file Path to evaluation XML file
optional arguments:
-h, --help Show help message
-t, --transport Transport type: stdio, sse, or http (default: stdio)
-m, --model Claude model to use (default: claude-3-7-sonnet-20250219)
-o, --output Output file for report (default: print to stdout)
stdio options:
-c, --command Command to run MCP server (e.g., python, node)
-a, --args Arguments for the command (e.g., server.py)
-e, --env Environment variables in KEY=VALUE format
sse/http options:
-u, --url MCP server URL
-H, --header HTTP headers in 'Key: Value' format
```
## Output
The evaluation script generates a detailed report including:
- **Summary Statistics**:
- Accuracy (correct/total)
- Average task duration
- Average tool calls per task
- Total tool calls
- **Per-Task Results**:
- Prompt and expected response
- Actual response from the agent
- Whether the answer was correct (✅/❌)
- Duration and tool call details
- Agent's summary of its approach
- Agent's feedback on the tools
### Save Report to File
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_server.py \
-o evaluation_report.md \
evaluation.xml
```
## Complete Example Workflow
Here's a complete example of creating and running an evaluation:
1. **Create your evaluation file** (`my_evaluation.xml`):
```xml
<evaluation>
<qa_pair>
<question>Find the user who created the most issues in January 2024. What is their username?</question>
<answer>alice_developer</answer>
</qa_pair>
<qa_pair>
<question>Among all pull requests merged in Q1 2024, which repository had the highest number? Provide the repository name.</question>
<answer>backend-api</answer>
</qa_pair>
<qa_pair>
<question>Find the project that was completed in December 2023 and had the longest duration from start to finish. How many days did it take?</question>
<answer>127</answer>
</qa_pair>
</evaluation>
```
2. **Install dependencies**:
```bash
pip install -r scripts/requirements.txt
export ANTHROPIC_API_KEY=your_api_key
```
3. **Run evaluation**:
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a github_mcp_server.py \
-e GITHUB_TOKEN=ghp_xxx \
-o github_eval_report.md \
my_evaluation.xml
```
4. **Review the report** in `github_eval_report.md` to:
- See which questions passed/failed
- Read the agent's feedback on your tools
- Identify areas for improvement
- Iterate on your MCP server design
## Troubleshooting
### Connection Errors
If you get connection errors:
- **STDIO**: Verify the command and arguments are correct
- **SSE/HTTP**: Check the URL is accessible and headers are correct
- Ensure any required API keys are set in environment variables or headers
### Low Accuracy
If many evaluations fail:
- Review the agent's feedback for each task
- Check if tool descriptions are clear and comprehensive
- Verify input parameters are well-documented
- Consider whether tools return too much or too little data
- Ensure error messages are actionable
### Timeout Issues
If tasks are timing out:
- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
- Check if tools are returning too much data
- Verify pagination is working correctly
- Consider simplifying complex questions
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/reference/mcp_best_practices.md
+249
View File
@@ -0,0 +1,249 @@
# MCP Server Best Practices
## Quick Reference
### Server Naming
- **Python**: `{service}_mcp` (e.g., `slack_mcp`)
- **Node/TypeScript**: `{service}-mcp-server` (e.g., `slack-mcp-server`)
### Tool Naming
- Use snake_case with service prefix
- Format: `{service}_{action}_{resource}`
- Example: `slack_send_message`, `github_create_issue`
### Response Formats
- Support both JSON and Markdown formats
- JSON for programmatic processing
- Markdown for human readability
### Pagination
- Always respect `limit` parameter
- Return `has_more`, `next_offset`, `total_count`
- Default to 20-50 items
### Transport
- **Streamable HTTP**: For remote servers, multi-client scenarios
- **stdio**: For local integrations, command-line tools
- Avoid SSE (deprecated in favor of streamable HTTP)
---
## Server Naming Conventions
Follow these standardized naming patterns:
**Python**: Use format `{service}_mcp` (lowercase with underscores)
- Examples: `slack_mcp`, `github_mcp`, `jira_mcp`
**Node/TypeScript**: Use format `{service}-mcp-server` (lowercase with hyphens)
- Examples: `slack-mcp-server`, `github-mcp-server`, `jira-mcp-server`
The name should be general, descriptive of the service being integrated, easy to infer from the task description, and without version numbers.
---
## Tool Naming and Design
### Tool Naming
1. **Use snake_case**: `search_users`, `create_project`, `get_channel_info`
2. **Include service prefix**: Anticipate that your MCP server may be used alongside other MCP servers
- Use `slack_send_message` instead of just `send_message`
- Use `github_create_issue` instead of just `create_issue`
3. **Be action-oriented**: Start with verbs (get, list, search, create, etc.)
4. **Be specific**: Avoid generic names that could conflict with other servers
### Tool Design
- Tool descriptions must narrowly and unambiguously describe functionality
- Descriptions must precisely match actual functionality
- Provide tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- Keep tool operations focused and atomic
---
## Response Formats
All tools that return data should support multiple formats:
### JSON Format (`response_format="json"`)
- Machine-readable structured data
- Include all available fields and metadata
- Consistent field names and types
- Use for programmatic processing
### Markdown Format (`response_format="markdown"`, typically default)
- Human-readable formatted text
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format
- Show display names with IDs in parentheses
- Omit verbose metadata
---
## Pagination
For tools that list resources:
- **Always respect the `limit` parameter**
- **Implement pagination**: Use `offset` or cursor-based pagination
- **Return pagination metadata**: Include `has_more`, `next_offset`/`next_cursor`, `total_count`
- **Never load all results into memory**: Especially important for large datasets
- **Default to reasonable limits**: 20-50 items is typical
Example pagination response:
```json
{
"total": 150,
"count": 20,
"offset": 0,
"items": [...],
"has_more": true,
"next_offset": 20
}
```
---
## Transport Options
### Streamable HTTP
**Best for**: Remote servers, web services, multi-client scenarios
**Characteristics**:
- Bidirectional communication over HTTP
- Supports multiple simultaneous clients
- Can be deployed as a web service
- Enables server-to-client notifications
**Use when**:
- Serving multiple clients simultaneously
- Deploying as a cloud service
- Integration with web applications
### stdio
**Best for**: Local integrations, command-line tools
**Characteristics**:
- Standard input/output stream communication
- Simple setup, no network configuration needed
- Runs as a subprocess of the client
**Use when**:
- Building tools for local development environments
- Integrating with desktop applications
- Single-user, single-session scenarios
**Note**: stdio servers should NOT log to stdout (use stderr for logging)
### Transport Selection
| Criterion | stdio | Streamable HTTP |
|-----------|-------|-----------------|
| **Deployment** | Local | Remote |
| **Clients** | Single | Multiple |
| **Complexity** | Low | Medium |
| **Real-time** | No | Yes |
---
## Security Best Practices
### Authentication and Authorization
**OAuth 2.1**:
- Use secure OAuth 2.1 with certificates from recognized authorities
- Validate access tokens before processing requests
- Only accept tokens specifically intended for your server
**API Keys**:
- Store API keys in environment variables, never in code
- Validate keys on server startup
- Provide clear error messages when authentication fails
### Input Validation
- Sanitize file paths to prevent directory traversal
- Validate URLs and external identifiers
- Check parameter sizes and ranges
- Prevent command injection in system calls
- Use schema validation (Pydantic/Zod) for all inputs
### Error Handling
- Don't expose internal errors to clients
- Log security-relevant errors server-side
- Provide helpful but not revealing error messages
- Clean up resources after errors
### DNS Rebinding Protection
For streamable HTTP servers running locally:
- Enable DNS rebinding protection
- Validate the `Origin` header on all incoming connections
- Bind to `127.0.0.1` rather than `0.0.0.0`
---
## Tool Annotations
Provide annotations to help clients understand tool behavior:
| Annotation | Type | Default | Description |
|-----------|------|---------|-------------|
| `readOnlyHint` | boolean | false | Tool does not modify its environment |
| `destructiveHint` | boolean | true | Tool may perform destructive updates |
| `idempotentHint` | boolean | false | Repeated calls with same args have no additional effect |
| `openWorldHint` | boolean | true | Tool interacts with external entities |
**Important**: Annotations are hints, not security guarantees. Clients should not make security-critical decisions based solely on annotations.
---
## Error Handling
- Use standard JSON-RPC error codes
- Report tool errors within result objects (not protocol-level errors)
- Provide helpful, specific error messages with suggested next steps
- Don't expose internal implementation details
- Clean up resources properly on errors
Example error handling:
```typescript
try {
const result = performOperation();
return { content: [{ type: "text", text: result }] };
} catch (error) {
return {
isError: true,
content: [{
type: "text",
text: `Error: ${error.message}. Try using filter='active_only' to reduce results.`
}]
};
}
```
---
## Testing Requirements
Comprehensive testing should cover:
- **Functional testing**: Verify correct execution with valid/invalid inputs
- **Integration testing**: Test interaction with external systems
- **Security testing**: Validate auth, input sanitization, rate limiting
- **Performance testing**: Check behavior under load, timeouts
- **Error handling**: Ensure proper error reporting and cleanup
---
## Documentation Requirements
- Provide clear documentation of all tools and capabilities
- Include working examples (at least 3 per major feature)
- Document security considerations
- Specify required permissions and access levels
- Document rate limits and performance characteristics
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/reference/node_mcp_server.md
+970
View File
@@ -0,0 +1,970 @@
# Node/TypeScript MCP Server Implementation Guide
## Overview
This document provides Node/TypeScript-specific best practices and examples for implementing MCP servers using the MCP TypeScript SDK. It covers project structure, server setup, tool registration patterns, input validation with Zod, error handling, and complete working examples.
---
## Quick Reference
### Key Imports
```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import express from "express";
import { z } from "zod";
```
### Server Initialization
```typescript
const server = new McpServer({
name: "service-mcp-server",
version: "1.0.0"
});
```
### Tool Registration Pattern
```typescript
server.registerTool(
"tool_name",
{
title: "Tool Display Name",
description: "What the tool does",
inputSchema: { param: z.string() },
outputSchema: { result: z.string() }
},
async ({ param }) => {
const output = { result: `Processed: ${param}` };
return {
content: [{ type: "text", text: JSON.stringify(output) }],
structuredContent: output // Modern pattern for structured data
};
}
);
```
---
## MCP TypeScript SDK
The official MCP TypeScript SDK provides:
- `McpServer` class for server initialization
- `registerTool` method for tool registration
- Zod schema integration for runtime input validation
- Type-safe tool handler implementations
**IMPORTANT - Use Modern APIs Only:**
- **DO use**: `server.registerTool()`, `server.registerResource()`, `server.registerPrompt()`
- **DO NOT use**: Old deprecated APIs such as `server.tool()`, `server.setRequestHandler(ListToolsRequestSchema, ...)`, or manual handler registration
- The `register*` methods provide better type safety, automatic schema handling, and are the recommended approach
See the MCP SDK documentation in the references for complete details.
## Server Naming Convention
Node/TypeScript MCP servers must follow this naming pattern:
- **Format**: `{service}-mcp-server` (lowercase with hyphens)
- **Examples**: `github-mcp-server`, `jira-mcp-server`, `stripe-mcp-server`
The name should be:
- General (not tied to specific features)
- Descriptive of the service/API being integrated
- Easy to infer from the task description
- Without version numbers or dates
## Project Structure
Create the following structure for Node/TypeScript MCP servers:
```
{service}-mcp-server/
├── package.json
├── tsconfig.json
├── README.md
├── src/
│ ├── index.ts # Main entry point with McpServer initialization
│ ├── types.ts # TypeScript type definitions and interfaces
│ ├── tools/ # Tool implementations (one file per domain)
│ ├── services/ # API clients and shared utilities
│ ├── schemas/ # Zod validation schemas
│ └── constants.ts # Shared constants (API_URL, CHARACTER_LIMIT, etc.)
└── dist/ # Built JavaScript files (entry point: dist/index.js)
```
## Tool Implementation
### Tool Naming
Use snake_case for tool names (e.g., "search_users", "create_project", "get_channel_info") with clear, action-oriented names.
**Avoid Naming Conflicts**: Include the service context to prevent overlaps:
- Use "slack_send_message" instead of just "send_message"
- Use "github_create_issue" instead of just "create_issue"
- Use "asana_list_tasks" instead of just "list_tasks"
### Tool Structure
Tools are registered using the `registerTool` method with the following requirements:
- Use Zod schemas for runtime input validation and type safety
- The `description` field must be explicitly provided - JSDoc comments are NOT automatically extracted
- Explicitly provide `title`, `description`, `inputSchema`, and `annotations`
- The `inputSchema` must be a Zod schema object (not a JSON schema)
- Type all parameters and return values explicitly
```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
const server = new McpServer({
name: "example-mcp",
version: "1.0.0"
});
// Zod schema for input validation
const UserSearchInputSchema = z.object({
query: z.string()
.min(2, "Query must be at least 2 characters")
.max(200, "Query must not exceed 200 characters")
.describe("Search string to match against names/emails"),
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip for pagination"),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
}).strict();
// Type definition from Zod schema
type UserSearchInput = z.infer<typeof UserSearchInputSchema>;
server.registerTool(
"example_search_users",
{
title: "Search Example Users",
description: `Search for users in the Example system by name, email, or team.
This tool searches across all user profiles in the Example platform, supporting partial matches and various search filters. It does NOT create or modify users, only searches existing ones.
Args:
- query (string): Search string to match against names/emails
- limit (number): Maximum results to return, between 1-100 (default: 20)
- offset (number): Number of results to skip for pagination (default: 0)
- response_format ('markdown' | 'json'): Output format (default: 'markdown')
Returns:
For JSON format: Structured data with schema:
{
"total": number, // Total number of matches found
"count": number, // Number of results in this response
"offset": number, // Current pagination offset
"users": [
{
"id": string, // User ID (e.g., "U123456789")
"name": string, // Full name (e.g., "John Doe")
"email": string, // Email address
"team": string, // Team name (optional)
"active": boolean // Whether user is active
}
],
"has_more": boolean, // Whether more results are available
"next_offset": number // Offset for next page (if has_more is true)
}
Examples:
- Use when: "Find all marketing team members" -> params with query="team:marketing"
- Use when: "Search for John's account" -> params with query="john"
- Don't use when: You need to create a user (use example_create_user instead)
Error Handling:
- Returns "Error: Rate limit exceeded" if too many requests (429 status)
- Returns "No users found matching '<query>'" if search returns empty`,
inputSchema: UserSearchInputSchema,
annotations: {
readOnlyHint: true,
destructiveHint: false,
idempotentHint: true,
openWorldHint: true
}
},
async (params: UserSearchInput) => {
try {
// Input validation is handled by Zod schema
// Make API request using validated parameters
const data = await makeApiRequest<any>(
"users/search",
"GET",
undefined,
{
q: params.query,
limit: params.limit,
offset: params.offset
}
);
const users = data.users || [];
const total = data.total || 0;
if (!users.length) {
return {
content: [{
type: "text",
text: `No users found matching '${params.query}'`
}]
};
}
// Prepare structured output
const output = {
total,
count: users.length,
offset: params.offset,
users: users.map((user: any) => ({
id: user.id,
name: user.name,
email: user.email,
...(user.team ? { team: user.team } : {}),
active: user.active ?? true
})),
has_more: total > params.offset + users.length,
...(total > params.offset + users.length ? {
next_offset: params.offset + users.length
} : {})
};
// Format text representation based on requested format
let textContent: string;
if (params.response_format === ResponseFormat.MARKDOWN) {
const lines = [`# User Search Results: '${params.query}'`, "",
`Found ${total} users (showing ${users.length})`, ""];
for (const user of users) {
lines.push(`## ${user.name} (${user.id})`);
lines.push(`- **Email**: ${user.email}`);
if (user.team) lines.push(`- **Team**: ${user.team}`);
lines.push("");
}
textContent = lines.join("\n");
} else {
textContent = JSON.stringify(output, null, 2);
}
return {
content: [{ type: "text", text: textContent }],
structuredContent: output // Modern pattern for structured data
};
} catch (error) {
return {
content: [{
type: "text",
text: handleApiError(error)
}]
};
}
}
);
```
## Zod Schemas for Input Validation
Zod provides runtime type validation:
```typescript
import { z } from "zod";
// Basic schema with validation
const CreateUserSchema = z.object({
name: z.string()
.min(1, "Name is required")
.max(100, "Name must not exceed 100 characters"),
email: z.string()
.email("Invalid email format"),
age: z.number()
.int("Age must be a whole number")
.min(0, "Age cannot be negative")
.max(150, "Age cannot be greater than 150")
}).strict(); // Use .strict() to forbid extra fields
// Enums
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
const SearchSchema = z.object({
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format")
});
// Optional fields with defaults
const PaginationSchema = z.object({
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip")
});
```
## Response Format Options
Support multiple output formats for flexibility:
```typescript
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
const inputSchema = z.object({
query: z.string(),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
});
```
**Markdown format**:
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format
- Show display names with IDs in parentheses
- Omit verbose metadata
- Group related information logically
**JSON format**:
- Return complete, structured data suitable for programmatic processing
- Include all available fields and metadata
- Use consistent field names and types
## Pagination Implementation
For tools that list resources:
```typescript
const ListSchema = z.object({
limit: z.number().int().min(1).max(100).default(20),
offset: z.number().int().min(0).default(0)
});
async function listItems(params: z.infer<typeof ListSchema>) {
const data = await apiRequest(params.limit, params.offset);
const response = {
total: data.total,
count: data.items.length,
offset: params.offset,
items: data.items,
has_more: data.total > params.offset + data.items.length,
next_offset: data.total > params.offset + data.items.length
? params.offset + data.items.length
: undefined
};
return JSON.stringify(response, null, 2);
}
```
## Character Limits and Truncation
Add a CHARACTER_LIMIT constant to prevent overwhelming responses:
```typescript
// At module level in constants.ts
export const CHARACTER_LIMIT = 25000; // Maximum response size in characters
async function searchTool(params: SearchInput) {
let result = generateResponse(data);
// Check character limit and truncate if needed
if (result.length > CHARACTER_LIMIT) {
const truncatedData = data.slice(0, Math.max(1, data.length / 2));
response.data = truncatedData;
response.truncated = true;
response.truncation_message =
`Response truncated from ${data.length} to ${truncatedData.length} items. ` +
`Use 'offset' parameter or add filters to see more results.`;
result = JSON.stringify(response, null, 2);
}
return result;
}
```
## Error Handling
Provide clear, actionable error messages:
```typescript
import axios, { AxiosError } from "axios";
function handleApiError(error: unknown): string {
if (error instanceof AxiosError) {
if (error.response) {
switch (error.response.status) {
case 404:
return "Error: Resource not found. Please check the ID is correct.";
case 403:
return "Error: Permission denied. You don't have access to this resource.";
case 429:
return "Error: Rate limit exceeded. Please wait before making more requests.";
default:
return `Error: API request failed with status ${error.response.status}`;
}
} else if (error.code === "ECONNABORTED") {
return "Error: Request timed out. Please try again.";
}
}
return `Error: Unexpected error occurred: ${error instanceof Error ? error.message : String(error)}`;
}
```
## Shared Utilities
Extract common functionality into reusable functions:
```typescript
// Shared API request function
async function makeApiRequest<T>(
endpoint: string,
method: "GET" | "POST" | "PUT" | "DELETE" = "GET",
data?: any,
params?: any
): Promise<T> {
try {
const response = await axios({
method,
url: `${API_BASE_URL}/${endpoint}`,
data,
params,
timeout: 30000,
headers: {
"Content-Type": "application/json",
"Accept": "application/json"
}
});
return response.data;
} catch (error) {
throw error;
}
}
```
## Async/Await Best Practices
Always use async/await for network requests and I/O operations:
```typescript
// Good: Async network request
async function fetchData(resourceId: string): Promise<ResourceData> {
const response = await axios.get(`${API_URL}/resource/${resourceId}`);
return response.data;
}
// Bad: Promise chains
function fetchData(resourceId: string): Promise<ResourceData> {
return axios.get(`${API_URL}/resource/${resourceId}`)
.then(response => response.data); // Harder to read and maintain
}
```
## TypeScript Best Practices
1. **Use Strict TypeScript**: Enable strict mode in tsconfig.json
2. **Define Interfaces**: Create clear interface definitions for all data structures
3. **Avoid `any`**: Use proper types or `unknown` instead of `any`
4. **Zod for Runtime Validation**: Use Zod schemas to validate external data
5. **Type Guards**: Create type guard functions for complex type checking
6. **Error Handling**: Always use try-catch with proper error type checking
7. **Null Safety**: Use optional chaining (`?.`) and nullish coalescing (`??`)
```typescript
// Good: Type-safe with Zod and interfaces
interface UserResponse {
id: string;
name: string;
email: string;
team?: string;
active: boolean;
}
const UserSchema = z.object({
id: z.string(),
name: z.string(),
email: z.string().email(),
team: z.string().optional(),
active: z.boolean()
});
type User = z.infer<typeof UserSchema>;
async function getUser(id: string): Promise<User> {
const data = await apiCall(`/users/${id}`);
return UserSchema.parse(data); // Runtime validation
}
// Bad: Using any
async function getUser(id: string): Promise<any> {
return await apiCall(`/users/${id}`); // No type safety
}
```
## Package Configuration
### package.json
```json
{
"name": "{service}-mcp-server",
"version": "1.0.0",
"description": "MCP server for {Service} API integration",
"type": "module",
"main": "dist/index.js",
"scripts": {
"start": "node dist/index.js",
"dev": "tsx watch src/index.ts",
"build": "tsc",
"clean": "rm -rf dist"
},
"engines": {
"node": ">=18"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.6.1",
"axios": "^1.7.9",
"zod": "^3.23.8"
},
"devDependencies": {
"@types/node": "^22.10.0",
"tsx": "^4.19.2",
"typescript": "^5.7.2"
}
}
```
### tsconfig.json
```json
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"allowSyntheticDefaultImports": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
```
## Complete Example
```typescript
#!/usr/bin/env node
/**
* MCP Server for Example Service.
*
* This server provides tools to interact with Example API, including user search,
* project management, and data export capabilities.
*/
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import axios, { AxiosError } from "axios";
// Constants
const API_BASE_URL = "https://api.example.com/v1";
const CHARACTER_LIMIT = 25000;
// Enums
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
// Zod schemas
const UserSearchInputSchema = z.object({
query: z.string()
.min(2, "Query must be at least 2 characters")
.max(200, "Query must not exceed 200 characters")
.describe("Search string to match against names/emails"),
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip for pagination"),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
}).strict();
type UserSearchInput = z.infer<typeof UserSearchInputSchema>;
// Shared utility functions
async function makeApiRequest<T>(
endpoint: string,
method: "GET" | "POST" | "PUT" | "DELETE" = "GET",
data?: any,
params?: any
): Promise<T> {
try {
const response = await axios({
method,
url: `${API_BASE_URL}/${endpoint}`,
data,
params,
timeout: 30000,
headers: {
"Content-Type": "application/json",
"Accept": "application/json"
}
});
return response.data;
} catch (error) {
throw error;
}
}
function handleApiError(error: unknown): string {
if (error instanceof AxiosError) {
if (error.response) {
switch (error.response.status) {
case 404:
return "Error: Resource not found. Please check the ID is correct.";
case 403:
return "Error: Permission denied. You don't have access to this resource.";
case 429:
return "Error: Rate limit exceeded. Please wait before making more requests.";
default:
return `Error: API request failed with status ${error.response.status}`;
}
} else if (error.code === "ECONNABORTED") {
return "Error: Request timed out. Please try again.";
}
}
return `Error: Unexpected error occurred: ${error instanceof Error ? error.message : String(error)}`;
}
// Create MCP server instance
const server = new McpServer({
name: "example-mcp",
version: "1.0.0"
});
// Register tools
server.registerTool(
"example_search_users",
{
title: "Search Example Users",
description: `[Full description as shown above]`,
inputSchema: UserSearchInputSchema,
annotations: {
readOnlyHint: true,
destructiveHint: false,
idempotentHint: true,
openWorldHint: true
}
},
async (params: UserSearchInput) => {
// Implementation as shown above
}
);
// Main function
// For stdio (local):
async function runStdio() {
if (!process.env.EXAMPLE_API_KEY) {
console.error("ERROR: EXAMPLE_API_KEY environment variable is required");
process.exit(1);
}
const transport = new StdioServerTransport();
await server.connect(transport);
console.error("MCP server running via stdio");
}
// For streamable HTTP (remote):
async function runHTTP() {
if (!process.env.EXAMPLE_API_KEY) {
console.error("ERROR: EXAMPLE_API_KEY environment variable is required");
process.exit(1);
}
const app = express();
app.use(express.json());
app.post('/mcp', async (req, res) => {
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: undefined,
enableJsonResponse: true
});
res.on('close', () => transport.close());
await server.connect(transport);
await transport.handleRequest(req, res, req.body);
});
const port = parseInt(process.env.PORT || '3000');
app.listen(port, () => {
console.error(`MCP server running on http://localhost:${port}/mcp`);
});
}
// Choose transport based on environment
const transport = process.env.TRANSPORT || 'stdio';
if (transport === 'http') {
runHTTP().catch(error => {
console.error("Server error:", error);
process.exit(1);
});
} else {
runStdio().catch(error => {
console.error("Server error:", error);
process.exit(1);
});
}
```
---
## Advanced MCP Features
### Resource Registration
Expose data as resources for efficient, URI-based access:
```typescript
import { ResourceTemplate } from "@modelcontextprotocol/sdk/types.js";
// Register a resource with URI template
server.registerResource(
{
uri: "file://documents/{name}",
name: "Document Resource",
description: "Access documents by name",
mimeType: "text/plain"
},
async (uri: string) => {
// Extract parameter from URI
const match = uri.match(/^file:\/\/documents\/(.+)$/);
if (!match) {
throw new Error("Invalid URI format");
}
const documentName = match[1];
const content = await loadDocument(documentName);
return {
contents: [{
uri,
mimeType: "text/plain",
text: content
}]
};
}
);
// List available resources dynamically
server.registerResourceList(async () => {
const documents = await getAvailableDocuments();
return {
resources: documents.map(doc => ({
uri: `file://documents/${doc.name}`,
name: doc.name,
mimeType: "text/plain",
description: doc.description
}))
};
});
```
**When to use Resources vs Tools:**
- **Resources**: For data access with simple URI-based parameters
- **Tools**: For complex operations requiring validation and business logic
- **Resources**: When data is relatively static or template-based
- **Tools**: When operations have side effects or complex workflows
### Transport Options
The TypeScript SDK supports two main transport mechanisms:
#### Streamable HTTP (Recommended for Remote Servers)
```typescript
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import express from "express";
const app = express();
app.use(express.json());
app.post('/mcp', async (req, res) => {
// Create new transport for each request (stateless, prevents request ID collisions)
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: undefined,
enableJsonResponse: true
});
res.on('close', () => transport.close());
await server.connect(transport);
await transport.handleRequest(req, res, req.body);
});
app.listen(3000);
```
#### stdio (For Local Integrations)
```typescript
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
const transport = new StdioServerTransport();
await server.connect(transport);
```
**Transport selection:**
- **Streamable HTTP**: Web services, remote access, multiple clients
- **stdio**: Command-line tools, local development, subprocess integration
### Notification Support
Notify clients when server state changes:
```typescript
// Notify when tools list changes
server.notification({
method: "notifications/tools/list_changed"
});
// Notify when resources change
server.notification({
method: "notifications/resources/list_changed"
});
```
Use notifications sparingly - only when server capabilities genuinely change.
---
## Code Best Practices
### Code Composability and Reusability
Your implementation MUST prioritize composability and code reuse:
1. **Extract Common Functionality**:
- Create reusable helper functions for operations used across multiple tools
- Build shared API clients for HTTP requests instead of duplicating code
- Centralize error handling logic in utility functions
- Extract business logic into dedicated functions that can be composed
- Extract shared markdown or JSON field selection & formatting functionality
2. **Avoid Duplication**:
- NEVER copy-paste similar code between tools
- If you find yourself writing similar logic twice, extract it into a function
- Common operations like pagination, filtering, field selection, and formatting should be shared
- Authentication/authorization logic should be centralized
## Building and Running
Always build your TypeScript code before running:
```bash
# Build the project
npm run build
# Run the server
npm start
# Development with auto-reload
npm run dev
```
Always ensure `npm run build` completes successfully before considering the implementation complete.
## Quality Checklist
Before finalizing your Node/TypeScript MCP server implementation, ensure:
### Strategic Design
- [ ] Tools enable complete workflows, not just API endpoint wrappers
- [ ] Tool names reflect natural task subdivisions
- [ ] Response formats optimize for agent context efficiency
- [ ] Human-readable identifiers used where appropriate
- [ ] Error messages guide agents toward correct usage
### Implementation Quality
- [ ] FOCUSED IMPLEMENTATION: Most important and valuable tools implemented
- [ ] All tools registered using `registerTool` with complete configuration
- [ ] All tools include `title`, `description`, `inputSchema`, and `annotations`
- [ ] Annotations correctly set (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- [ ] All tools use Zod schemas for runtime input validation with `.strict()` enforcement
- [ ] All Zod schemas have proper constraints and descriptive error messages
- [ ] All tools have comprehensive descriptions with explicit input/output types
- [ ] Descriptions include return value examples and complete schema documentation
- [ ] Error messages are clear, actionable, and educational
### TypeScript Quality
- [ ] TypeScript interfaces are defined for all data structures
- [ ] Strict TypeScript is enabled in tsconfig.json
- [ ] No use of `any` type - use `unknown` or proper types instead
- [ ] All async functions have explicit Promise<T> return types
- [ ] Error handling uses proper type guards (e.g., `axios.isAxiosError`, `z.ZodError`)
### Advanced Features (where applicable)
- [ ] Resources registered for appropriate data endpoints
- [ ] Appropriate transport configured (stdio or streamable HTTP)
- [ ] Notifications implemented for dynamic server capabilities
- [ ] Type-safe with SDK interfaces
### Project Configuration
- [ ] Package.json includes all necessary dependencies
- [ ] Build script produces working JavaScript in dist/ directory
- [ ] Main entry point is properly configured as dist/index.js
- [ ] Server name follows format: `{service}-mcp-server`
- [ ] tsconfig.json properly configured with strict mode
### Code Quality
- [ ] Pagination is properly implemented where applicable
- [ ] Large responses check CHARACTER_LIMIT constant and truncate with clear messages
- [ ] Filtering options are provided for potentially large result sets
- [ ] All network operations handle timeouts and connection errors gracefully
- [ ] Common functionality is extracted into reusable functions
- [ ] Return types are consistent across similar operations
### Testing and Build
- [ ] `npm run build` completes successfully without errors
- [ ] dist/index.js created and executable
- [ ] Server runs: `node dist/index.js --help`
- [ ] All imports resolve correctly
- [ ] Sample tool calls work as expected
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/reference/python_mcp_server.md
+719
View File
@@ -0,0 +1,719 @@
# Python MCP Server Implementation Guide
## Overview
This document provides Python-specific best practices and examples for implementing MCP servers using the MCP Python SDK. It covers server setup, tool registration patterns, input validation with Pydantic, error handling, and complete working examples.
---
## Quick Reference
### Key Imports
```python
from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field, field_validator, ConfigDict
from typing import Optional, List, Dict, Any
from enum import Enum
import httpx
```
### Server Initialization
```python
mcp = FastMCP("service_mcp")
```
### Tool Registration Pattern
```python
@mcp.tool(name="tool_name", annotations={...})
async def tool_function(params: InputModel) -> str:
# Implementation
pass
```
---
## MCP Python SDK and FastMCP
The official MCP Python SDK provides FastMCP, a high-level framework for building MCP servers. It provides:
- Automatic description and inputSchema generation from function signatures and docstrings
- Pydantic model integration for input validation
- Decorator-based tool registration with `@mcp.tool`
**For complete SDK documentation, use WebFetch to load:**
`https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
## Server Naming Convention
Python MCP servers must follow this naming pattern:
- **Format**: `{service}_mcp` (lowercase with underscores)
- **Examples**: `github_mcp`, `jira_mcp`, `stripe_mcp`
The name should be:
- General (not tied to specific features)
- Descriptive of the service/API being integrated
- Easy to infer from the task description
- Without version numbers or dates
## Tool Implementation
### Tool Naming
Use snake_case for tool names (e.g., "search_users", "create_project", "get_channel_info") with clear, action-oriented names.
**Avoid Naming Conflicts**: Include the service context to prevent overlaps:
- Use "slack_send_message" instead of just "send_message"
- Use "github_create_issue" instead of just "create_issue"
- Use "asana_list_tasks" instead of just "list_tasks"
### Tool Structure with FastMCP
Tools are defined using the `@mcp.tool` decorator with Pydantic models for input validation:
```python
from pydantic import BaseModel, Field, ConfigDict
from mcp.server.fastmcp import FastMCP
# Initialize the MCP server
mcp = FastMCP("example_mcp")
# Define Pydantic model for input validation
class ServiceToolInput(BaseModel):
'''Input model for service tool operation.'''
model_config = ConfigDict(
str_strip_whitespace=True, # Auto-strip whitespace from strings
validate_assignment=True, # Validate on assignment
extra='forbid' # Forbid extra fields
)
param1: str = Field(..., description="First parameter description (e.g., 'user123', 'project-abc')", min_length=1, max_length=100)
param2: Optional[int] = Field(default=None, description="Optional integer parameter with constraints", ge=0, le=1000)
tags: Optional[List[str]] = Field(default_factory=list, description="List of tags to apply", max_items=10)
@mcp.tool(
name="service_tool_name",
annotations={
"title": "Human-Readable Tool Title",
"readOnlyHint": True, # Tool does not modify environment
"destructiveHint": False, # Tool does not perform destructive operations
"idempotentHint": True, # Repeated calls have no additional effect
"openWorldHint": False # Tool does not interact with external entities
}
)
async def service_tool_name(params: ServiceToolInput) -> str:
'''Tool description automatically becomes the 'description' field.
This tool performs a specific operation on the service. It validates all inputs
using the ServiceToolInput Pydantic model before processing.
Args:
params (ServiceToolInput): Validated input parameters containing:
- param1 (str): First parameter description
- param2 (Optional[int]): Optional parameter with default
- tags (Optional[List[str]]): List of tags
Returns:
str: JSON-formatted response containing operation results
'''
# Implementation here
pass
```
## Pydantic v2 Key Features
- Use `model_config` instead of nested `Config` class
- Use `field_validator` instead of deprecated `validator`
- Use `model_dump()` instead of deprecated `dict()`
- Validators require `@classmethod` decorator
- Type hints are required for validator methods
```python
from pydantic import BaseModel, Field, field_validator, ConfigDict
class CreateUserInput(BaseModel):
model_config = ConfigDict(
str_strip_whitespace=True,
validate_assignment=True
)
name: str = Field(..., description="User's full name", min_length=1, max_length=100)
email: str = Field(..., description="User's email address", pattern=r'^[\w\.-]+@[\w\.-]+\.\w+$')
age: int = Field(..., description="User's age", ge=0, le=150)
@field_validator('email')
@classmethod
def validate_email(cls, v: str) -> str:
if not v.strip():
raise ValueError("Email cannot be empty")
return v.lower()
```
## Response Format Options
Support multiple output formats for flexibility:
```python
from enum import Enum
class ResponseFormat(str, Enum):
'''Output format for tool responses.'''
MARKDOWN = "markdown"
JSON = "json"
class UserSearchInput(BaseModel):
query: str = Field(..., description="Search query")
response_format: ResponseFormat = Field(
default=ResponseFormat.MARKDOWN,
description="Output format: 'markdown' for human-readable or 'json' for machine-readable"
)
```
**Markdown format**:
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format (e.g., "2024-01-15 10:30:00 UTC" instead of epoch)
- Show display names with IDs in parentheses (e.g., "@john.doe (U123456)")
- Omit verbose metadata (e.g., show only one profile image URL, not all sizes)
- Group related information logically
**JSON format**:
- Return complete, structured data suitable for programmatic processing
- Include all available fields and metadata
- Use consistent field names and types
## Pagination Implementation
For tools that list resources:
```python
class ListInput(BaseModel):
limit: Optional[int] = Field(default=20, description="Maximum results to return", ge=1, le=100)
offset: Optional[int] = Field(default=0, description="Number of results to skip for pagination", ge=0)
async def list_items(params: ListInput) -> str:
# Make API request with pagination
data = await api_request(limit=params.limit, offset=params.offset)
# Return pagination info
response = {
"total": data["total"],
"count": len(data["items"]),
"offset": params.offset,
"items": data["items"],
"has_more": data["total"] > params.offset + len(data["items"]),
"next_offset": params.offset + len(data["items"]) if data["total"] > params.offset + len(data["items"]) else None
}
return json.dumps(response, indent=2)
```
## Error Handling
Provide clear, actionable error messages:
```python
def _handle_api_error(e: Exception) -> str:
'''Consistent error formatting across all tools.'''
if isinstance(e, httpx.HTTPStatusError):
if e.response.status_code == 404:
return "Error: Resource not found. Please check the ID is correct."
elif e.response.status_code == 403:
return "Error: Permission denied. You don't have access to this resource."
elif e.response.status_code == 429:
return "Error: Rate limit exceeded. Please wait before making more requests."
return f"Error: API request failed with status {e.response.status_code}"
elif isinstance(e, httpx.TimeoutException):
return "Error: Request timed out. Please try again."
return f"Error: Unexpected error occurred: {type(e).__name__}"
```
## Shared Utilities
Extract common functionality into reusable functions:
```python
# Shared API request function
async def _make_api_request(endpoint: str, method: str = "GET", **kwargs) -> dict:
'''Reusable function for all API calls.'''
async with httpx.AsyncClient() as client:
response = await client.request(
method,
f"{API_BASE_URL}/{endpoint}",
timeout=30.0,
**kwargs
)
response.raise_for_status()
return response.json()
```
## Async/Await Best Practices
Always use async/await for network requests and I/O operations:
```python
# Good: Async network request
async def fetch_data(resource_id: str) -> dict:
async with httpx.AsyncClient() as client:
response = await client.get(f"{API_URL}/resource/{resource_id}")
response.raise_for_status()
return response.json()
# Bad: Synchronous request
def fetch_data(resource_id: str) -> dict:
response = requests.get(f"{API_URL}/resource/{resource_id}") # Blocks
return response.json()
```
## Type Hints
Use type hints throughout:
```python
from typing import Optional, List, Dict, Any
async def get_user(user_id: str) -> Dict[str, Any]:
data = await fetch_user(user_id)
return {"id": data["id"], "name": data["name"]}
```
## Tool Docstrings
Every tool must have comprehensive docstrings with explicit type information:
```python
async def search_users(params: UserSearchInput) -> str:
'''
Search for users in the Example system by name, email, or team.
This tool searches across all user profiles in the Example platform,
supporting partial matches and various search filters. It does NOT
create or modify users, only searches existing ones.
Args:
params (UserSearchInput): Validated input parameters containing:
- query (str): Search string to match against names/emails (e.g., "john", "@example.com", "team:marketing")
- limit (Optional[int]): Maximum results to return, between 1-100 (default: 20)
- offset (Optional[int]): Number of results to skip for pagination (default: 0)
Returns:
str: JSON-formatted string containing search results with the following schema:
Success response:
{
"total": int, # Total number of matches found
"count": int, # Number of results in this response
"offset": int, # Current pagination offset
"users": [
{
"id": str, # User ID (e.g., "U123456789")
"name": str, # Full name (e.g., "John Doe")
"email": str, # Email address (e.g., "john@example.com")
"team": str # Team name (e.g., "Marketing") - optional
}
]
}
Error response:
"Error: <error message>" or "No users found matching '<query>'"
Examples:
- Use when: "Find all marketing team members" -> params with query="team:marketing"
- Use when: "Search for John's account" -> params with query="john"
- Don't use when: You need to create a user (use example_create_user instead)
- Don't use when: You have a user ID and need full details (use example_get_user instead)
Error Handling:
- Input validation errors are handled by Pydantic model
- Returns "Error: Rate limit exceeded" if too many requests (429 status)
- Returns "Error: Invalid API authentication" if API key is invalid (401 status)
- Returns formatted list of results or "No users found matching 'query'"
'''
```
## Complete Example
See below for a complete Python MCP server example:
```python
#!/usr/bin/env python3
'''
MCP Server for Example Service.
This server provides tools to interact with Example API, including user search,
project management, and data export capabilities.
'''
from typing import Optional, List, Dict, Any
from enum import Enum
import httpx
from pydantic import BaseModel, Field, field_validator, ConfigDict
from mcp.server.fastmcp import FastMCP
# Initialize the MCP server
mcp = FastMCP("example_mcp")
# Constants
API_BASE_URL = "https://api.example.com/v1"
# Enums
class ResponseFormat(str, Enum):
'''Output format for tool responses.'''
MARKDOWN = "markdown"
JSON = "json"
# Pydantic Models for Input Validation
class UserSearchInput(BaseModel):
'''Input model for user search operations.'''
model_config = ConfigDict(
str_strip_whitespace=True,
validate_assignment=True
)
query: str = Field(..., description="Search string to match against names/emails", min_length=2, max_length=200)
limit: Optional[int] = Field(default=20, description="Maximum results to return", ge=1, le=100)
offset: Optional[int] = Field(default=0, description="Number of results to skip for pagination", ge=0)
response_format: ResponseFormat = Field(default=ResponseFormat.MARKDOWN, description="Output format")
@field_validator('query')
@classmethod
def validate_query(cls, v: str) -> str:
if not v.strip():
raise ValueError("Query cannot be empty or whitespace only")
return v.strip()
# Shared utility functions
async def _make_api_request(endpoint: str, method: str = "GET", **kwargs) -> dict:
'''Reusable function for all API calls.'''
async with httpx.AsyncClient() as client:
response = await client.request(
method,
f"{API_BASE_URL}/{endpoint}",
timeout=30.0,
**kwargs
)
response.raise_for_status()
return response.json()
def _handle_api_error(e: Exception) -> str:
'''Consistent error formatting across all tools.'''
if isinstance(e, httpx.HTTPStatusError):
if e.response.status_code == 404:
return "Error: Resource not found. Please check the ID is correct."
elif e.response.status_code == 403:
return "Error: Permission denied. You don't have access to this resource."
elif e.response.status_code == 429:
return "Error: Rate limit exceeded. Please wait before making more requests."
return f"Error: API request failed with status {e.response.status_code}"
elif isinstance(e, httpx.TimeoutException):
return "Error: Request timed out. Please try again."
return f"Error: Unexpected error occurred: {type(e).__name__}"
# Tool definitions
@mcp.tool(
name="example_search_users",
annotations={
"title": "Search Example Users",
"readOnlyHint": True,
"destructiveHint": False,
"idempotentHint": True,
"openWorldHint": True
}
)
async def example_search_users(params: UserSearchInput) -> str:
'''Search for users in the Example system by name, email, or team.
[Full docstring as shown above]
'''
try:
# Make API request using validated parameters
data = await _make_api_request(
"users/search",
params={
"q": params.query,
"limit": params.limit,
"offset": params.offset
}
)
users = data.get("users", [])
total = data.get("total", 0)
if not users:
return f"No users found matching '{params.query}'"
# Format response based on requested format
if params.response_format == ResponseFormat.MARKDOWN:
lines = [f"# User Search Results: '{params.query}'", ""]
lines.append(f"Found {total} users (showing {len(users)})")
lines.append("")
for user in users:
lines.append(f"## {user['name']} ({user['id']})")
lines.append(f"- **Email**: {user['email']}")
if user.get('team'):
lines.append(f"- **Team**: {user['team']}")
lines.append("")
return "\n".join(lines)
else:
# Machine-readable JSON format
import json
response = {
"total": total,
"count": len(users),
"offset": params.offset,
"users": users
}
return json.dumps(response, indent=2)
except Exception as e:
return _handle_api_error(e)
if __name__ == "__main__":
mcp.run()
```
---
## Advanced FastMCP Features
### Context Parameter Injection
FastMCP can automatically inject a `Context` parameter into tools for advanced capabilities like logging, progress reporting, resource reading, and user interaction:
```python
from mcp.server.fastmcp import FastMCP, Context
mcp = FastMCP("example_mcp")
@mcp.tool()
async def advanced_search(query: str, ctx: Context) -> str:
'''Advanced tool with context access for logging and progress.'''
# Report progress for long operations
await ctx.report_progress(0.25, "Starting search...")
# Log information for debugging
await ctx.log_info("Processing query", {"query": query, "timestamp": datetime.now()})
# Perform search
results = await search_api(query)
await ctx.report_progress(0.75, "Formatting results...")
# Access server configuration
server_name = ctx.fastmcp.name
return format_results(results)
@mcp.tool()
async def interactive_tool(resource_id: str, ctx: Context) -> str:
'''Tool that can request additional input from users.'''
# Request sensitive information when needed
api_key = await ctx.elicit(
prompt="Please provide your API key:",
input_type="password"
)
# Use the provided key
return await api_call(resource_id, api_key)
```
**Context capabilities:**
- `ctx.report_progress(progress, message)` - Report progress for long operations
- `ctx.log_info(message, data)` / `ctx.log_error()` / `ctx.log_debug()` - Logging
- `ctx.elicit(prompt, input_type)` - Request input from users
- `ctx.fastmcp.name` - Access server configuration
- `ctx.read_resource(uri)` - Read MCP resources
### Resource Registration
Expose data as resources for efficient, template-based access:
```python
@mcp.resource("file://documents/{name}")
async def get_document(name: str) -> str:
'''Expose documents as MCP resources.
Resources are useful for static or semi-static data that doesn't
require complex parameters. They use URI templates for flexible access.
'''
document_path = f"./docs/{name}"
with open(document_path, "r") as f:
return f.read()
@mcp.resource("config://settings/{key}")
async def get_setting(key: str, ctx: Context) -> str:
'''Expose configuration as resources with context.'''
settings = await load_settings()
return json.dumps(settings.get(key, {}))
```
**When to use Resources vs Tools:**
- **Resources**: For data access with simple parameters (URI templates)
- **Tools**: For complex operations with validation and business logic
### Structured Output Types
FastMCP supports multiple return types beyond strings:
```python
from typing import TypedDict
from dataclasses import dataclass
from pydantic import BaseModel
# TypedDict for structured returns
class UserData(TypedDict):
id: str
name: str
email: str
@mcp.tool()
async def get_user_typed(user_id: str) -> UserData:
'''Returns structured data - FastMCP handles serialization.'''
return {"id": user_id, "name": "John Doe", "email": "john@example.com"}
# Pydantic models for complex validation
class DetailedUser(BaseModel):
id: str
name: str
email: str
created_at: datetime
metadata: Dict[str, Any]
@mcp.tool()
async def get_user_detailed(user_id: str) -> DetailedUser:
'''Returns Pydantic model - automatically generates schema.'''
user = await fetch_user(user_id)
return DetailedUser(**user)
```
### Lifespan Management
Initialize resources that persist across requests:
```python
from contextlib import asynccontextmanager
@asynccontextmanager
async def app_lifespan():
'''Manage resources that live for the server's lifetime.'''
# Initialize connections, load config, etc.
db = await connect_to_database()
config = load_configuration()
# Make available to all tools
yield {"db": db, "config": config}
# Cleanup on shutdown
await db.close()
mcp = FastMCP("example_mcp", lifespan=app_lifespan)
@mcp.tool()
async def query_data(query: str, ctx: Context) -> str:
'''Access lifespan resources through context.'''
db = ctx.request_context.lifespan_state["db"]
results = await db.query(query)
return format_results(results)
```
### Transport Options
FastMCP supports two main transport mechanisms:
```python
# stdio transport (for local tools) - default
if __name__ == "__main__":
mcp.run()
# Streamable HTTP transport (for remote servers)
if __name__ == "__main__":
mcp.run(transport="streamable_http", port=8000)
```
**Transport selection:**
- **stdio**: Command-line tools, local integrations, subprocess execution
- **Streamable HTTP**: Web services, remote access, multiple clients
---
## Code Best Practices
### Code Composability and Reusability
Your implementation MUST prioritize composability and code reuse:
1. **Extract Common Functionality**:
- Create reusable helper functions for operations used across multiple tools
- Build shared API clients for HTTP requests instead of duplicating code
- Centralize error handling logic in utility functions
- Extract business logic into dedicated functions that can be composed
- Extract shared markdown or JSON field selection & formatting functionality
2. **Avoid Duplication**:
- NEVER copy-paste similar code between tools
- If you find yourself writing similar logic twice, extract it into a function
- Common operations like pagination, filtering, field selection, and formatting should be shared
- Authentication/authorization logic should be centralized
### Python-Specific Best Practices
1. **Use Type Hints**: Always include type annotations for function parameters and return values
2. **Pydantic Models**: Define clear Pydantic models for all input validation
3. **Avoid Manual Validation**: Let Pydantic handle input validation with constraints
4. **Proper Imports**: Group imports (standard library, third-party, local)
5. **Error Handling**: Use specific exception types (httpx.HTTPStatusError, not generic Exception)
6. **Async Context Managers**: Use `async with` for resources that need cleanup
7. **Constants**: Define module-level constants in UPPER_CASE
## Quality Checklist
Before finalizing your Python MCP server implementation, ensure:
### Strategic Design
- [ ] Tools enable complete workflows, not just API endpoint wrappers
- [ ] Tool names reflect natural task subdivisions
- [ ] Response formats optimize for agent context efficiency
- [ ] Human-readable identifiers used where appropriate
- [ ] Error messages guide agents toward correct usage
### Implementation Quality
- [ ] FOCUSED IMPLEMENTATION: Most important and valuable tools implemented
- [ ] All tools have descriptive names and documentation
- [ ] Return types are consistent across similar operations
- [ ] Error handling is implemented for all external calls
- [ ] Server name follows format: `{service}_mcp`
- [ ] All network operations use async/await
- [ ] Common functionality is extracted into reusable functions
- [ ] Error messages are clear, actionable, and educational
- [ ] Outputs are properly validated and formatted
### Tool Configuration
- [ ] All tools implement 'name' and 'annotations' in the decorator
- [ ] Annotations correctly set (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- [ ] All tools use Pydantic BaseModel for input validation with Field() definitions
- [ ] All Pydantic Fields have explicit types and descriptions with constraints
- [ ] All tools have comprehensive docstrings with explicit input/output types
- [ ] Docstrings include complete schema structure for dict/JSON returns
- [ ] Pydantic models handle input validation (no manual validation needed)
### Advanced Features (where applicable)
- [ ] Context injection used for logging, progress, or elicitation
- [ ] Resources registered for appropriate data endpoints
- [ ] Lifespan management implemented for persistent connections
- [ ] Structured output types used (TypedDict, Pydantic models)
- [ ] Appropriate transport configured (stdio or streamable HTTP)
### Code Quality
- [ ] File includes proper imports including Pydantic imports
- [ ] Pagination is properly implemented where applicable
- [ ] Filtering options are provided for potentially large result sets
- [ ] All async functions are properly defined with `async def`
- [ ] HTTP client usage follows async patterns with proper context managers
- [ ] Type hints are used throughout the code
- [ ] Constants are defined at module level in UPPER_CASE
### Testing
- [ ] Server runs successfully: `python your_server.py --help`
- [ ] All imports resolve correctly
- [ ] Sample tool calls work as expected
- [ ] Error scenarios handled gracefully
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/scripts/connections.py
+151
View File
@@ -0,0 +1,151 @@
"""Lightweight connection handling for MCP servers."""
from abc import ABC, abstractmethod
from contextlib import AsyncExitStack
from typing import Any
from mcp import ClientSession, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
from mcp.client.streamable_http import streamablehttp_client
class MCPConnection(ABC):
"""Base class for MCP server connections."""
def __init__(self):
self.session = None
self._stack = None
@abstractmethod
def _create_context(self):
"""Create the connection context based on connection type."""
async def __aenter__(self):
"""Initialize MCP server connection."""
self._stack = AsyncExitStack()
await self._stack.__aenter__()
try:
ctx = self._create_context()
result = await self._stack.enter_async_context(ctx)
if len(result) == 2:
read, write = result
elif len(result) == 3:
read, write, _ = result
else:
raise ValueError(f"Unexpected context result: {result}")
session_ctx = ClientSession(read, write)
self.session = await self._stack.enter_async_context(session_ctx)
await self.session.initialize()
return self
except BaseException:
await self._stack.__aexit__(None, None, None)
raise
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Clean up MCP server connection resources."""
if self._stack:
await self._stack.__aexit__(exc_type, exc_val, exc_tb)
self.session = None
self._stack = None
async def list_tools(self) -> list[dict[str, Any]]:
"""Retrieve available tools from the MCP server."""
response = await self.session.list_tools()
return [
{
"name": tool.name,
"description": tool.description,
"input_schema": tool.inputSchema,
}
for tool in response.tools
]
async def call_tool(self, tool_name: str, arguments: dict[str, Any]) -> Any:
"""Call a tool on the MCP server with provided arguments."""
result = await self.session.call_tool(tool_name, arguments=arguments)
return result.content
class MCPConnectionStdio(MCPConnection):
"""MCP connection using standard input/output."""
def __init__(self, command: str, args: list[str] = None, env: dict[str, str] = None):
super().__init__()
self.command = command
self.args = args or []
self.env = env
def _create_context(self):
return stdio_client(
StdioServerParameters(command=self.command, args=self.args, env=self.env)
)
class MCPConnectionSSE(MCPConnection):
"""MCP connection using Server-Sent Events."""
def __init__(self, url: str, headers: dict[str, str] = None):
super().__init__()
self.url = url
self.headers = headers or {}
def _create_context(self):
return sse_client(url=self.url, headers=self.headers)
class MCPConnectionHTTP(MCPConnection):
"""MCP connection using Streamable HTTP."""
def __init__(self, url: str, headers: dict[str, str] = None):
super().__init__()
self.url = url
self.headers = headers or {}
def _create_context(self):
return streamablehttp_client(url=self.url, headers=self.headers)
def create_connection(
transport: str,
command: str = None,
args: list[str] = None,
env: dict[str, str] = None,
url: str = None,
headers: dict[str, str] = None,
) -> MCPConnection:
"""Factory function to create the appropriate MCP connection.
Args:
transport: Connection type ("stdio", "sse", or "http")
command: Command to run (stdio only)
args: Command arguments (stdio only)
env: Environment variables (stdio only)
url: Server URL (sse and http only)
headers: HTTP headers (sse and http only)
Returns:
MCPConnection instance
"""
transport = transport.lower()
if transport == "stdio":
if not command:
raise ValueError("Command is required for stdio transport")
return MCPConnectionStdio(command=command, args=args, env=env)
elif transport == "sse":
if not url:
raise ValueError("URL is required for sse transport")
return MCPConnectionSSE(url=url, headers=headers)
elif transport in ["http", "streamable_http", "streamable-http"]:
if not url:
raise ValueError("URL is required for http transport")
return MCPConnectionHTTP(url=url, headers=headers)
else:
raise ValueError(f"Unsupported transport type: {transport}. Use 'stdio', 'sse', or 'http'")
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/scripts/evaluation.py
+373
View File
@@ -0,0 +1,373 @@
"""MCP Server Evaluation Harness
This script evaluates MCP servers by running test questions against them using Claude.
"""
import argparse
import asyncio
import json
import re
import sys
import time
import traceback
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any
from anthropic import Anthropic
from connections import create_connection
EVALUATION_PROMPT = """You are an AI assistant with access to tools.
When given a task, you MUST:
1. Use the available tools to complete the task
2. Provide summary of each step in your approach, wrapped in <summary> tags
3. Provide feedback on the tools provided, wrapped in <feedback> tags
4. Provide your final response, wrapped in <response> tags
Summary Requirements:
- In your <summary> tags, you must explain:
- The steps you took to complete the task
- Which tools you used, in what order, and why
- The inputs you provided to each tool
- The outputs you received from each tool
- A summary for how you arrived at the response
Feedback Requirements:
- In your <feedback> tags, provide constructive feedback on the tools:
- Comment on tool names: Are they clear and descriptive?
- Comment on input parameters: Are they well-documented? Are required vs optional parameters clear?
- Comment on descriptions: Do they accurately describe what the tool does?
- Comment on any errors encountered during tool usage: Did the tool fail to execute? Did the tool return too many tokens?
- Identify specific areas for improvement and explain WHY they would help
- Be specific and actionable in your suggestions
Response Requirements:
- Your response should be concise and directly address what was asked
- Always wrap your final response in <response> tags
- If you cannot solve the task return <response>NOT_FOUND</response>
- For numeric responses, provide just the number
- For IDs, provide just the ID
- For names or text, provide the exact text requested
- Your response should go last"""
def parse_evaluation_file(file_path: Path) -> list[dict[str, Any]]:
"""Parse XML evaluation file with qa_pair elements."""
try:
tree = ET.parse(file_path)
root = tree.getroot()
evaluations = []
for qa_pair in root.findall(".//qa_pair"):
question_elem = qa_pair.find("question")
answer_elem = qa_pair.find("answer")
if question_elem is not None and answer_elem is not None:
evaluations.append({
"question": (question_elem.text or "").strip(),
"answer": (answer_elem.text or "").strip(),
})
return evaluations
except Exception as e:
print(f"Error parsing evaluation file {file_path}: {e}")
return []
def extract_xml_content(text: str, tag: str) -> str | None:
"""Extract content from XML tags."""
pattern = rf"<{tag}>(.*?)</{tag}>"
matches = re.findall(pattern, text, re.DOTALL)
return matches[-1].strip() if matches else None
async def agent_loop(
client: Anthropic,
model: str,
question: str,
tools: list[dict[str, Any]],
connection: Any,
) -> tuple[str, dict[str, Any]]:
"""Run the agent loop with MCP tools."""
messages = [{"role": "user", "content": question}]
response = await asyncio.to_thread(
client.messages.create,
model=model,
max_tokens=4096,
system=EVALUATION_PROMPT,
messages=messages,
tools=tools,
)
messages.append({"role": "assistant", "content": response.content})
tool_metrics = {}
while response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_start_ts = time.time()
try:
tool_result = await connection.call_tool(tool_name, tool_input)
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
except Exception as e:
tool_response = f"Error executing tool {tool_name}: {str(e)}\n"
tool_response += traceback.format_exc()
tool_duration = time.time() - tool_start_ts
if tool_name not in tool_metrics:
tool_metrics[tool_name] = {"count": 0, "durations": []}
tool_metrics[tool_name]["count"] += 1
tool_metrics[tool_name]["durations"].append(tool_duration)
messages.append({
"role": "user",
"content": [{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_response,
}]
})
response = await asyncio.to_thread(
client.messages.create,
model=model,
max_tokens=4096,
system=EVALUATION_PROMPT,
messages=messages,
tools=tools,
)
messages.append({"role": "assistant", "content": response.content})
response_text = next(
(block.text for block in response.content if hasattr(block, "text")),
None,
)
return response_text, tool_metrics
async def evaluate_single_task(
client: Anthropic,
model: str,
qa_pair: dict[str, Any],
tools: list[dict[str, Any]],
connection: Any,
task_index: int,
) -> dict[str, Any]:
"""Evaluate a single QA pair with the given tools."""
start_time = time.time()
print(f"Task {task_index + 1}: Running task with question: {qa_pair['question']}")
response, tool_metrics = await agent_loop(client, model, qa_pair["question"], tools, connection)
response_value = extract_xml_content(response, "response")
summary = extract_xml_content(response, "summary")
feedback = extract_xml_content(response, "feedback")
duration_seconds = time.time() - start_time
return {
"question": qa_pair["question"],
"expected": qa_pair["answer"],
"actual": response_value,
"score": int(response_value == qa_pair["answer"]) if response_value else 0,
"total_duration": duration_seconds,
"tool_calls": tool_metrics,
"num_tool_calls": sum(len(metrics["durations"]) for metrics in tool_metrics.values()),
"summary": summary,
"feedback": feedback,
}
REPORT_HEADER = """
# Evaluation Report
## Summary
- **Accuracy**: {correct}/{total} ({accuracy:.1f}%)
- **Average Task Duration**: {average_duration_s:.2f}s
- **Average Tool Calls per Task**: {average_tool_calls:.2f}
- **Total Tool Calls**: {total_tool_calls}
---
"""
TASK_TEMPLATE = """
### Task {task_num}
**Question**: {question}
**Ground Truth Answer**: `{expected_answer}`
**Actual Answer**: `{actual_answer}`
**Correct**: {correct_indicator}
**Duration**: {total_duration:.2f}s
**Tool Calls**: {tool_calls}
**Summary**
{summary}
**Feedback**
{feedback}
---
"""
async def run_evaluation(
eval_path: Path,
connection: Any,
model: str = "claude-3-7-sonnet-20250219",
) -> str:
"""Run evaluation with MCP server tools."""
print("🚀 Starting Evaluation")
client = Anthropic()
tools = await connection.list_tools()
print(f"📋 Loaded {len(tools)} tools from MCP server")
qa_pairs = parse_evaluation_file(eval_path)
print(f"📋 Loaded {len(qa_pairs)} evaluation tasks")
results = []
for i, qa_pair in enumerate(qa_pairs):
print(f"Processing task {i + 1}/{len(qa_pairs)}")
result = await evaluate_single_task(client, model, qa_pair, tools, connection, i)
results.append(result)
correct = sum(r["score"] for r in results)
accuracy = (correct / len(results)) * 100 if results else 0
average_duration_s = sum(r["total_duration"] for r in results) / len(results) if results else 0
average_tool_calls = sum(r["num_tool_calls"] for r in results) / len(results) if results else 0
total_tool_calls = sum(r["num_tool_calls"] for r in results)
report = REPORT_HEADER.format(
correct=correct,
total=len(results),
accuracy=accuracy,
average_duration_s=average_duration_s,
average_tool_calls=average_tool_calls,
total_tool_calls=total_tool_calls,
)
report += "".join([
TASK_TEMPLATE.format(
task_num=i + 1,
question=qa_pair["question"],
expected_answer=qa_pair["answer"],
actual_answer=result["actual"] or "N/A",
correct_indicator="✅" if result["score"] else "❌",
total_duration=result["total_duration"],
tool_calls=json.dumps(result["tool_calls"], indent=2),
summary=result["summary"] or "N/A",
feedback=result["feedback"] or "N/A",
)
for i, (qa_pair, result) in enumerate(zip(qa_pairs, results))
])
return report
def parse_headers(header_list: list[str]) -> dict[str, str]:
"""Parse header strings in format 'Key: Value' into a dictionary."""
headers = {}
if not header_list:
return headers
for header in header_list:
if ":" in header:
key, value = header.split(":", 1)
headers[key.strip()] = value.strip()
else:
print(f"Warning: Ignoring malformed header: {header}")
return headers
def parse_env_vars(env_list: list[str]) -> dict[str, str]:
"""Parse environment variable strings in format 'KEY=VALUE' into a dictionary."""
env = {}
if not env_list:
return env
for env_var in env_list:
if "=" in env_var:
key, value = env_var.split("=", 1)
env[key.strip()] = value.strip()
else:
print(f"Warning: Ignoring malformed environment variable: {env_var}")
return env
async def main():
parser = argparse.ArgumentParser(
description="Evaluate MCP servers using test questions",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Evaluate a local stdio MCP server
python evaluation.py -t stdio -c python -a my_server.py eval.xml
# Evaluate an SSE MCP server
python evaluation.py -t sse -u https://example.com/mcp -H "Authorization: Bearer token" eval.xml
# Evaluate an HTTP MCP server with custom model
python evaluation.py -t http -u https://example.com/mcp -m claude-3-5-sonnet-20241022 eval.xml
""",
)
parser.add_argument("eval_file", type=Path, help="Path to evaluation XML file")
parser.add_argument("-t", "--transport", choices=["stdio", "sse", "http"], default="stdio", help="Transport type (default: stdio)")
parser.add_argument("-m", "--model", default="claude-3-7-sonnet-20250219", help="Claude model to use (default: claude-3-7-sonnet-20250219)")
stdio_group = parser.add_argument_group("stdio options")
stdio_group.add_argument("-c", "--command", help="Command to run MCP server (stdio only)")
stdio_group.add_argument("-a", "--args", nargs="+", help="Arguments for the command (stdio only)")
stdio_group.add_argument("-e", "--env", nargs="+", help="Environment variables in KEY=VALUE format (stdio only)")
remote_group = parser.add_argument_group("sse/http options")
remote_group.add_argument("-u", "--url", help="MCP server URL (sse/http only)")
remote_group.add_argument("-H", "--header", nargs="+", dest="headers", help="HTTP headers in 'Key: Value' format (sse/http only)")
parser.add_argument("-o", "--output", type=Path, help="Output file for evaluation report (default: stdout)")
args = parser.parse_args()
if not args.eval_file.exists():
print(f"Error: Evaluation file not found: {args.eval_file}")
sys.exit(1)
headers = parse_headers(args.headers) if args.headers else None
env_vars = parse_env_vars(args.env) if args.env else None
try:
connection = create_connection(
transport=args.transport,
command=args.command,
args=args.args,
env=env_vars,
url=args.url,
headers=headers,
)
except ValueError as e:
print(f"Error: {e}")
sys.exit(1)
print(f"🔗 Connecting to MCP server via {args.transport}...")
async with connection:
print("✅ Connected successfully")
report = await run_evaluation(args.eval_file, connection, args.model)
if args.output:
args.output.write_text(report)
print(f"\n✅ Report saved to {args.output}")
else:
print("\n" + report)
if __name__ == "__main__":
asyncio.run(main())
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/scripts/example_evaluation.xml
+22
View File
@@ -0,0 +1,22 @@
<evaluation>
<qa_pair>
<question>Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years. What is the final amount in dollars (rounded to 2 decimal places)?</question>
<answer>11614.72</answer>
</qa_pair>
<qa_pair>
<question>A projectile is launched at a 45-degree angle with an initial velocity of 50 m/s. Calculate the total distance (in meters) it has traveled from the launch point after 2 seconds, assuming g=9.8 m/s². Round to 2 decimal places.</question>
<answer>87.25</answer>
</qa_pair>
<qa_pair>
<question>A sphere has a volume of 500 cubic meters. Calculate its surface area in square meters. Round to 2 decimal places.</question>
<answer>304.65</answer>
</qa_pair>
<qa_pair>
<question>Calculate the population standard deviation of this dataset: [12, 15, 18, 22, 25, 30, 35]. Round to 2 decimal places.</question>
<answer>7.61</answer>
</qa_pair>
<qa_pair>
<question>Calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Round to 2 decimal places.</question>
<answer>4.46</answer>
</qa_pair>
</evaluation>
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-builder/scripts/requirements.txt
+2
View File
@@ -0,0 +1,2 @@
anthropic>=0.39.0
mcp>=1.1.0
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/mcp-tool-developer/SKILL.md
+132
View File
@@ -0,0 +1,132 @@
---
name: mcp-tool-developer
description: "Build Model Context Protocol (MCP) servers and tools from scratch. Full-stack MCP development with TypeScript/Python, testing, deployment, and registry publishing."
category: developer-tools
risk: safe
source: community
source_repo: demo112/yunqu-ai-skills
source_type: community
date_added: "2026-05-13"
author: yundu-ai
tags: [mcp, ai-agent, tool-development, typescript, python, llm, model-context-protocol]
tools: [claude, cursor, gemini]
---
# MCP Tool Developer
## Overview
Expert at building Model Context Protocol (MCP) servers that give AI agents new capabilities. Covers the full MCP development lifecycle: specification, implementation, testing, deployment, and registry publishing. Supports both TypeScript and Python with production-ready patterns.
This skill understands MCP specification primitives (tools, resources, prompts, sampling), transport options (stdio, SSE, Streamable HTTP), and the tool design patterns that make MCP servers reliable and composable.
## When to Use This Skill
- Use when building a new MCP server from scratch
- Use when wrapping an existing API as an MCP tool
- Use when debugging MCP server issues
- Use when designing the tool schema for an MCP server
- Use when publishing an MCP server to a registry
## How It Works
### Step 1: Define the MCP Server Scope
Identify what capabilities the server should expose:
- **Tools** - Functions the LLM can call (primary use case)
- **Resources** - Data the LLM can read (files, APIs, databases)
- **Prompts** - Reusable prompt templates
Choose the transport:
- **stdio** - For local CLI tools (Claude Code, Cursor)
- **SSE (Server-Sent Events)** - For remote/hosted tools
- **Streamable HTTP** - New in MCP spec for modern deployments
### Step 2: Design the Tool Schema
Define input/output schemas before writing implementation:
```typescript
{
name: "tool_name",
description: "What this tool does (visible to the LLM)",
inputSchema: {
type: "object",
properties: { ... },
required: [ ... ]
}
}
```
### Step 3: Implement the Server
Create the server with proper error handling, validation, and logging. Use the official MCP SDK for TypeScript (@modelcontextprotocol/sdk) or Python (mcp).
### Step 4: Test and Deploy
Test with the MCP Inspector, validate tool schemas, handle edge cases, then deploy locally or remotely.
## Examples
### Example 1: TypeScript MCP Server
```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
const server = new McpServer({ name: "my-tools", version: "1.0.0" });
server.tool("greet", "Greet someone by name",
{ name: z.string().describe("Person's name") },
async ({ name }) => ({ content: [{ type: "text", text: `Hello, ${name}!` }] })
);
const transport = new StdioServerTransport();
await server.connect(transport);
```
### Example 2: API Wrapper Pattern
Wrap an external API as an MCP tool with auth, rate limiting, and error handling:
- Map API endpoints to tools
- Handle auth via environment variables
- Transform API responses to LLM-friendly format
- Add retry logic with exponential backoff
## Best Practices
- Build small, focused tools that can be chained rather than monolithic tools
- Return structured errors, not crashes - tools should fail gracefully
- Define schemas before implementation
- Include descriptions that help the LLM understand when and how to use each tool
- Validate all inputs against the schema
- Add rate limiting for external API calls
- Use environment variables for secrets, never hardcode credentials
## Limitations
- This skill provides guidance and code generation; actual runtime testing requires a development environment
- MCP specification is evolving; always check the latest spec version
- Security review is essential before deploying tools that handle sensitive data
## Security and Safety Notes
- Never hardcode API keys or credentials in tool implementations
- Use environment variables or secret managers for all authentication
- Validate and sanitize all inputs to prevent injection attacks
- Rate limit external API calls to prevent abuse
- Review tool permissions carefully - tools can access files, networks, and execute code
## Common Pitfalls
- **Problem:** LLM calls tools with wrong parameters
**Solution:** Improve tool descriptions and add examples in the description field. The LLM reads descriptions to decide how to call tools.
- **Problem:** Tool times out on large inputs
**Solution:** Add input size validation and pagination. Stream large responses instead of buffering.
## Related Skills
- `api-integration-architect` - For API design patterns used in MCP tools
- `security-audit-code-reviewer` - For reviewing MCP server code security
antigravity-awesome-skills/plugins/antigravity-bundle-aas-agent-mcp-builder/skills/rag-engineer/SKILL.md
+341
View File
@@ -0,0 +1,341 @@
---
name: rag-engineer
description: Expert in building Retrieval-Augmented Generation systems. Masters
embedding models, vector databases, chunking strategies, and retrieval
optimization for LLM applications.
risk: unknown
source: vibeship-spawner-skills (Apache 2.0)
date_added: 2026-02-27
---
# RAG Engineer
Expert in building Retrieval-Augmented Generation systems. Masters embedding models,
vector databases, chunking strategies, and retrieval optimization for LLM applications.
**Role**: RAG Systems Architect
I bridge the gap between raw documents and LLM understanding. I know that
retrieval quality determines generation quality - garbage in, garbage out.
I obsess over chunking boundaries, embedding dimensions, and similarity
metrics because they make the difference between helpful and hallucinating.
### Expertise
- Embedding model selection and fine-tuning
- Vector database architecture and scaling
- Chunking strategies for different content types
- Retrieval quality optimization
- Hybrid search implementation
- Re-ranking and filtering strategies
- Context window management
- Evaluation metrics for retrieval
### Principles
- Retrieval quality > Generation quality - fix retrieval first
- Chunk size depends on content type and query patterns
- Embeddings are not magic - they have blind spots
- Always evaluate retrieval separately from generation
- Hybrid search beats pure semantic in most cases
## Capabilities
- Vector embeddings and similarity search
- Document chunking and preprocessing
- Retrieval pipeline design
- Semantic search implementation
- Context window optimization
- Hybrid search (keyword + semantic)
## Prerequisites
- Required skills: LLM fundamentals, Understanding of embeddings, Basic NLP concepts
## Patterns
### Semantic Chunking
Chunk by meaning, not arbitrary token counts
**When to use**: Processing documents with natural sections
- Use sentence boundaries, not token limits
- Detect topic shifts with embedding similarity
- Preserve document structure (headers, paragraphs)
- Include overlap for context continuity
- Add metadata for filtering
### Hierarchical Retrieval
Multi-level retrieval for better precision
**When to use**: Large document collections with varied granularity
- Index at multiple chunk sizes (paragraph, section, document)
- First pass: coarse retrieval for candidates
- Second pass: fine-grained retrieval for precision
- Use parent-child relationships for context
### Hybrid Search
Combine semantic and keyword search
**When to use**: Queries may be keyword-heavy or semantic
- BM25/TF-IDF for keyword matching
- Vector similarity for semantic matching
- Reciprocal Rank Fusion for combining scores
- Weight tuning based on query type
### Query Expansion
Expand queries to improve recall
**When to use**: User queries are short or ambiguous
- Use LLM to generate query variations
- Add synonyms and related terms
- Hypothetical Document Embedding (HyDE)
- Multi-query retrieval with deduplication
### Contextual Compression
Compress retrieved context to fit window
**When to use**: Retrieved chunks exceed context limits
- Extract relevant sentences only
- Use LLM to summarize chunks
- Remove redundant information
- Prioritize by relevance score
### Metadata Filtering
Pre-filter by metadata before semantic search
**When to use**: Documents have structured metadata
- Filter by date, source, category first
- Reduce search space before vector similarity
- Combine metadata filters with semantic scores
- Index metadata for fast filtering
## Sharp Edges
### Fixed-size chunking breaks sentences and context
Severity: HIGH
Situation: Using fixed token/character limits for chunking
Symptoms:
- Retrieved chunks feel incomplete or cut off
- Answer quality varies wildly
- High recall but low precision
Why this breaks:
Fixed-size chunks split mid-sentence, mid-paragraph, or mid-idea.
The resulting embeddings represent incomplete thoughts, leading to
poor retrieval quality. Users search for concepts but get fragments.
Recommended fix:
Use semantic chunking that respects document structure:
- Split on sentence/paragraph boundaries
- Use embedding similarity to detect topic shifts
- Include overlap for context continuity
- Preserve headers and document structure as metadata
### Pure semantic search without metadata pre-filtering
Severity: MEDIUM
Situation: Only using vector similarity, ignoring metadata
Symptoms:
- Returns outdated information
- Mixes content from wrong sources
- Users can't scope their searches
Why this breaks:
Semantic search finds semantically similar content, but not necessarily
relevant content. Without metadata filtering, you return old docs when
user wants recent, wrong categories, or inapplicable content.
Recommended fix:
Implement hybrid filtering:
- Pre-filter by metadata (date, source, category) before vector search
- Post-filter results by relevance criteria
- Include metadata in the retrieval API
- Allow users to specify filters
### Using same embedding model for different content types
Severity: MEDIUM
Situation: One embedding model for code, docs, and structured data
Symptoms:
- Code search returns irrelevant results
- Domain terms not matched properly
- Similar concepts not clustered
Why this breaks:
Embedding models are trained on specific content types. Using a text
embedding model for code, or a general model for domain-specific
content, produces poor similarity matches.
Recommended fix:
Evaluate embeddings per content type:
- Use code-specific embeddings for code (e.g., CodeBERT)
- Consider domain-specific or fine-tuned embeddings
- Benchmark retrieval quality before choosing
- Separate indices for different content types if needed
### Using first-stage retrieval results directly
Severity: MEDIUM
Situation: Taking top-K from vector search without reranking
Symptoms:
- Clearly relevant docs not in top results
- Results order seems arbitrary
- Adding more results helps quality
Why this breaks:
First-stage retrieval (vector search) optimizes for recall, not precision.
The top results by embedding similarity may not be the most relevant
for the specific query. Cross-encoder reranking dramatically improves
precision for the final results.
Recommended fix:
Add reranking step:
- Retrieve larger candidate set (e.g., top 20-50)
- Rerank with cross-encoder (query-document pairs)
- Return reranked top-K (e.g., top 5)
- Cache reranker for performance
### Cramming maximum context into LLM prompt
Severity: MEDIUM
Situation: Using all retrieved context regardless of relevance
Symptoms:
- Answers drift with more context
- LLM ignores key information
- High token costs
Why this breaks:
More context isn't always better. Irrelevant context confuses the LLM,
increases latency and cost, and can cause the model to ignore the
most relevant information. Models have attention limits.
Recommended fix:
Use relevance thresholds:
- Set minimum similarity score cutoff
- Limit context to truly relevant chunks
- Summarize or compress if needed
- Order context by relevance
### Not measuring retrieval quality separately from generation
Severity: HIGH
Situation: Only evaluating end-to-end RAG quality
Symptoms:
- Can't diagnose poor RAG performance
- Prompt changes don't help
- Random quality variations
Why this breaks:
If answers are wrong, you can't tell if retrieval failed or generation
failed. This makes debugging impossible and leads to wrong fixes
(tuning prompts when retrieval is the problem).
Recommended fix:
Separate retrieval evaluation:
- Create retrieval test set with relevant docs labeled
- Measure MRR, NDCG, Recall@K for retrieval
- Evaluate generation only on correct retrievals
- Track metrics over time
### Not updating embeddings when source documents change
Severity: MEDIUM
Situation: Embeddings generated once, never refreshed
Symptoms:
- Returns outdated information
- References deleted content
- Inconsistent with source
Why this breaks:
Documents change but embeddings don't. Users retrieve outdated content
or, worse, content that no longer exists. This erodes trust in the
system.
Recommended fix:
Implement embedding refresh:
- Track document versions/hashes
- Re-embed on document change
- Handle deleted documents
- Consider TTL for embeddings
### Same retrieval strategy for all query types
Severity: MEDIUM
Situation: Using pure semantic search for keyword-heavy queries
Symptoms:
- Exact term searches miss results
- Concept searches too literal
- Users frustrated with both
Why this breaks:
Some queries are keyword-oriented (looking for specific terms) while
others are semantic (looking for concepts). Pure semantic search fails
on exact matches; pure keyword search fails on paraphrases.
Recommended fix:
Implement hybrid search:
- BM25/TF-IDF for keyword matching
- Vector similarity for semantic matching
- Reciprocal Rank Fusion to combine
- Tune weights based on query patterns
## Related Skills
Works well with: `ai-agents-architect`, `prompt-engineer`, `database-architect`, `backend`
## When to Use
- User mentions or implies: building RAG
- User mentions or implies: vector search
- User mentions or implies: embeddings
- User mentions or implies: semantic search
- User mentions or implies: document retrieval
- User mentions or implies: context retrieval
- User mentions or implies: knowledge base
- User mentions or implies: LLM with documents
- User mentions or implies: chunking strategy
- User mentions or implies: pinecone
- User mentions or implies: weaviate
- User mentions or implies: chromadb
- User mentions or implies: pgvector
## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.