📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
+602
@@ -0,0 +1,602 @@
|
||||
---
|
||||
name: llm-council
|
||||
description: "Run Fireworks-hosted open-weight model councils that compare responses and synthesize a final answer."
|
||||
allowed-tools: Read, Write, Bash, AskUserQuestion
|
||||
category: "ai-agents"
|
||||
risk: "safe"
|
||||
source: "official"
|
||||
source_repo: "dair-ai/dair-academy-plugins"
|
||||
source_type: "official"
|
||||
date_added: "2026-06-19"
|
||||
author: "DAIR.AI"
|
||||
license: "MIT"
|
||||
license_source: "https://github.com/dair-ai/dair-academy-plugins/blob/main/README.md#license"
|
||||
tags:
|
||||
- dair-academy
|
||||
- ai
|
||||
- workflow
|
||||
tools:
|
||||
- claude-code
|
||||
- codex-cli
|
||||
- cursor
|
||||
---
|
||||
|
||||
# LLM Council (Fireworks AI)
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when this workflow matches the user request: Use this skill for its documented workflow.
|
||||
|
||||
|
||||
_Source: [dair-ai/dair-academy-plugins](https://github.com/dair-ai/dair-academy-plugins) (MIT)._
|
||||
|
||||
This skill implements Karpathy's LLM Council concept where multiple open-weight LLMs deliberate on a query, powered entirely by Fireworks AI:
|
||||
|
||||
1. **Phase 1**: All models respond to the query independently (parallel)
|
||||
2. **Phase 2**: Models rank each other's anonymized responses
|
||||
3. **Phase 3**: A Chairman LLM synthesizes the final answer
|
||||
|
||||
All inference runs through **Fireworks AI** using open-weight models. The speed and pricing of Fireworks makes it practical to run multi-model deliberation that would be slow or expensive on other providers.
|
||||
|
||||
## CRITICAL RULES
|
||||
|
||||
1. **ALWAYS use AskUserQuestion** to let the user select council models (multiselect) and the Chairman model
|
||||
2. **ALWAYS save raw responses to files** - never summarize or truncate API outputs
|
||||
3. **ALWAYS show full transparency** - display all individual responses, all rankings, AND the final synthesis
|
||||
4. **NEVER skip the ranking phase** - it is essential to the council deliberation process
|
||||
5. **Read from files for display** - ensures content is shown unmodified
|
||||
6. **ALWAYS display the final output to the user** after Phase 3 completes
|
||||
|
||||
## Pre-flight Check
|
||||
|
||||
Before running any phase, verify the Fireworks API key is set:
|
||||
|
||||
```bash
|
||||
if [ -z "$FIREWORKS_API_KEY" ]; then
|
||||
echo "ERROR: FIREWORKS_API_KEY is not set."
|
||||
echo "Create a Fireworks AI account at: https://fireworks.ai/"
|
||||
echo "Then export it in your shell profile (~/.zshrc or ~/.bashrc):"
|
||||
echo ' export FIREWORKS_API_KEY="your_api_key_here"'
|
||||
exit 1
|
||||
fi
|
||||
echo "FIREWORKS_API_KEY is set."
|
||||
```
|
||||
|
||||
## Available Models
|
||||
|
||||
Present these options to the user via AskUserQuestion (multiselect):
|
||||
|
||||
| Model | Fireworks ID | Provider |
|
||||
|-------|-------------|----------|
|
||||
| GLM 5 | accounts/fireworks/models/glm-5 | Z.ai |
|
||||
| DeepSeek V3.1 | accounts/fireworks/models/deepseek-v3p1 | DeepSeek |
|
||||
| DeepSeek V3.2 | accounts/fireworks/models/deepseek-v3p2 | DeepSeek |
|
||||
| MiniMax M2.1 | accounts/fireworks/models/minimax-m2p1 | MiniMax |
|
||||
| Kimi K2.5 | accounts/fireworks/models/kimi-k2p5 | Moonshot |
|
||||
| Qwen3 235B | accounts/fireworks/models/qwen3-235b-a22b | Alibaba |
|
||||
| Llama 4 Maverick | accounts/fireworks/models/llama4-maverick-instruct-basic | Meta |
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Gather User Input
|
||||
|
||||
Use AskUserQuestion to get:
|
||||
1. The query/question for the council (or accept it from the conversation)
|
||||
2. Which models to include (multiselect, recommend 3-5 models)
|
||||
3. Which model should be the Chairman (single select)
|
||||
|
||||
Note: AskUserQuestion supports max 4 options per question. Since there are 7 models, split model selection across two questions, or show the most popular 4 and let the user type "Other" for the rest. A good default is to show 4 models in the first question and note the others are available via "Other". Rotate which models are shown based on variety.
|
||||
|
||||
Example AskUserQuestion for model selection (show 4, mention others):
|
||||
```
|
||||
question: "Which models should participate in the LLM Council? (Also available via Other: Llama 4 Maverick, Qwen3 235B, GLM 5)"
|
||||
header: "Models"
|
||||
multiSelect: true
|
||||
options:
|
||||
- label: "DeepSeek V3.2"
|
||||
description: "DeepSeek's newest and most capable model"
|
||||
- label: "MiniMax M2.1"
|
||||
description: "MiniMax's strong open-weight model"
|
||||
- label: "Kimi K2.5"
|
||||
description: "Moonshot's strong open-weight model"
|
||||
- label: "DeepSeek V3.1"
|
||||
description: "DeepSeek's proven reasoning model"
|
||||
```
|
||||
|
||||
Example AskUserQuestion for chairman:
|
||||
```
|
||||
question: "Which model should be the Chairman (synthesizes the final answer)?"
|
||||
header: "Chairman"
|
||||
multiSelect: false
|
||||
options:
|
||||
- label: "DeepSeek V3.2 (Recommended)"
|
||||
description: "Newest DeepSeek, strong at comprehensive analysis"
|
||||
- label: "GLM 5"
|
||||
description: "Strong reasoning for synthesis"
|
||||
- label: "Kimi K2.5"
|
||||
description: "Strong at structured synthesis"
|
||||
- label: "MiniMax M2.1"
|
||||
description: "Strong open-weight model for synthesis"
|
||||
```
|
||||
|
||||
### Model Name to ID Mapping
|
||||
|
||||
Use this mapping to convert user selections to Fireworks model IDs:
|
||||
|
||||
```python
|
||||
MODEL_MAP = {
|
||||
"GLM 5": "accounts/fireworks/models/glm-5",
|
||||
"DeepSeek V3.1": "accounts/fireworks/models/deepseek-v3p1",
|
||||
"DeepSeek V3.2": "accounts/fireworks/models/deepseek-v3p2",
|
||||
"MiniMax M2.1": "accounts/fireworks/models/minimax-m2p1",
|
||||
"Kimi K2.5": "accounts/fireworks/models/kimi-k2p5",
|
||||
"Qwen3 235B": "accounts/fireworks/models/qwen3-235b-a22b",
|
||||
"Llama 4 Maverick": "accounts/fireworks/models/llama4-maverick-instruct-basic",
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Run Phase 1 - Individual Responses
|
||||
|
||||
After gathering input, run this script to get responses from all selected models in parallel:
|
||||
|
||||
```bash
|
||||
QUERY="USER_QUERY_HERE"
|
||||
MODELS='["accounts/fireworks/models/glm-5", "accounts/fireworks/models/deepseek-v3p1"]'
|
||||
|
||||
python3 << 'PYEOF'
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
FIREWORKS_API_KEY = os.environ.get("FIREWORKS_API_KEY")
|
||||
API_URL = "https://api.fireworks.ai/inference/v1/chat/completions"
|
||||
|
||||
QUERY = os.environ.get("QUERY", "")
|
||||
MODELS = json.loads(os.environ.get("MODELS", "[]"))
|
||||
|
||||
# Create session directory
|
||||
timestamp = time.strftime("%Y%m%d-%H%M%S")
|
||||
SESSION_DIR = f"/tmp/llm-council/{timestamp}"
|
||||
os.makedirs(SESSION_DIR, exist_ok=True)
|
||||
|
||||
# Save config
|
||||
config = {"query": QUERY, "models": MODELS, "timestamp": timestamp}
|
||||
with open(f"{SESSION_DIR}/config.json", "w") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
|
||||
def call_model(model_id, query):
|
||||
"""Call a single model via Fireworks AI"""
|
||||
try:
|
||||
start = time.time()
|
||||
response = requests.post(
|
||||
API_URL,
|
||||
headers={
|
||||
"Authorization": f"Bearer {FIREWORKS_API_KEY}",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={
|
||||
"model": model_id,
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are participating in an LLM council deliberation. Provide your best, most thoughtful response to the query. Be comprehensive but focused."},
|
||||
{"role": "user", "content": query}
|
||||
],
|
||||
"max_tokens": 4000,
|
||||
"temperature": 1
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
response.raise_for_status()
|
||||
elapsed = time.time() - start
|
||||
data = response.json()
|
||||
usage = data.get("usage", {})
|
||||
return {
|
||||
"success": True,
|
||||
"content": data["choices"][0]["message"]["content"],
|
||||
"model": model_id,
|
||||
"latency_seconds": round(elapsed, 2),
|
||||
"tokens": {
|
||||
"prompt": usage.get("prompt_tokens", 0),
|
||||
"completion": usage.get("completion_tokens", 0),
|
||||
"total": usage.get("total_tokens", 0)
|
||||
}
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"content": f"[ERROR: {str(e)}]",
|
||||
"model": model_id,
|
||||
"latency_seconds": 0,
|
||||
"tokens": {"prompt": 0, "completion": 0, "total": 0}
|
||||
}
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("PHASE 1: Collecting Individual Responses")
|
||||
print(f"{'='*60}")
|
||||
print(f"Query: {QUERY[:200]}...")
|
||||
print(f"Models: {', '.join([m.split('/')[-1] for m in MODELS])}")
|
||||
print(f"Session: {SESSION_DIR}")
|
||||
print()
|
||||
|
||||
# Parallel execution
|
||||
results = {}
|
||||
with ThreadPoolExecutor(max_workers=len(MODELS)) as executor:
|
||||
futures = {executor.submit(call_model, m, QUERY): m for m in MODELS}
|
||||
for future in as_completed(futures):
|
||||
model = futures[future]
|
||||
result = future.result()
|
||||
results[model] = result
|
||||
status = "OK" if result["success"] else "FAILED"
|
||||
latency = f"{result['latency_seconds']}s" if result["success"] else "N/A"
|
||||
print(f" [{status}] {model.split('/')[-1]} ({latency})")
|
||||
|
||||
# Save raw results
|
||||
with open(f"{SESSION_DIR}/phase1_responses.json", "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
|
||||
print(f"\nPhase 1 complete. Results saved to: {SESSION_DIR}/phase1_responses.json")
|
||||
print(f"SESSION_DIR={SESSION_DIR}")
|
||||
PYEOF
|
||||
```
|
||||
|
||||
### Step 3: Run Phase 2 - Cross-Model Ranking
|
||||
|
||||
Each model reviews and ranks the anonymized responses from Phase 1:
|
||||
|
||||
```bash
|
||||
SESSION_DIR="/tmp/llm-council/TIMESTAMP_HERE"
|
||||
|
||||
python3 << 'PYEOF'
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
FIREWORKS_API_KEY = os.environ.get("FIREWORKS_API_KEY")
|
||||
API_URL = "https://api.fireworks.ai/inference/v1/chat/completions"
|
||||
SESSION_DIR = os.environ.get("SESSION_DIR")
|
||||
|
||||
# Load Phase 1 results
|
||||
with open(f"{SESSION_DIR}/config.json") as f:
|
||||
config = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase1_responses.json") as f:
|
||||
phase1_results = json.load(f)
|
||||
|
||||
QUERY = config["query"]
|
||||
MODELS = config["models"]
|
||||
|
||||
# Create anonymized mapping
|
||||
labels = ["A", "B", "C", "D", "E", "F", "G"][:len(MODELS)]
|
||||
model_to_label = dict(zip(MODELS, labels))
|
||||
label_to_model = {v: k for k, v in model_to_label.items()}
|
||||
|
||||
# Format anonymized responses
|
||||
anonymized_responses = []
|
||||
for model_id in MODELS:
|
||||
label = model_to_label[model_id]
|
||||
content = phase1_results[model_id]["content"]
|
||||
anonymized_responses.append(f"=== Response {label} ===\n{content}")
|
||||
|
||||
anonymized_text = "\n\n".join(anonymized_responses)
|
||||
|
||||
def get_rankings(model_id, query, anonymized, own_label):
|
||||
"""Get rankings from a single model"""
|
||||
ranking_prompt = f"""You are evaluating responses from multiple AI models to this query:
|
||||
|
||||
QUERY: {query}
|
||||
|
||||
Here are the anonymized responses:
|
||||
|
||||
{anonymized}
|
||||
|
||||
Please rank these responses from BEST to WORST. For each ranking:
|
||||
1. State the response letter (A, B, C, etc.)
|
||||
2. Give a brief reason (1-2 sentences)
|
||||
3. You may skip ranking your own response (labeled {own_label}) or rank it fairly
|
||||
|
||||
Format your response EXACTLY as:
|
||||
RANKINGS:
|
||||
1. [Letter] - [Brief reason]
|
||||
2. [Letter] - [Brief reason]
|
||||
3. [Letter] - [Brief reason]
|
||||
..."""
|
||||
|
||||
try:
|
||||
start = time.time()
|
||||
response = requests.post(
|
||||
API_URL,
|
||||
headers={
|
||||
"Authorization": f"Bearer {FIREWORKS_API_KEY}",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={
|
||||
"model": model_id,
|
||||
"messages": [
|
||||
{"role": "system", "content": f"You are ranking AI responses objectively. Your own response is labeled '{own_label}'."},
|
||||
{"role": "user", "content": ranking_prompt}
|
||||
],
|
||||
"max_tokens": 1000,
|
||||
"temperature": 1
|
||||
},
|
||||
timeout=90
|
||||
)
|
||||
response.raise_for_status()
|
||||
elapsed = time.time() - start
|
||||
return {
|
||||
"success": True,
|
||||
"content": response.json()["choices"][0]["message"]["content"],
|
||||
"model": model_id,
|
||||
"latency_seconds": round(elapsed, 2)
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"content": f"[ERROR: {str(e)}]",
|
||||
"model": model_id,
|
||||
"latency_seconds": 0
|
||||
}
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("PHASE 2: Cross-Model Ranking")
|
||||
print(f"{'='*60}")
|
||||
print(f"Label mapping: {json.dumps({v: k.split('/')[-1] for k, v in model_to_label.items()})}")
|
||||
print()
|
||||
|
||||
# Collect rankings from all models in parallel
|
||||
rankings = {}
|
||||
with ThreadPoolExecutor(max_workers=len(MODELS)) as executor:
|
||||
futures = {
|
||||
executor.submit(get_rankings, mid, QUERY, anonymized_text, model_to_label[mid]): mid
|
||||
for mid in MODELS
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
model = futures[future]
|
||||
result = future.result()
|
||||
rankings[model] = result
|
||||
status = "OK" if result["success"] else "FAILED"
|
||||
latency = f"{result['latency_seconds']}s" if result["success"] else "N/A"
|
||||
print(f" [{status}] {model.split('/')[-1]} ({latency})")
|
||||
|
||||
# Save rankings
|
||||
output = {
|
||||
"label_mapping": label_to_model,
|
||||
"model_to_label": model_to_label,
|
||||
"rankings": rankings
|
||||
}
|
||||
with open(f"{SESSION_DIR}/phase2_rankings.json", "w") as f:
|
||||
json.dump(output, f, indent=2)
|
||||
|
||||
print(f"\nPhase 2 complete. Rankings saved to: {SESSION_DIR}/phase2_rankings.json")
|
||||
PYEOF
|
||||
```
|
||||
|
||||
### Step 4: Run Phase 3 - Chairman Synthesis
|
||||
|
||||
The Chairman model receives all responses and rankings, then produces the final synthesis:
|
||||
|
||||
```bash
|
||||
SESSION_DIR="/tmp/llm-council/TIMESTAMP_HERE"
|
||||
CHAIRMAN_MODEL="accounts/fireworks/models/glm-5"
|
||||
|
||||
python3 << 'PYEOF'
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import time
|
||||
|
||||
FIREWORKS_API_KEY = os.environ.get("FIREWORKS_API_KEY")
|
||||
API_URL = "https://api.fireworks.ai/inference/v1/chat/completions"
|
||||
SESSION_DIR = os.environ.get("SESSION_DIR")
|
||||
CHAIRMAN_MODEL = os.environ.get("CHAIRMAN_MODEL")
|
||||
|
||||
# Load all previous results
|
||||
with open(f"{SESSION_DIR}/config.json") as f:
|
||||
config = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase1_responses.json") as f:
|
||||
phase1 = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase2_rankings.json") as f:
|
||||
phase2 = json.load(f)
|
||||
|
||||
QUERY = config["query"]
|
||||
label_to_model = phase2["label_mapping"]
|
||||
model_to_label = phase2["model_to_label"]
|
||||
|
||||
# Format responses with model names revealed
|
||||
responses_text = []
|
||||
for model_id, result in phase1.items():
|
||||
label = model_to_label.get(model_id, "?")
|
||||
model_name = model_id.split("/")[-1]
|
||||
responses_text.append(f"=== {label}: {model_name} ===\n{result['content']}")
|
||||
|
||||
# Format rankings
|
||||
rankings_text = []
|
||||
for model_id, result in phase2["rankings"].items():
|
||||
model_name = model_id.split("/")[-1]
|
||||
rankings_text.append(f"[{model_name}'s Rankings]\n{result['content']}")
|
||||
|
||||
synthesis_prompt = f"""You are the Chairman of an LLM Council. Your task is to synthesize the best possible answer from multiple AI responses.
|
||||
|
||||
ORIGINAL QUERY:
|
||||
{QUERY}
|
||||
|
||||
INDIVIDUAL RESPONSES:
|
||||
{chr(10).join(responses_text)}
|
||||
|
||||
MODEL RANKINGS:
|
||||
{chr(10).join(rankings_text)}
|
||||
|
||||
As Chairman, produce a FINAL SYNTHESIS that:
|
||||
1. Incorporates the strongest elements from the best-ranked responses
|
||||
2. Resolves any contradictions between responses
|
||||
3. Addresses aspects that multiple models agreed on
|
||||
4. Corrects any errors identified through cross-ranking
|
||||
5. Provides the most complete, accurate, and helpful answer
|
||||
|
||||
Begin your synthesis:"""
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("PHASE 3: Chairman Synthesis")
|
||||
print(f"{'='*60}")
|
||||
print(f"Chairman: {CHAIRMAN_MODEL.split('/')[-1]}")
|
||||
print()
|
||||
|
||||
try:
|
||||
start = time.time()
|
||||
response = requests.post(
|
||||
API_URL,
|
||||
headers={
|
||||
"Authorization": f"Bearer {FIREWORKS_API_KEY}",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={
|
||||
"model": CHAIRMAN_MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are the Chairman of an LLM Council. Synthesize multiple AI perspectives into a definitive, comprehensive response."},
|
||||
{"role": "user", "content": synthesis_prompt}
|
||||
],
|
||||
"max_tokens": 4000,
|
||||
"temperature": 1
|
||||
},
|
||||
timeout=180
|
||||
)
|
||||
response.raise_for_status()
|
||||
elapsed = time.time() - start
|
||||
synthesis = response.json()["choices"][0]["message"]["content"]
|
||||
|
||||
with open(f"{SESSION_DIR}/phase3_synthesis.txt", "w") as f:
|
||||
f.write(synthesis)
|
||||
|
||||
print(f"Phase 3 complete ({elapsed:.2f}s). Synthesis saved to: {SESSION_DIR}/phase3_synthesis.txt")
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: {e}")
|
||||
synthesis = f"[ERROR: {str(e)}]"
|
||||
with open(f"{SESSION_DIR}/phase3_synthesis.txt", "w") as f:
|
||||
f.write(synthesis)
|
||||
|
||||
# Update config with chairman
|
||||
config["chairman"] = CHAIRMAN_MODEL
|
||||
with open(f"{SESSION_DIR}/config.json", "w") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
PYEOF
|
||||
```
|
||||
|
||||
### Step 5: Display Full Results
|
||||
|
||||
Read all saved files and display the complete council deliberation:
|
||||
|
||||
```bash
|
||||
SESSION_DIR="/tmp/llm-council/TIMESTAMP_HERE"
|
||||
|
||||
python3 << 'PYEOF'
|
||||
import os
|
||||
import json
|
||||
|
||||
SESSION_DIR = os.environ.get("SESSION_DIR")
|
||||
|
||||
# Load all data
|
||||
with open(f"{SESSION_DIR}/config.json") as f:
|
||||
config = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase1_responses.json") as f:
|
||||
phase1 = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase2_rankings.json") as f:
|
||||
phase2 = json.load(f)
|
||||
with open(f"{SESSION_DIR}/phase3_synthesis.txt") as f:
|
||||
synthesis = f.read()
|
||||
|
||||
model_to_label = phase2["model_to_label"]
|
||||
label_to_model = phase2["label_mapping"]
|
||||
|
||||
# Build formatted output
|
||||
output = []
|
||||
output.append("=" * 70)
|
||||
output.append(" LLM COUNCIL DELIBERATION")
|
||||
output.append(" Powered by Fireworks AI")
|
||||
output.append("=" * 70)
|
||||
output.append("")
|
||||
output.append(f"QUERY: {config['query']}")
|
||||
output.append(f"COUNCIL: {', '.join([m.split('/')[-1] for m in config['models']])}")
|
||||
output.append(f"CHAIRMAN: {config.get('chairman', 'N/A').split('/')[-1]}")
|
||||
output.append("")
|
||||
|
||||
# Phase 1: Individual Responses
|
||||
output.append("-" * 70)
|
||||
output.append(" PHASE 1: INDIVIDUAL RESPONSES")
|
||||
output.append("-" * 70)
|
||||
output.append("")
|
||||
|
||||
for model_id, result in phase1.items():
|
||||
model_name = model_id.split("/")[-1]
|
||||
label = model_to_label.get(model_id, "?")
|
||||
latency = result.get("latency_seconds", "N/A")
|
||||
tokens = result.get("tokens", {})
|
||||
output.append(f"[{label}] {model_name} (latency: {latency}s, tokens: {tokens.get('total', 'N/A')})")
|
||||
output.append("-" * 40)
|
||||
output.append(result["content"])
|
||||
output.append("")
|
||||
|
||||
# Phase 2: Cross-Model Rankings
|
||||
output.append("-" * 70)
|
||||
output.append(" PHASE 2: CROSS-MODEL RANKINGS")
|
||||
output.append("-" * 70)
|
||||
output.append("")
|
||||
output.append(f"Label mapping: {json.dumps({v: k.split('/')[-1] for k, v in model_to_label.items()}, indent=2)}")
|
||||
output.append("")
|
||||
|
||||
for model_id, result in phase2["rankings"].items():
|
||||
model_name = model_id.split("/")[-1]
|
||||
output.append(f"[{model_name}'s Rankings]")
|
||||
output.append(result["content"])
|
||||
output.append("")
|
||||
|
||||
# Phase 3: Chairman Synthesis
|
||||
output.append("-" * 70)
|
||||
output.append(" PHASE 3: CHAIRMAN'S SYNTHESIS")
|
||||
output.append("-" * 70)
|
||||
output.append("")
|
||||
chairman_name = config.get("chairman", "Chairman").split("/")[-1]
|
||||
output.append(f"[{chairman_name} - Chairman]")
|
||||
output.append("")
|
||||
output.append(synthesis)
|
||||
output.append("")
|
||||
output.append("=" * 70)
|
||||
output.append(f"Session files: {SESSION_DIR}/")
|
||||
|
||||
# Save formatted output
|
||||
final_output = "\n".join(output)
|
||||
with open(f"{SESSION_DIR}/final_output.md", "w") as f:
|
||||
f.write(final_output)
|
||||
|
||||
print(final_output)
|
||||
print(f"\nFull output saved to: {SESSION_DIR}/final_output.md")
|
||||
PYEOF
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Session Directory**: Each run creates a unique session in `/tmp/llm-council/{timestamp}/`
|
||||
2. **Raw Data Preserved**: All API responses are saved as-is to JSON files for full transparency
|
||||
3. **Cost**: Fireworks pricing is per-token. More models and longer queries cost more. Check current pricing at https://fireworks.ai/pricing
|
||||
4. **Latency Tracking**: Each API call tracks latency so you can see Fireworks' speed in action
|
||||
5. **Token Usage**: Phase 1 responses include token counts for cost awareness
|
||||
6. **Rate Limits**: If you hit rate limits, wait briefly and retry
|
||||
7. **Model Availability**: Check https://app.fireworks.ai/ for current model status
|
||||
|
||||
## Setup
|
||||
|
||||
1. Create a Fireworks AI account at https://fireworks.ai/ and grab your API key from the dashboard
|
||||
2. Export it in your shell profile:
|
||||
```bash
|
||||
export FIREWORKS_API_KEY="your_api_key_here"
|
||||
```
|
||||
3. Restart your terminal or run `source ~/.zshrc`
|
||||
4. Invoke this skill when you want multiple open-weight AI perspectives on a question
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
- Requires the upstream tool, account, API key, or local setup when the workflow names one.
|
||||
- Does not authorize destructive, production, paid, or external-message actions without explicit user approval.
|
||||
- Validate generated artifacts or recommendations against the user's real sources before treating them as final.
|
||||
Reference in New Issue
Block a user