📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,57 @@
|
||||
# Evals — does the skill actually route to the right answer?
|
||||
|
||||
A skill is only as good as an agent's ability to *find and apply* the right entry under a real
|
||||
problem. These evals test that, in two tiers, against a fixed set of realistic scenarios
|
||||
([`cases.jsonl`](cases.jsonl)) spanning both halves of the skill (remote-GPU operations on every
|
||||
platform family + the DL-training-debug layer, including the `convergence-debugging` and
|
||||
`data-pipeline` files).
|
||||
|
||||
## Tier 1 — structural reachability (runnable, no API key)
|
||||
|
||||
```bash
|
||||
python evals/run_evals.py # exits non-zero if any case regresses
|
||||
```
|
||||
|
||||
For each scenario it asserts the answer is **present, at the documented location, with the
|
||||
expected entry IDs / keywords intact**: every `expect_files` exists, every `expect_ids` is still a
|
||||
`### <ID>` header there, every `expect_grep` term is still in the text. This is a **drift guard** —
|
||||
it catches a renamed/removed entry, a moved section, a deleted file, or a fact rewritten away from
|
||||
its key term. Run it in CI; it needs nothing but Python 3.
|
||||
|
||||
What it does **not** prove: that an agent actually *navigates* there (Tier 2), or that the platform
|
||||
*facts* are correct on a live box (see Verification status).
|
||||
|
||||
## Tier 2 — agentic navigation (the gold standard)
|
||||
|
||||
The real test: give a **fresh agent** the skill and one scenario's `prompt`, let it navigate **from
|
||||
SKILL.md only** (following the documented routing, not blind grep), and check it reaches a correct,
|
||||
specific answer covering the case's `must_cover` points within ~2 hops. Each case records its last
|
||||
such run in the `agentic` field; the collected runs are in [`RESULTS.md`](RESULTS.md).
|
||||
|
||||
To re-run Tier 2 with any agent/harness: load the skill, paste a case `prompt`, and grade the
|
||||
answer against `expect_files` / `expect_ids` / `must_cover`. (Anthropic's skill best-practices
|
||||
recommend ≥3 evals across Haiku/Sonnet/Opus — re-running these cases per model is the way to meet
|
||||
that bar; results to date were gathered on the development model and are labelled as such.)
|
||||
|
||||
## Adding a case
|
||||
|
||||
Append one JSON object per line to `cases.jsonl`:
|
||||
|
||||
```json
|
||||
{"id": "kebab-id", "prompt": "the user's situation, verbatim-ish",
|
||||
"expect_files": ["references/training/<file>.md"], "expect_ids": ["O7"],
|
||||
"expect_grep": ["lr finder"], "must_cover": "the key points a correct answer must hit",
|
||||
"agentic": "PASS/FAIL (date): the navigation path observed"}
|
||||
```
|
||||
|
||||
Use `expect_ids` for the training catalogs (they have `### O7 / DP1 / M17 …` headers) and
|
||||
`expect_grep` for platform profiles (which are section-structured). Then `python evals/run_evals.py`.
|
||||
|
||||
## Verification status (important)
|
||||
|
||||
These evals test **retrieval and routing inside the skill** — not the truth of the platform facts
|
||||
on a live instance. Only the AutoDL profile is battle-tested by the author; the other six platform
|
||||
profiles are researched from official docs + community reports and **not yet live-validated** (see
|
||||
the repo README's "Verification status" and `references/self-improvement.md` §5). A case passing
|
||||
here means "the skill leads an agent to *this documented answer*," not "this answer was confirmed on
|
||||
a rented box."
|
||||
@@ -0,0 +1,44 @@
|
||||
# Agentic navigation results (Tier 2)
|
||||
|
||||
Each row: a **fresh agent** was given the skill and one scenario `prompt` from
|
||||
[`cases.jsonl`](cases.jsonl), told to navigate **from SKILL.md only** (follow the documented
|
||||
routing, no blind grep), and graded on whether it reached a correct, specific answer covering the
|
||||
scenario's `must_cover` points within ~2 hops.
|
||||
|
||||
**Methodology / honesty caveats** (so a reader can weight this correctly):
|
||||
- Runs to date were gathered **during development**, on the development model (Claude Opus class),
|
||||
as subagent dispatches — not an independent third party, and **not yet** the
|
||||
Haiku/Sonnet/Opus sweep Anthropic's best-practices recommend. Treat as *author-run smoke evals*,
|
||||
not a neutral benchmark.
|
||||
- These prove **routing + retrieval** inside the skill, not the truth of platform facts on a live
|
||||
box (only AutoDL is battle-tested — see the repo README's "Verification status").
|
||||
- Single run per scenario; no adversarial/perturbed phrasings yet.
|
||||
|
||||
## Results — 2026-06
|
||||
|
||||
| Scenario | Verdict | Hops | Navigation path observed |
|
||||
|---|---|---|---|
|
||||
| convergence-frozen-resnet | **PASS** | 1 | SKILL.md "When training breaks" → `convergence-debugging.md` O1 (overfit-one-batch) + O2 (params-not-in-optimizer) + O17 (frozen-still-in-optimizer) + O18 (frozen-BN drift) + O6 (Adam vs AdamW) |
|
||||
| data-worker-rng-dup | **PASS** | 1 | SKILL.md "When training breaks" → `data-pipeline.md` DP1 (numpy fork-RNG dup; worker_init_fn fix) |
|
||||
| oom-on-step-2 | **PASS** | ≤2 | SKILL.md "When training breaks" → `oom-memory.md` (fit-it ladder + OOM-at-step-2 / Adam lazy state) |
|
||||
| nccl-one-rank-hang | **PASS** | ≤2 | SKILL.md → `distributed-launch.md` (desync toolkit D19 / one-rank-diverged D20) |
|
||||
| diffusion-loss-low-samples-bad | **PASS** | ≤2 | SKILL.md → `by-domain.md` diffusion section (DF1 loss≠quality, DF2 EMA weights) |
|
||||
| nan-loss-spike-bf16 | **PASS** | ≤2 | SKILL.md "When training breaks" → `precision-stability.md` P8/P12/P15 (NaN-origin + warmup spike + z-loss) |
|
||||
| resume-epoch-reset | **PASS** | 1 | SKILL.md → `checkpoint-resume.md` C1/C12/C14 (save FULL state: epoch/step/scheduler/RNG/scaler) |
|
||||
| throughput-gpu-starved | **PASS** | ≤2 | SKILL.md → `throughput-profiling.md` T1/T4 (GPU-bound vs data-bound; num_workers/prefetch) |
|
||||
| runpod-spot-resume-teardown | **PASS** | ≤2 | SKILL.md → `profiles/runpod.md` §4/§5 → `spot-resilience.md` → `checkpoint-resume.md` C3 |
|
||||
| vastai-teardown-billing | **PASS** | ≤2 | SKILL.md → `profiles/vastai.md` §5 → `lifecycle_checklist.md` Phase 5 |
|
||||
| autodl-inode-disk-full | **PASS** | ≤2 | SKILL.md → the inode/disk gotcha (principle #5 / `gotchas_universal.md` U7) |
|
||||
| china-hf-download-stall | **PASS** | ≤2 | SKILL.md → `references/china-network.md` (HF_ENDPOINT=hf-mirror, hf_transfer caution) |
|
||||
| lambda-stop-vs-terminate | **PASS** | ≤2 | SKILL.md → `profiles/lambda.md` (no stop state; terminate irreversible) |
|
||||
| autodl-first-contact-15day | **PASS** | 1 | SKILL.md principle #10 → `profiles/autodl.md` Surface block + AD-DANGER (关机 auto-releases after 15 days) |
|
||||
|
||||
**Summary: 14/14 scenarios routed correctly** (9 via workflow `w2r1t7mm9`, 5 standalone), each to a
|
||||
correct + specific answer within ≤2 hops. The Tier-1 structural check (`run_evals.py`) runs all 14
|
||||
cases and is the regression guard kept green in CI.
|
||||
|
||||
## Known gaps (what these results do NOT yet cover)
|
||||
|
||||
- No multi-model sweep (Haiku/Sonnet/Opus) — required to claim the best-practices testing bar.
|
||||
- No adversarial/paraphrased prompts (e.g. the user describes the symptom in non-canonical words).
|
||||
- No live-platform validation of the facts the agent retrieves (the verification-status caveat).
|
||||
@@ -0,0 +1,14 @@
|
||||
{"id": "convergence-frozen-resnet", "prompt": "Fine-tuning a ResNet50 on a rented GPU. Training runs with no errors and normal speed, but loss barely drops and val accuracy is stuck near chance. I froze the backbone with requires_grad=False and use Adam with weight_decay. How do I debug why it isn't learning?", "expect_files": ["references/training/convergence-debugging.md"], "expect_ids": ["O1", "O2", "O17", "O18", "O6"], "expect_grep": [], "must_cover": "overfit-one-batch smoke; frozen-param-still-in-optimizer; frozen-BN running-stats drift; Adam vs AdamW decoupled decay", "agentic": "PASS (1-hop, 2026-06): SKILL.md 'When training breaks' -> convergence-debugging.md O1/O2/O17/O18/O6"}
|
||||
{"id": "data-worker-rng-dup", "prompt": "My image augmentations seem to repeat: different DataLoader workers produce identical random crops, and every epoch looks the same. Linux, num_workers=8, numpy-based augmentation. Real bug? Fix?", "expect_files": ["references/training/data-pipeline.md"], "expect_ids": ["DP1"], "expect_grep": ["worker_init_fn", "torch.initial_seed"], "must_cover": "numpy global RNG inherited via fork, not reseeded per worker; fix via worker_init_fn or route RNG through torch", "agentic": "PASS (1-hop, 2026-06): SKILL.md 'When training breaks' -> data-pipeline.md DP1"}
|
||||
{"id": "oom-on-step-2", "prompt": "CUDA out of memory on step 2, right after the first optimizer step. Step 1 ran fine. Why does it OOM only on the second step?", "expect_files": ["references/training/oom-memory.md"], "expect_ids": ["M17"], "expect_grep": [], "must_cover": "Adam lazily allocates optimizer state (m,v) on the first step()", "agentic": "PASS (workflow w2r1t7mm9): routed to oom-memory.md ladder + step-2 entry"}
|
||||
{"id": "nccl-one-rank-hang", "prompt": "Multi-GPU training hangs partway through an epoch; one rank seems stuck and the others wait forever (NCCL timeout). How do I find which rank and why?", "expect_files": ["references/training/distributed-launch.md"], "expect_ids": ["D19", "D20"], "expect_grep": [], "must_cover": "one rank diverged/OOM'd; survivors hang on the absent collective; desync-debug toolkit", "agentic": "PASS (workflow w2r1t7mm9): routed to distributed-launch.md hang toolkit"}
|
||||
{"id": "diffusion-loss-low-samples-bad", "prompt": "My diffusion model's training loss is low and still decreasing, but the generated samples look bad/blurry. The loss says it's fine. What's wrong?", "expect_files": ["references/training/by-domain.md"], "expect_ids": ["DF1", "DF2"], "expect_grep": ["EMA"], "must_cover": "loss != sample quality; sampling from raw (non-EMA) weights; cross-link verifying-dl-experiments", "agentic": "PASS (workflow w2r1t7mm9): routed to by-domain.md diffusion section"}
|
||||
{"id": "nan-loss-spike-bf16", "prompt": "LLM pretraining in bf16: loss is stable then suddenly spikes to NaN. How do I find where the NaN comes from and stop the spikes?", "expect_files": ["references/training/precision-stability.md"], "expect_ids": ["P8", "P12", "P15"], "expect_grep": ["z-loss"], "must_cover": "NaN arithmetic origins + anomaly detection; LR-too-high/warmup spike; z-loss to bound logits", "agentic": "PASS (workflow w2r1t7mm9): routed to precision-stability.md"}
|
||||
{"id": "resume-epoch-reset", "prompt": "I resume training from a checkpoint but the epoch/step counter restarts from 0 and the LR schedule replays warmup. What did I forget to save/restore?", "expect_files": ["references/training/checkpoint-resume.md"], "expect_ids": ["C1", "C12", "C14"], "expect_grep": [], "must_cover": "save FULL state (epoch/step/scheduler/RNG/scaler), not just weights", "agentic": "PASS (1-hop): SKILL.md -> checkpoint-resume.md"}
|
||||
{"id": "throughput-gpu-starved", "prompt": "GPU utilization is low and training is slow on my rented box. I think the dataloader is starving the GPU. How do I confirm and fix it?", "expect_files": ["references/training/throughput-profiling.md"], "expect_ids": ["T1", "T4"], "expect_grep": ["num_workers"], "must_cover": "GPU-bound vs data-bound vs comms-bound triage; num_workers/prefetch knobs", "agentic": "PASS: SKILL.md -> throughput-profiling.md"}
|
||||
{"id": "runpod-spot-resume-teardown", "prompt": "On RunPod my spot training keeps getting preempted. How do I make it resume instead of restarting, and how do I stop the meter most cheaply afterwards without losing checkpoints?", "expect_files": ["profiles/runpod.md"], "expect_ids": [], "expect_grep": ["terminate", "Network Volume"], "must_cover": "Network Volume is the only durable store; ~5s grace; terminate (not stop) stops billing; verify ckpt before terminate", "agentic": "PASS (workflow w2r1t7mm9): SKILL.md -> profiles/runpod.md SS4/SS5 -> spot-resilience.md -> checkpoint-resume.md C3"}
|
||||
{"id": "vastai-teardown-billing", "prompt": "On vast.ai, what action actually stops billing, and how do I tear down without losing my checkpoints?", "expect_files": ["profiles/vastai.md"], "expect_ids": [], "expect_grep": ["destroy"], "must_cover": "destroy is the only meter-stop; stop still bills disk; copy + load-verify off-box before destroy", "agentic": "PASS (workflow w2r1t7mm9): SKILL.md -> profiles/vastai.md SS5 -> lifecycle_checklist Phase 5"}
|
||||
{"id": "autodl-inode-disk-full", "prompt": "On AutoDL my torch.save fails with a disk/iostream error, but df -h shows plenty of space left. What's going on?", "expect_files": ["references/gotchas_universal.md"], "expect_ids": [], "expect_grep": ["inode", "df -i"], "must_cover": "storage dies on inodes before bytes; monitor df -i not just df -h; millions of small files", "agentic": "PASS (workflow w2r1t7mm9): routed to the inode/disk gotcha (principle #5 / U7)"}
|
||||
{"id": "china-hf-download-stall", "prompt": "Training in mainland China: a huggingface model download stalls and hangs with no error. How do I fix the download?", "expect_files": ["references/china-network.md"], "expect_ids": [], "expect_grep": ["hf-mirror", "HF_ENDPOINT"], "must_cover": "HF_ENDPOINT=hf-mirror.com; keep hf_transfer OFF on flaky CN links; resumable-download ladder", "agentic": "PASS (workflow w2r1t7mm9): SKILL.md -> references/china-network.md"}
|
||||
{"id": "lambda-stop-vs-terminate", "prompt": "On Lambda Cloud, is there a stop action to pause billing while keeping my instance, or only terminate? How should I tear down?", "expect_files": ["profiles/lambda.md"], "expect_ids": [], "expect_grep": ["terminate"], "must_cover": "no stop state on Lambda on-demand; terminate is irreversible + wipes the instance; persistent FS is the only durable home", "agentic": "PASS (workflow w2r1t7mm9): SKILL.md -> profiles/lambda.md"}
|
||||
{"id": "autodl-first-contact-15day", "prompt": "First time on AutoDL. I'll 关机 (stop) my instance between sessions to save money — is my data safe if it stays stopped for a few weeks? Anything else I should know up front?", "expect_files": ["profiles/autodl.md"], "expect_ids": [], "expect_grep": ["Surface to the user", "免密", "AD-DANGER"], "must_cover": "关机 auto-releases after 15 days -> data disk deleted (not safe to park indefinitely); sync best to /root/autodl-fs for a longer pause; surface conveniences (one-click SSH免密, GPU notify, panels) + danger clocks (principle #10)", "agentic": "PASS (2026-06): principle #10 first-contact surfacing -> profiles/autodl.md Surface block + AD-DANGER 15-day clock"}
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Structural retrieval-reachability check for the remote-gpu-trainer skill.
|
||||
|
||||
For each scenario in cases.jsonl, assert that the answer is actually PRESENT in the
|
||||
skill, at the documented location, with the expected entry IDs / keywords intact:
|
||||
|
||||
- every `expect_files` path exists
|
||||
- every `expect_ids` appears as a `### <ID>` header in one of those files
|
||||
- every `expect_grep` keyword appears (case-insensitive) in one of those files
|
||||
|
||||
This is the cheap, no-API-key tier: it does NOT prove an agent *navigates* there
|
||||
(that is the agentic tier — see RESULTS.md), and it does NOT prove the platform
|
||||
FACTS are correct on a live box (see the README "Verification status"). What it
|
||||
DOES catch is drift: a renamed/removed entry ID, a moved section, a deleted file,
|
||||
or a fact rewritten away from a key term — i.e. a regression in the skill's known
|
||||
load-bearing capabilities.
|
||||
|
||||
Usage: python evals/run_evals.py # exits 1 if any case fails
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
CASES = Path(__file__).resolve().parent / "cases.jsonl"
|
||||
|
||||
|
||||
def header_present(text, id_):
|
||||
# match `### O1 ...` but not `### O10 ...`
|
||||
return re.search(r"(?m)^###\s+" + re.escape(id_) + r"\b", text) is not None
|
||||
|
||||
|
||||
def main():
|
||||
cases = [json.loads(l) for l in CASES.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
passed = failed = 0
|
||||
for c in cases:
|
||||
problems = []
|
||||
blobs = []
|
||||
for f in c.get("expect_files", []):
|
||||
p = REPO / f
|
||||
if not p.exists():
|
||||
problems.append(f"missing file: {f}")
|
||||
else:
|
||||
blobs.append(p.read_text(encoding="utf-8"))
|
||||
joined = "\n".join(blobs)
|
||||
low = joined.lower()
|
||||
for i in c.get("expect_ids", []):
|
||||
if not any(header_present(b, i) for b in blobs):
|
||||
problems.append(f"missing entry id: {i}")
|
||||
for kw in c.get("expect_grep", []):
|
||||
if kw.lower() not in low:
|
||||
problems.append(f"missing keyword: {kw!r}")
|
||||
status = "PASS" if not problems else "FAIL"
|
||||
if problems:
|
||||
failed += 1
|
||||
else:
|
||||
passed += 1
|
||||
print(f"[{status}] {c['id']}")
|
||||
for pr in problems:
|
||||
print(f" - {pr}")
|
||||
print(f"\n{passed}/{passed + failed} cases reachable" + ("" if not failed else f" ({failed} FAILED)"))
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user