csh/playbook
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

📦 deps(thirdparty): update snapshots

Browse Source
This commit is contained in:
ci[bot]
2026-06-18 16:04:36 +00:00
parent 71b421806e
commit dd4c084042
416 changed files with 35467 additions and 3065 deletions
Download Patch File Download Diff File Expand all files Collapse all files
brooks-lint/CLAUDE.md
+6 -1
View File
@@ -60,12 +60,16 @@ Checklist:
## Eval Suite
`evals/evals.json` contains 49 benchmark scenarios covering R1–R6 (code decay) and T1–T6 (test decay), including false-positive / tradeoff cases that must NOT be flagged. Each scenario has `id`, `name`, `prompt`, `expected_output`, `mode`, `files`. Optional flags (mutually exclusive): `no_risk_codes: true` (no risk codes expected in output) or `no_health_score: true` (Health Score suppression test).
`evals/evals.json` contains 57 benchmark scenarios covering R1–R6 (code decay) and T1–T6 (test decay), including false-positive / tradeoff cases that must NOT be flagged. Each scenario has `id`, `name`, `prompt`, `expected_output`, `mode`, `files`. Optional flags (mutually exclusive): `no_risk_codes: true` (no risk codes expected in output) or `no_health_score: true` (Health Score suppression test).
To add a scenario: append to the `evals` array with the next sequential `id` and the relevant risk code. Validate structure with `npm run evals`; live-test with `npm run evals:live` (requires `ANTHROPIC_API_KEY`).
`expected_output` should describe the Iron Law finding (Symptom + risk code) and a Health Score range; it does NOT need to be verbatim — the evaluator matches semantics. For false-positive / tradeoff scenarios, set `no_risk_codes: true` and describe what must NOT appear in output.
## Parser-Fidelity Benchmark
`evals/benchmark-corpus.json` is a FROZEN corpus of 30 real, model-generated reports (one per curated sample, across all six modes) each paired with an independently-graded finding inventory. `scripts/benchmark.mjs` (`npm run benchmark`) runs the shipped parser (`report-parse.mjs` / `sarif.mjs`) against it and reports severity-count fidelity, risk-code precision/recall, and SARIF validity; `npm test` guards the same as a deterministic regression. This is distinct from the eval suite: it benchmarks the **parser/SARIF plumbing**, not model judgment. The corpus is a frozen artifact — regenerate it only by re-running the generation workflow and hand-checking the new ground truth; do NOT hand-edit `report` or `truth` to make a failing parser pass.
## Development Commands
```bash
@@ -74,6 +78,7 @@ npm run validate # Repo consistency: manifests, README badge, changelog
npm test # Unit tests for validate-repo helpers
npm run evals # Eval structural validation (IDs, fields, risk-code refs)
npm run evals:live # Live evals against the AI (requires ANTHROPIC_API_KEY)
npm run benchmark # Parser-fidelity benchmark on the frozen real-report corpus
npm run history # View Health Score trend (.brooks-lint-history.json)
# Test hooks locally