📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
+45
-1
@@ -179,6 +179,26 @@ Tested across 3 real-world scenarios (PR review, architecture audit, tech debt a
|
||||
|
||||
The gap isn't what Claude *can* find — it's what it *consistently* finds, with traceable evidence and actionable remedies every time.
|
||||
|
||||
### Reproducible benchmarks
|
||||
|
||||
The table above is illustrative. These numbers are **deterministic and you can reproduce them locally**:
|
||||
|
||||
**Parser fidelity** — SARIF export and the CI gates depend on parsing the model's Markdown report correctly. Against a **frozen corpus of 30 real, model-generated reports** spanning all six modes (`evals/benchmark-corpus.json`), each paired with an **independently graded** finding inventory (a separate model pass, spot-checked by hand), the shipped parser scores — run `npm run benchmark`:
|
||||
|
||||
| Metric (n = 30, frozen corpus) | Result |
|
||||
|---|:---:|
|
||||
| Exact severity-count match (parser vs. graded truth) | 30 / 30 |
|
||||
| Risk-code precision / recall | 100% / 100% (56 finding-level codes, 0 FP / 0 FN) |
|
||||
| Valid SARIF 2.1.0 emitted | 30 / 30 |
|
||||
|
||||
Because the parser is deterministic and the corpus is frozen, `npm run benchmark` gives everyone the same result, and `npm test` guards it as a regression. The corpus deliberately includes 9 false-positive / tradeoff reports (e.g. a ports-and-adapters design that *looks* like a dependency cycle) that must stay clean.
|
||||
|
||||
**Scoring determinism** — for a fixed finding set (2 Critical / 3 Warning / 1 Suggestion), the strictness presets produce exactly the scores their `common.md` table predicts: strict **34**, balanced **54**, legacy-friendly **74** — and only `legacy-friendly` leads with the top-three fixes.
|
||||
|
||||
**Model quality** — whether the model finds the *right* risks on real code is measured by the **57-scenario eval suite** (`evals/evals.json`): `npm run evals` (structural) and `npm run evals:live` (live, needs `ANTHROPIC_API_KEY`).
|
||||
|
||||
> Scope & honesty: the parser numbers are deterministic and exactly reproducible. The strictness and eval-suite figures are single-run live measurements against the model and vary slightly run to run. The parser benchmark measures report-parsing fidelity (does the tooling read every finding the report states?), not whether a given finding is "correct." The severity-count match is the fully independent signal; risk-code agreement also reflects the shared canonical name→code legend.
|
||||
|
||||
## How It Compares
|
||||
|
||||
| | brooks-lint | ESLint / Pylint | GitHub Copilot Review | Plain Claude |
|
||||
@@ -436,6 +456,8 @@ Place a `.brooks-lint.yaml` in your project root to customize review behavior:
|
||||
```yaml
|
||||
version: 1
|
||||
|
||||
strictness: balanced # strict | balanced (default) | legacy-friendly — softer scoring for legacy code
|
||||
|
||||
disable:
|
||||
- T5 # skip coverage metrics check — we don't enforce coverage
|
||||
|
||||
@@ -445,6 +467,9 @@ severity:
|
||||
ignore:
|
||||
- "**/*.generated.*"
|
||||
- "**/vendor/**"
|
||||
|
||||
# custom_risks: # define project-specific Cx codes — see skills/_shared/custom-risks-guide.md
|
||||
# suppress: # downgrade specific findings by risk + path (e.g. accepted legacy debt)
|
||||
```
|
||||
|
||||
Copy [`.brooks-lint.example.yaml`](.brooks-lint.example.yaml) as a starting point.
|
||||
@@ -452,10 +477,13 @@ All settings are optional — omit the file entirely for default behavior.
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| `strictness` | Scoring preset: `strict`, `balanced` (default), or `legacy-friendly` (lighter deductions, leads with top fixes) |
|
||||
| `disable` | Risk codes to skip (`R1`–`R6`, `T1`–`T6`) |
|
||||
| `severity` | Override severity tier (`critical` / `warning` / `suggestion`) |
|
||||
| `ignore` | Glob patterns for files to exclude |
|
||||
| `focus` | Evaluate only these risk codes (cannot combine with `disable`) |
|
||||
| `custom_risks` | Define project-specific risk codes (`C1`, `C2`, …) — see [`custom-risks-guide.md`](skills/_shared/custom-risks-guide.md) |
|
||||
| `suppress` | Downgrade specific findings by risk + path (optional `expires:` date) |
|
||||
|
||||
---
|
||||
|
||||
@@ -544,11 +572,24 @@ See [`docs/github-action-example.yml`](docs/github-action-example.yml) for the f
|
||||
|
||||
The action posts the review as a PR comment and optionally fails the check if the Health Score drops below a threshold. If `.brooks-lint-history.json` is committed to your repo, the comment also includes a trend delta (e.g., "85 → 82 (−3) over last 3 runs").
|
||||
|
||||
**Quality gates and Code Scanning.** Beyond `fail-below`, the action exposes:
|
||||
|
||||
```yaml
|
||||
with:
|
||||
mode: review
|
||||
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
fail-on: critical # fail on any Critical finding (none | warning | critical)
|
||||
fail-on-regression: true # fail if the Health Score dropped vs the last run
|
||||
sarif-file: brooks-lint.sarif # also upload findings to GitHub Code Scanning
|
||||
```
|
||||
|
||||
`fail-on-regression` reads `.brooks-lint-history.json`, so commit that file to enforce "no new regressions". Setting `sarif-file` makes findings appear inline on the PR's **Files changed** tab and requires `security-events: write` permission on the job.
|
||||
|
||||
**Cost:** ~$0.05–0.15 per PR run depending on diff size and model. Recommend running on `pull_request` events only.
|
||||
|
||||
## Roadmap
|
||||
|
||||
> **Current state (v1.0):** 12-book foundation, 6 production decay risks (R1–R6) + 6 test decay risks (T1–T6), 5 skills — PR Review, Architecture Audit, Tech Debt, Test Quality, Health Dashboard. Earlier entries below describe historical milestones, not the current feature set.
|
||||
> **Current state (v1.3):** 12-book foundation, 6 production decay risks (R1–R6) + 6 test decay risks (T1–T6), 6 skills — PR Review, Architecture Audit, Tech Debt, Test Quality, Health Dashboard, Full Sweep. Earlier entries below describe historical milestones, not the current feature set.
|
||||
|
||||
- [x] **v0.2**: Plugin infrastructure (`.claude-plugin/`, hooks, slash commands)
|
||||
- [x] **v0.3**: Eight Brooks dimensions, documentation completeness scoring
|
||||
@@ -559,6 +600,9 @@ The action posts the review as a PR comment and optionally fails the check if th
|
||||
- [x] **v0.8**: Independent skill architecture with namespaced commands
|
||||
- [x] **v0.9**: Step validation, auto-diff scope, `/brooks-health` dashboard, trend tracking, triage mode, `--fix` remedies, onboarding report, GitHub Action
|
||||
- [x] **v1.0**: Eval automation (`run-evals-live.mjs`), custom risk extension (`Cx` codes)
|
||||
- [x] **v1.1**: Full Sweep skill (`brooks-sweep`) — unified multi-dimension auto-fix
|
||||
- [x] **v1.2**: Autonomous sweep pipeline, `npm run bump` version propagation
|
||||
- [x] **v1.3**: Codex marketplace metadata, one-command installer for multiple agent platforms, bilingual README + landing site
|
||||
|
||||
Want to help? The best contributions right now are new eval test cases and improved decay risk symptom patterns. See [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user