2.6 KiB
Confidence Calibration
Confidence measures how strongly the available evidence supports a codebase finding. It does not measure requirement clarity or implementation readiness.
Calibration Dimensions
Consider these dimensions together; do not average them mechanically:
- Directness: runtime observation and source code outweigh inference.
- Coverage: evidence spans the relevant execution paths and modules.
- Corroboration: code, tests, documentation, and history agree.
- Counter-evidence: plausible alternatives and contradictory cases were checked.
- Remaining gaps: unknown behavior is bounded and stated explicitly.
Typical Levels
| Level | Evidence state |
|---|---|
| 0 | No repository evidence collected yet. |
| 1 | Broad structure or search results only. |
| 2 | Relevant code located, but behavior or scope is not verified. |
| 3 | Main path traced with some corroboration; meaningful gaps remain. |
| 4 | Multiple sources support the finding and counter-examples were checked. |
| 5 | Relevant scope is covered, alternatives are ruled out, and conclusions are directly supported. |
A clear user request can narrow the investigation but cannot raise confidence in a finding by itself. Documentation alone normally remains at level 1-2. A single code path without tests, history, or runtime confirmation normally remains at level 2-3.
Examples
Focused Question, Strong Evidence
Question: "Does deleting this adapter break the HTTP client?"
Evidence: imports show no production consumer, tests cover the replacement adapter, and history confirms the old adapter was retained only for migration. After checking dynamic loading and configuration references, report at level 4-5. Cite each source and distinguish observed facts from the final inference.
Cross-Module Finding, Incomplete Evidence
Question: "Why do all write operations depend on the legacy package?"
Evidence: searches show imports in several modules, but runtime registration and generated code have not been inspected. Stay at level 2-3. Continue tracing composition roots and tests before claiming an architectural dependency.
Early Delivery
The user asks for findings before runtime behavior or history can be checked.
Deliver the verified portion at its current level and include △ Caveats with:
- the evidence already inspected;
- assumptions required by the conclusion;
- unverified modules or execution paths;
- the next checks that would materially change confidence.
Do not present the early report as level 5 even when the likely explanation is simple.
Delivery Check
Before assigning level 4 or 5, confirm that: