diff --git a/skills/thirdparty/codebase-recon/SKILL.md b/skills/thirdparty/codebase-recon/SKILL.md --- a/skills/thirdparty/codebase-recon/SKILL.md +++ b/skills/thirdparty/codebase-recon/SKILL.md @@ -252 +252 @@ Core methodology: -- [confidence.md](../pathfinding/references/confidence.md) — confidence calibration (shared with pathfinding) +- [confidence-calibration.md](references/confidence-calibration.md) — evidence-based confidence calibration @@ -263 +263,0 @@ Related skills: -- `outfitter:pathfinding` — clarifying requirements before analysis diff --git a/skills/thirdparty/codebase-recon/references/confidence-calibration.md b/skills/thirdparty/codebase-recon/references/confidence-calibration.md new file mode 100644 --- /dev/null +++ b/skills/thirdparty/codebase-recon/references/confidence-calibration.md @@ -0,0 +1,66 @@ +# Confidence Calibration + +Confidence measures how strongly the available evidence supports a codebase +finding. It does not measure requirement clarity or implementation readiness. + +## Calibration Dimensions + +Consider these dimensions together; do not average them mechanically: + +- **Directness**: runtime observation and source code outweigh inference. +- **Coverage**: evidence spans the relevant execution paths and modules. +- **Corroboration**: code, tests, documentation, and history agree. +- **Counter-evidence**: plausible alternatives and contradictory cases were checked. +- **Remaining gaps**: unknown behavior is bounded and stated explicitly. + +## Typical Levels + +| Level | Evidence state | +| --- | --- | +| 0 | No repository evidence collected yet. | +| 1 | Broad structure or search results only. | +| 2 | Relevant code located, but behavior or scope is not verified. | +| 3 | Main path traced with some corroboration; meaningful gaps remain. | +| 4 | Multiple sources support the finding and counter-examples were checked. | +| 5 | Relevant scope is covered, alternatives are ruled out, and conclusions are directly supported. | + +A clear user request can narrow the investigation but cannot raise confidence in +a finding by itself. Documentation alone normally remains at level 1-2. A +single code path without tests, history, or runtime confirmation normally +remains at level 2-3. + +## Examples + +### Focused Question, Strong Evidence + +Question: "Does deleting this adapter break the HTTP client?" + +Evidence: imports show no production consumer, tests cover the replacement +adapter, and history confirms the old adapter was retained only for migration. +After checking dynamic loading and configuration references, report at level +4-5. Cite each source and distinguish observed facts from the final inference. + +### Cross-Module Finding, Incomplete Evidence + +Question: "Why do all write operations depend on the legacy package?" + +Evidence: searches show imports in several modules, but runtime registration +and generated code have not been inspected. Stay at level 2-3. Continue tracing +composition roots and tests before claiming an architectural dependency. + +### Early Delivery + +The user asks for findings before runtime behavior or history can be checked. +Deliver the verified portion at its current level and include `△ Caveats` with: + +- the evidence already inspected; +- assumptions required by the conclusion; +- unverified modules or execution paths; +- the next checks that would materially change confidence. + +Do not present the early report as level 5 even when the likely explanation is +simple. + +## Delivery Check + +Before assigning level 4 or 5, confirm that: + +- more than one relevant evidence source supports the conclusion; +- direct observations and inferences are labeled separately; +- at least one plausible counter-example was considered; +- the investigated scope matches the wording of the conclusion; +- remaining uncertainty is absent or explicitly bounded.