43 lines
3.2 KiB
Markdown
43 lines
3.2 KiB
Markdown
# Planning Stopped Us from Making Search Worse
|
||
|
||
**Date**: 2026-08-12 18:44
|
||
**Severity**: High
|
||
**Component**: Search, datasets, agent guidance, UI taxonomy
|
||
**Status**: Resolved
|
||
|
||
## What Happened
|
||
|
||
We completed a nine-phase overhaul plan, not an implementation. Data expansion was gated behind a human-judged relevance corpus because the existing 36 unit tests, 12 domain smokes, and 22 stack smokes mostly prove that output is non-empty—not relevant. Adding rows before measuring P@1, MRR@3, nDCG@3, abstention, and typo recovery would amplify noise and erase our baseline.
|
||
|
||
## The Brutal Truth
|
||
|
||
The repository looked healthier than it was. Passing tests concealed a router where 37 of 122 non-product keywords return zero hits in their own domain, while normalization can turn `navigation` into `navigationigation`. Shipping more data into that system would have been busywork dressed up as progress. It is frustrating that basic relevance remained unmeasured while catalog counts made the product look mature.
|
||
|
||
## Technical Details
|
||
|
||
BM25 stays because the offline, stdlib-only CLI and public command contract are deliberate constraints. We rejected embeddings, a vector database, and a graph database without evidence that lexical retrieval is the bottleneck. The planned repair is normalization, field and phrase boosts, and score calibration; hybrid retrieval is reconsidered only if measured paraphrase recall remains the dominant failure.
|
||
|
||
Agent query guidance is first-class because bad instructions corrupt retrieval before ranking starts. “Always use design-system” is wrong for fixes, reviews, and stack questions, and the advertised AI example currently selects a Marketplace/Directory landing pattern. The plan therefore requires intent separation, category/result validation, and low-confidence abstention.
|
||
|
||
UI research also rejected trend dumping. Consolidate Bento, Aurora/gradient, and Swiss families; preserve mobile Neumorphism/Claymorphism variants; add Fluent 2, Shopify Polaris, and Clean Science only with official evidence. Holographic aliases to HUD, High Imagery remains a treatment, and Spectrum 2 plus Canvas/Workspace stay gated.
|
||
|
||
## What We Tried
|
||
|
||
- Compared runtime behavior, 35 CSVs, agent guidance, and official UI-system sources.
|
||
- Red-teamed all nine phases, then propagated three blockers and seven important corrections.
|
||
- Rejected immediate dataset expansion and architecture replacement because neither had relevance evidence.
|
||
|
||
## Root Cause Analysis
|
||
|
||
We optimized structural validity before relevance and let cross-file contracts drift: 192 products exist, but only 161 reasoning rows do, with unsafe substring fallback masking gaps.
|
||
|
||
## Lessons Learned
|
||
|
||
Measure before expanding. Treat prompts as production interfaces. A generated catalog is not correct merely because it parses.
|
||
|
||
## Next Steps
|
||
|
||
Phase 1 owner must obtain human judgments and freeze executable thresholds before ranking changes. Phase 2 owns BM25/router repair; Phases 3–5 own schema, agent contract, and taxonomy. Red-team corrections require exact reasoning for all 192 products, explicit Phase 7/8 runtime ownership, a closed `Decision_Rules` grammar, and one aggregate CI gate.
|
||
|
||
The plan artifacts are local and untracked. No implementation occurred, and no source or CSV file was changed.
|