📦 deps(thirdparty): update snapshots
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Planning Stopped Us from Making Search Worse
|
||||
|
||||
**Date**: 2026-08-12 18:44
|
||||
**Severity**: High
|
||||
**Component**: Search, datasets, agent guidance, UI taxonomy
|
||||
**Status**: Resolved
|
||||
|
||||
## What Happened
|
||||
|
||||
We completed a nine-phase overhaul plan, not an implementation. Data expansion was gated behind a human-judged relevance corpus because the existing 36 unit tests, 12 domain smokes, and 22 stack smokes mostly prove that output is non-empty—not relevant. Adding rows before measuring P@1, MRR@3, nDCG@3, abstention, and typo recovery would amplify noise and erase our baseline.
|
||||
|
||||
## The Brutal Truth
|
||||
|
||||
The repository looked healthier than it was. Passing tests concealed a router where 37 of 122 non-product keywords return zero hits in their own domain, while normalization can turn `navigation` into `navigationigation`. Shipping more data into that system would have been busywork dressed up as progress. It is frustrating that basic relevance remained unmeasured while catalog counts made the product look mature.
|
||||
|
||||
## Technical Details
|
||||
|
||||
BM25 stays because the offline, stdlib-only CLI and public command contract are deliberate constraints. We rejected embeddings, a vector database, and a graph database without evidence that lexical retrieval is the bottleneck. The planned repair is normalization, field and phrase boosts, and score calibration; hybrid retrieval is reconsidered only if measured paraphrase recall remains the dominant failure.
|
||||
|
||||
Agent query guidance is first-class because bad instructions corrupt retrieval before ranking starts. “Always use design-system” is wrong for fixes, reviews, and stack questions, and the advertised AI example currently selects a Marketplace/Directory landing pattern. The plan therefore requires intent separation, category/result validation, and low-confidence abstention.
|
||||
|
||||
UI research also rejected trend dumping. Consolidate Bento, Aurora/gradient, and Swiss families; preserve mobile Neumorphism/Claymorphism variants; add Fluent 2, Shopify Polaris, and Clean Science only with official evidence. Holographic aliases to HUD, High Imagery remains a treatment, and Spectrum 2 plus Canvas/Workspace stay gated.
|
||||
|
||||
## What We Tried
|
||||
|
||||
- Compared runtime behavior, 35 CSVs, agent guidance, and official UI-system sources.
|
||||
- Red-teamed all nine phases, then propagated three blockers and seven important corrections.
|
||||
- Rejected immediate dataset expansion and architecture replacement because neither had relevance evidence.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
We optimized structural validity before relevance and let cross-file contracts drift: 192 products exist, but only 161 reasoning rows do, with unsafe substring fallback masking gaps.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
Measure before expanding. Treat prompts as production interfaces. A generated catalog is not correct merely because it parses.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 1 owner must obtain human judgments and freeze executable thresholds before ranking changes. Phase 2 owns BM25/router repair; Phases 3–5 own schema, agent contract, and taxonomy. Red-team corrections require exact reasoning for all 192 products, explicit Phase 7/8 runtime ownership, a closed `Decision_Rules` grammar, and one aggregate CI gate.
|
||||
|
||||
The plan artifacts are local and untracked. No implementation occurred, and no source or CSV file was changed.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
date: 2026-08-13
|
||||
session: phase-6-search-data-accessibility-refresh
|
||||
severity: high
|
||||
component: search-data-accessibility
|
||||
status: resolved
|
||||
---
|
||||
|
||||
# Phase 6 Passed Only After the Review Found What Gates Missed
|
||||
|
||||
## Context
|
||||
|
||||
Phase 6 refreshed search-facing accessibility contracts across colors, charts, UX guidance, landing patterns, typography, icons, motion, native interfaces, and React guidance. Initial gates passed. Adversarial review still found semantic gaps that those gates did not encode.
|
||||
|
||||
## What Happened
|
||||
|
||||
- Preserved chart backward compatibility: legacy `Accessibility Grade` remains with `deprecated: use Accessibility Risk`; controlled risk is now required.
|
||||
- Added role-aware palette validation and generator token export. Natural-language accessibility queries gained explicit coverage and correct routing.
|
||||
- Repaired decorative, meaningful, and interactive icon semantics; typography family/import/weight agreement; React ref/effect safety; reduced-motion, offscreen, landing-media, and single-pointer alternatives.
|
||||
- Added form error-summary guidance, strict row-bound provenance, error boundaries, and accurate guide/catalog counts.
|
||||
- Final evidence: `npm run verify:data` passed 112/112 tests; relevance stayed P@1 0.7763, MRR@3 0.8465, nDCG@3 0.8447, abstention 0.9804, typo recovery/coherence 1.0; 7/7 Playwright tests and all CLI, gallery, mirror, and diff release gates passed.
|
||||
|
||||
## Reflection
|
||||
|
||||
The brutal truth: we trusted broad green gates that proved files parsed and familiar queries worked, but did not prove the advice was semantically safe. That was our testing failure, not bad luck. It is frustrating to discover after an apparent pass that a chart grade could overclaim conformance, generated colors could lose role meaning, and “accessible” landing guidance could omit actual interaction fallbacks. Relief came only after the adversarial cases became executable contracts.
|
||||
|
||||
## Decisions Made
|
||||
|
||||
| Decision | Rationale | Impact |
|
||||
|---|---|---|
|
||||
| Preserve legacy chart shape; deprecate in place | Deletion would break downstream CSV consumers | Compatibility retained without preserving misleading semantics |
|
||||
| Enforce strict official-source provenance and validation boundaries | Permissive metadata let unsupported claims survive | Volatile claims now fail closed |
|
||||
| Expand natural-query and accessibility semantics instead of only adding rows | More rows would amplify retrieval ambiguity | Coverage improved with unchanged relevance |
|
||||
| Pause after Phase 6 | User explicitly requested a checkpoint | Phases 7–9 remain pending |
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Main agent owns Phases 7–9 when the user resumes; preserve the Phase 6 release gates on every later change.
|
||||
- Keep `plans/` local. Do not change `.gitignore` yet; user deferred that decision until the whole plan completes.
|
||||
- Unresolved questions: none for Phase 6; final relevance threshold promotion remains Phase 9 work.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
date: 2026-08-13
|
||||
session: phase-7-web-stack-freshness
|
||||
severity: high
|
||||
component: web-stack-search-guidance
|
||||
status: resolved
|
||||
---
|
||||
|
||||
# Phase 7 Nearly Shipped Fresh Labels on Stale Framework Advice
|
||||
|
||||
## What Happened
|
||||
|
||||
Phase 7 refreshed 589 rows across React, Next.js, Vue, Svelte, Astro, Angular, Tailwind, shadcn, Nuxt, and Nuxt UI. We now have 573 active rows and 16 explicitly deprecated legacy rows, each with applicability, status, verification date, and official-source coverage for all 179 Critical/High rows. Default search isolates current guidance; explicit supported-old-major queries return labeled legacy rows; unsupported old majors abstain.
|
||||
|
||||
## The Brutal Truth
|
||||
|
||||
We began with an Astro 7.2 assumption that sounded current and was simply false. Official release evidence said 7.1.6. Worse, the first refresh still carried removed contracts: `output: 'hybrid'`, `<ViewTransitions />`, `@astrojs/tailwind`, and the old prefetch integration. Calling that “freshness” would have been embarrassing. The reviewer then found the legacy parser missed ordinary forms such as `svelte@4` and `Next.js (v15)`, letting explicit legacy intent fall into current guidance. Seeing green gates before those defects surfaced was maddening; the relief came only when each failure became a regression test.
|
||||
|
||||
## Technical Details
|
||||
|
||||
The corrected Astro rows use `ClientRouter`, on-demand rendering, Tailwind 4 through Vite, and built-in prefetch. Shared stack selection now recognizes plain majors, `@major`, and `(vMajor)`, preserves current-major migration queries such as `Next.js 16 migration`, and returns zero rows when an older generation has no curated corpus.
|
||||
|
||||
Final evidence: `npm --prefix cli run verify:data` passed 124/124 Python tests; focused web freshness passed 12/12; smoke coverage passed 12/12 domains and 22/22 stacks. Relevance held at P@1 `0.8026315789`, MRR@3 `0.8618421053`, nDCG@3 `0.8561593731`, abstention `0.9803921569`, and coherence `1.0`. Asset sync, CLI typecheck, CLI build, `git diff --check`, and final reviewer all reported PASS.
|
||||
|
||||
## What We Tried
|
||||
|
||||
- Rejected blanket version-string replacement because most rows needed semantic verification, not cosmetic edits.
|
||||
- Retained curated Svelte, Tailwind 3, Next.js 15, and Nuxt 3 guidance as deprecated instead of deleting migration value.
|
||||
- Rejected fabricating old-major coverage; abstention is safer than mixing generations.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
We treated a planned version number and passing broad tests as evidence. They were not. Research had to verify the release, while adversarial review had to exercise syntax variants and removed APIs that the initial tests omitted.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
Version freshness means executable API validity plus retrieval isolation. Every supported query spelling needs a test, and every removed symbol needs a denylist assertion.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Main agent owns Phase 8 native/desktop/3D freshness, then Phase 9 provenance, final judgment review, threshold promotion, and release gates when the user resumes.
|
||||
- Pause now: the user explicitly mandated a Phase 7 checkpoint.
|
||||
- Keep `plans/` local and defer any `.gitignore` decision until all phases finish.
|
||||
|
||||
Unresolved questions: none for Phase 7; final relevance targets remain Phase 9 work.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
date: 2026-08-13
|
||||
session: search-data-overhaul-completion
|
||||
severity: high
|
||||
component: search-relevance-data-catalogs-and-release-gates
|
||||
status: resolved-with-carryover
|
||||
---
|
||||
|
||||
# Search and Data Overhaul Finished, but Green Was Not Enough
|
||||
|
||||
## Context
|
||||
|
||||
The nine-phase overhaul audited BM25 routing, query interpretation, agent guidance, CSV quality, current UI styles, framework/native stack freshness, generated catalogs, provenance, and release gates. The local plan is complete at 9/9 phases and 148/148 items. `plans/` is ignored by Git as intended; none of those local artifacts belongs in a release.
|
||||
|
||||
## What Happened
|
||||
|
||||
The final evidence is concrete: 148 Python tests passed; smoke coverage passed 12/12 domains and 22/22 stacks; Playwright passed 8/8; and the relevance suite passed its approved provisional regression gate. The canonical catalogs now contain 1,934 approved Google Fonts plus eight explicit `needs-review` exclusions, and 1,512 upstream Phosphor icons supporting 105 curated icon rows. Google Fonts provenance is pinned to repository revision `038b637da7b3fd956a4ed93ffc607c3d5e4ce172`.
|
||||
|
||||
The reviewer still found four release-grade defects after broad gates were green. The release workflow ran only a subset of Playwright tests. Font provenance omitted the exact source revision. Refresh and validation disagreed about allowed exclusion evidence URLs. Generated mirrors could remain stale after canonical-source edits. We fixed all four: full Playwright now gates release, the revision is required and validated, both paths share the same narrow source policy, and mirror synchronization/checks are part of the final verification.
|
||||
|
||||
## Reflection
|
||||
|
||||
The brutal truth is that we almost called this complete while the release pipeline could certify less than the test suite, and while a future font refresh could not be reproduced from an exact upstream tree. That is not polish; it is a broken evidence chain. It was exhausting to watch “all green” repeatedly collapse under adversarial review, especially after nine phases of careful work. The relief is earned now because the failures became executable contracts instead of comments developers must remember.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Kept the relevance threshold as a provisional regression gate. We rejected silently promoting it to a final quality target because the held-out judgments still need a second human review.
|
||||
- Chose fail-closed provenance and evidence policies over permissive refreshes. Unknown revisions, unsupported exclusion URLs, interrupted publication, and stale mirrors must stop validation.
|
||||
- Kept 8 unresolved font families visible as `needs-review` exclusions instead of inventing license certainty.
|
||||
- Did not commit or push. This session prepared and verified the work only.
|
||||
|
||||
## Next
|
||||
|
||||
- Project owner: complete a second human relevance judgment review before promoting final thresholds; preserve the current baseline until then.
|
||||
- Maintainers: inspect weekly refresh artifacts before accepting catalog changes; never auto-merge upstream data.
|
||||
- Release owner: retain full Playwright, provenance, mirror, catalog-summary, actionlint, and diff gates on every release.
|
||||
|
||||
Unresolved question: when will the independent relevance judgment review be scheduled?
|
||||
Reference in New Issue
Block a user