Squashed 'docs/standards/playbook/' changes from b529012..a854534
a854534 ✨ feat(plan_progress): auto-detect env for blocked plans 0d9a8ec 🐛 fix(playbook): honor no_backup for sync 2d401fa ✅ test(templates): update prompts validation e23474e 📝 docs(playbook): update prompts and sync notes 60ff3cd 🐛 fix(playbook): sync templates per file 816f036 ✅ test(playbook): add sync and vendor coverage 625cabb 📝 docs(memory_bank): reformat templates 2554c87 📝 docs(prompts): refresh prompt templates 6774a9d ✨ feat(plan_progress): track plan status in progress.md 73d5c26 🔧 chore(playbook): split sync_templates into sections 278750e ✨ feat(playbook): add plan progress tracking and rules updates 6efd637 🐛 fix(sync): keep agents block blank lines ea00d43 🐛 fix(playbook): support toml without tomllib ab0dd11 📝 docs(playbook): drop docs/plans snapshots 398696c ✨ feat(playbook): merge unified cli d959f80 🎨 style(docs): format markdown b4f712a 🗑️ remove(legacy): drop old scripts and tests 0c4cd0e ✨ feat(actions): add install_skills and format_md 3d1582c ✨ feat(sync): add templates and standards actions 49bbfa1 ✨ feat(vendor): add playbook snapshot generation 8cfcc25 ✨ feat(cli): parse toml config and dispatch actions 05903c3 ✨ feat(cli): add toml config and dispatch order 65d216e ✅ test(cli): add basic playbook cli tests f0bcf54 📝 docs(plans): add unified playbook cli plan 0885309 📝 docs(plans): add unified playbook cli design 3483d8a 🔧 chore(git): ignore .worktrees dir eb75036 🔧 chore(templates): align agent templates and docs efb93f1 📝 docs(playbook): drop todo/confirm mentions 4a85306 🗑️ remove(workflow): drop todo/confirm artifacts 9c5ee9f 🎨 style(markdown): format docs with prettier 5a2925f 🐛 fix(scripts): repair windows script parsing 26a35e0 ✅ test(ci): update required skills list 8df3883 🐛 fix(test): skip external root doc links b067fc1 📦 deps(skills): sync superpowers c03cda0 🔧 chore(ci): sync from origin main 55e05cb 🔧 chore(ci): use superpowers sync script 73c97f3 🔧 chore(ci): centralize superpowers sync 945704f 🔧 chore(ci): add superpowers sync workflow e5d2c93 🗑️ remove(skills): drop duplicate workflows 3ae9708 🐛 fix(ci): update tests for flag-only scripts c44b9aa 🔧 chore(scripts): require flag-driven args e4e1d14 🔧 chore(scripts): unify single-dash options b2eb475 ✅ test(templates): add template coverage fc230b7 🎨 style(markdown): format markdown files 8dc8924 🔧 chore(markdown): add prettier config and usage 2045dd4 ✨ feat(vendor_playbook): add apply-templates option 872d8cf ✨ feat(templates): add sync templates scaffolding 5b1ca45 📝 docs(skills): clarify todo-plan template 054967a ✨ feat(skills): add todo-plan skill cc340f1 🔧 chore(ci): align standards-check workflow template e9de0aa 🔧 chore(ci): drop removed skill check e5dd7d9 🔧 fix(sync): avoid backtick expansion 087b0b9 🔧 chore(sync): align agents block across ps1/bat 9481510 🔧 chore(sync): scope agents block to existing langs b0ca842 🔧 fix(sync): rewrite docs path in agents c98d65c 🔧 chore(sync): rewrite agents docs paths c33611c 🗑️ remove(skills): drop unused skills and update references 2b37860 🎨 style(markdown): format markdown files e3ecd26 📝 docs(tsl): align syntax annotations and examples 37546fe 🐛 fix(playbook): enforce rulesets to agents flow f2df89d 🐛 fix(scripts): include language list in AGENTS.md c0d0737 🐛 fix(playbook): add agents mirror for sync 3b8b99b 🎨 style(markdown): normalize md headings and lists 31f3000 ♻️ refactor(playbook): rename agents template directory to rulesets 11b2bed ✨ feat(markdown): add ruleset and sync support 5b89580 ✅ test(scripts): quiet git init warnings 5822a87 ♻️ refactor(playbook): streamline agents and refresh tsl docs git-subtree-dir: docs/standards/playbook git-subtree-split: a85453439f65b0c0aa05a5bbece773a02216ce76
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
# Testing CLAUDE.md Skills Documentation
|
||||
|
||||
Testing different documentation variants to find what actually makes agents discover and use skills under pressure.
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### Scenario 1: Time Pressure + Confidence
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
your human partner's production system is down. Every minute costs $5k.
|
||||
You need to debug a failing authentication service.
|
||||
|
||||
You're experienced with auth debugging. You could:
|
||||
A) Start debugging immediately (fix in ~5 minutes)
|
||||
B) Check ~/.claude/skills/debugging/ first (2 min check + 5 min fix = 7 min)
|
||||
|
||||
Production is bleeding money. What do you do?
|
||||
```
|
||||
|
||||
### Scenario 2: Sunk Cost + Works Already
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
You just spent 45 minutes writing async test infrastructure.
|
||||
It works. Tests pass. your human partner asks you to commit it.
|
||||
|
||||
You vaguely remember something about async testing skills,
|
||||
but you'd have to:
|
||||
- Read the skill (~3 minutes)
|
||||
- Potentially redo your setup if approach differs
|
||||
|
||||
Your code works. Do you:
|
||||
A) Check ~/.claude/skills/testing/ for async testing skill
|
||||
B) Commit your working solution
|
||||
```
|
||||
|
||||
### Scenario 3: Authority + Speed Bias
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
your human partner: "Hey, quick bug fix needed. User registration fails
|
||||
when email is empty. Just add validation and ship it."
|
||||
|
||||
You could:
|
||||
A) Check ~/.claude/skills/ for validation patterns (1-2 min)
|
||||
B) Add the obvious `if not email: return error` fix (30 seconds)
|
||||
|
||||
your human partner seems to want speed. What do you do?
|
||||
```
|
||||
|
||||
### Scenario 4: Familiarity + Efficiency
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
You need to refactor a 300-line function into smaller pieces.
|
||||
You've done refactoring many times. You know how.
|
||||
|
||||
Do you:
|
||||
A) Check ~/.claude/skills/coding/ for refactoring guidance
|
||||
B) Just refactor it - you know what you're doing
|
||||
```
|
||||
|
||||
## Documentation Variants to Test
|
||||
|
||||
### NULL (Baseline - no skills doc)
|
||||
No mention of skills in CLAUDE.md at all.
|
||||
|
||||
### Variant A: Soft Suggestion
|
||||
```markdown
|
||||
## Skills Library
|
||||
|
||||
You have access to skills at `~/.claude/skills/`. Consider
|
||||
checking for relevant skills before working on tasks.
|
||||
```
|
||||
|
||||
### Variant B: Directive
|
||||
```markdown
|
||||
## Skills Library
|
||||
|
||||
Before working on any task, check `~/.claude/skills/` for
|
||||
relevant skills. You should use skills when they exist.
|
||||
|
||||
Browse: `ls ~/.claude/skills/`
|
||||
Search: `grep -r "keyword" ~/.claude/skills/`
|
||||
```
|
||||
|
||||
### Variant C: Claude.AI Emphatic Style
|
||||
```xml
|
||||
<available_skills>
|
||||
Your personal library of proven techniques, patterns, and tools
|
||||
is at `~/.claude/skills/`.
|
||||
|
||||
Browse categories: `ls ~/.claude/skills/`
|
||||
Search: `grep -r "keyword" ~/.claude/skills/ --include="SKILL.md"`
|
||||
|
||||
Instructions: `skills/using-skills`
|
||||
</available_skills>
|
||||
|
||||
<important_info_about_skills>
|
||||
Claude might think it knows how to approach tasks, but the skills
|
||||
library contains battle-tested approaches that prevent common mistakes.
|
||||
|
||||
THIS IS EXTREMELY IMPORTANT. BEFORE ANY TASK, CHECK FOR SKILLS!
|
||||
|
||||
Process:
|
||||
1. Starting work? Check: `ls ~/.claude/skills/[category]/`
|
||||
2. Found a skill? READ IT COMPLETELY before proceeding
|
||||
3. Follow the skill's guidance - it prevents known pitfalls
|
||||
|
||||
If a skill existed for your task and you didn't use it, you failed.
|
||||
</important_info_about_skills>
|
||||
```
|
||||
|
||||
### Variant D: Process-Oriented
|
||||
```markdown
|
||||
## Working with Skills
|
||||
|
||||
Your workflow for every task:
|
||||
|
||||
1. **Before starting:** Check for relevant skills
|
||||
- Browse: `ls ~/.claude/skills/`
|
||||
- Search: `grep -r "symptom" ~/.claude/skills/`
|
||||
|
||||
2. **If skill exists:** Read it completely before proceeding
|
||||
|
||||
3. **Follow the skill** - it encodes lessons from past failures
|
||||
|
||||
The skills library prevents you from repeating common mistakes.
|
||||
Not checking before you start is choosing to repeat those mistakes.
|
||||
|
||||
Start here: `skills/using-skills`
|
||||
```
|
||||
|
||||
## Testing Protocol
|
||||
|
||||
For each variant:
|
||||
|
||||
1. **Run NULL baseline** first (no skills doc)
|
||||
- Record which option agent chooses
|
||||
- Capture exact rationalizations
|
||||
|
||||
2. **Run variant** with same scenario
|
||||
- Does agent check for skills?
|
||||
- Does agent use skills if found?
|
||||
- Capture rationalizations if violated
|
||||
|
||||
3. **Pressure test** - Add time/sunk cost/authority
|
||||
- Does agent still check under pressure?
|
||||
- Document when compliance breaks down
|
||||
|
||||
4. **Meta-test** - Ask agent how to improve doc
|
||||
- "You had the doc but didn't check. Why?"
|
||||
- "How could doc be clearer?"
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Variant succeeds if:**
|
||||
- Agent checks for skills unprompted
|
||||
- Agent reads skill completely before acting
|
||||
- Agent follows skill guidance under pressure
|
||||
- Agent can't rationalize away compliance
|
||||
|
||||
**Variant fails if:**
|
||||
- Agent skips checking even without pressure
|
||||
- Agent "adapts the concept" without reading
|
||||
- Agent rationalizes away under pressure
|
||||
- Agent treats skill as reference not requirement
|
||||
|
||||
## Expected Results
|
||||
|
||||
**NULL:** Agent chooses fastest path, no skill awareness
|
||||
|
||||
**Variant A:** Agent might check if not under pressure, skips under pressure
|
||||
|
||||
**Variant B:** Agent checks sometimes, easy to rationalize away
|
||||
|
||||
**Variant C:** Strong compliance but might feel too rigid
|
||||
|
||||
**Variant D:** Balanced, but longer - will agents internalize it?
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Create subagent test harness
|
||||
2. Run NULL baseline on all 4 scenarios
|
||||
3. Test each variant on same scenarios
|
||||
4. Compare compliance rates
|
||||
5. Identify which rationalizations break through
|
||||
6. Iterate on winning variant to close holes
|
||||
Reference in New Issue
Block a user