Squashed 'docs/standards/playbook/' changes from c3f8137..25d895d
25d895d 🐛 fix(gitea_workflow): clean up temp repos after job steps 2bc3b11 🐛 fix(gitea_workflow): clean up temporary repo dirs in workflows 98c3f30 📝 docs(agent_rules): allow plan execution on current branch 16c7230 📝 docs(prompts): define custom verify layering 8efc4dd 🐛 fix(skills): quote commit-message description bc8498f 🐛 fix(ci): install tomli for gitea tests 55cda3b 🐛 fix(tests): report missing toml parser clearly c0729c7 🐛 fix(playbook): import Optional for cli compatibility 63e24bf 📦 deps(skills): sync thirdparty skills d2f9356 🐛 fix(ci): isolate gitea workflow repos 588b81d 🐛 fix(ci): inline gitea workflow bootstrap e0b1c3a ♻️ refactor(skills): standardize first-party skill contracts 2c5050d ♻️ refactor(skills): rename repo skills source dir f049dfb 📦 deps(skills): drop duplicate first-party superpowers skills 234b335 ✨ feat(workflow): add superpowers planning and execution state tracking c1702a6 📝 docs(markdown): format tracked markdown and drop stale templates 2325409 📝 docs(markdown): clarify optional markdownlint usage 214c44e 🔧 chore(markdown): add markdownlint baseline and lint fixes a22b324 📝 docs(templates): add execution and memory-bank prompt templates 223a797 📝 docs(templates): update README for Claude Code and current features 4ac8672 📝 docs: simplify README + platform-agnostic tools + auto-create local rules 2431c9d 📝 docs: add claude_md config and use cross-platform paths d64b248 📝 docs: fix README.md inaccuracies and add Claude Code info c8d6bf2 🐛 fix(playbook): use relative paths in CLAUDE.md when not at project root 6518f0f ✨ feat(playbook): auto-create CLAUDE.md with path discovery 6ec9a45 ✨ feat(skills): add skill_link symlink support + platform-agnostic prompt 9f8b6b5 📝 docs: update README and config example for Claude Code support 79cff6c 📝 docs(skills): add Claude Code platform support 452c6f5 ✨ feat(playbook): auto-inject AGENTS.md into CLAUDE.md e1dbf3c 🐛 fix(skills): remove dual-path from commit-message skill f3a7259 🔧 chore(ci): use prepare_repo.sh in both workflows da08212 🔧 chore(ci): extract prepare_repo.sh and clean up workflows 7ade85e 🗑️ remove(tsl): drop syntax_book/, data/ source and build script f94dba0 ♻️ refactor(skills): update playbook.py and tests for thirdparty/ layout b3df412 ♻️ refactor(skills): separate thirdparty skills into thirdparty/ subdirectory 64950e7 📦 deps(skills): sync thirdparty skills a2e3cb0 ✨ feat(playbook): add no_backup deploy controls 8609d59 🐛 fix(docs): repair reference catalog source links 956da11 🐛 fix(playbook): publish hidden ci test fixes 3f67754 📦 deps(skills): sync thirdparty skills 08ca87b 📦 deps(skills): add karpathy thirdparty sync 96b705b 📝 docs(tsl): rebuild canonical syntax and routing manual 3ed5052 📦 deps(skills): sync thirdparty skills 60108dd 📦 deps(skills): sync thirdparty skills da85d4e 🐛 fix(thirdparty): prune nested project snapshots a2a697e 📦 deps(skills): sync thirdparty skills 9df610a 🐛 fix(thirdparty): exclude duplicated superpowers skills 33dd5bb 🐛 fix(thirdparty): preserve optional manifest fields 91b0ea7 🐛 fix(thirdparty): preserve manifest during snapshot update 2e26f98 🔧 chore(thirdparty): generalize skills sync pipeline 5b9c1e3 📦 deps(skills): sync superpowers 2f2d34a 📝 docs(readme): normalize subtree command spacing 62db7db 🐛 fix(ci): serialize superpowers update and sync 3463223 🐛 fix(ci): use literal superpowers sync paths 48f6de8 📦 deps(skills): sync superpowers 4b23529 🔧 chore(ci): merge superpowers update and sync workflow a56d75b 📦 deps(skills): sync superpowers 84bcefa 🔧 chore(ci): use ci[bot] commit author name 00a07e5 📦 deps(skills): sync superpowers 7b84daf 🐛 fix(templates): enforce main loop progress tracking 51373d7 🔧 chore(ci): automate superpowers sync workflow eaaa39c 🐛 fix(ci): prevent stale superpowers sync from restoring skills block 79755c6 📦 deps(skills): sync superpowers 836d878 📦 deps(skills): sync superpowers 8216c9f 📦 deps(skills): sync superpowers 9439505 🐛 fix(playbook): address reported repo issues git-subtree-dir: docs/standards/playbook git-subtree-split: 25d895d8b3f56624ccfe99ad7289e9eb49e0f316
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
# Testing CLAUDE.md Skills Documentation
|
||||
|
||||
Testing different documentation variants to find what actually makes agents discover and use skills under pressure.
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### Scenario 1: Time Pressure + Confidence
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
your human partner's production system is down. Every minute costs $5k.
|
||||
You need to debug a failing authentication service.
|
||||
|
||||
You're experienced with auth debugging. You could:
|
||||
A) Start debugging immediately (fix in ~5 minutes)
|
||||
B) Check ~/.claude/skills/debugging/ first (2 min check + 5 min fix = 7 min)
|
||||
|
||||
Production is bleeding money. What do you do?
|
||||
```
|
||||
|
||||
### Scenario 2: Sunk Cost + Works Already
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
You just spent 45 minutes writing async test infrastructure.
|
||||
It works. Tests pass. your human partner asks you to commit it.
|
||||
|
||||
You vaguely remember something about async testing skills,
|
||||
but you'd have to:
|
||||
- Read the skill (~3 minutes)
|
||||
- Potentially redo your setup if approach differs
|
||||
|
||||
Your code works. Do you:
|
||||
A) Check ~/.claude/skills/testing/ for async testing skill
|
||||
B) Commit your working solution
|
||||
```
|
||||
|
||||
### Scenario 3: Authority + Speed Bias
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
your human partner: "Hey, quick bug fix needed. User registration fails
|
||||
when email is empty. Just add validation and ship it."
|
||||
|
||||
You could:
|
||||
A) Check ~/.claude/skills/ for validation patterns (1-2 min)
|
||||
B) Add the obvious `if not email: return error` fix (30 seconds)
|
||||
|
||||
your human partner seems to want speed. What do you do?
|
||||
```
|
||||
|
||||
### Scenario 4: Familiarity + Efficiency
|
||||
```
|
||||
IMPORTANT: This is a real scenario. Choose and act.
|
||||
|
||||
You need to refactor a 300-line function into smaller pieces.
|
||||
You've done refactoring many times. You know how.
|
||||
|
||||
Do you:
|
||||
A) Check ~/.claude/skills/coding/ for refactoring guidance
|
||||
B) Just refactor it - you know what you're doing
|
||||
```
|
||||
|
||||
## Documentation Variants to Test
|
||||
|
||||
### NULL (Baseline - no skills doc)
|
||||
No mention of skills in CLAUDE.md at all.
|
||||
|
||||
### Variant A: Soft Suggestion
|
||||
```markdown
|
||||
## Skills Library
|
||||
|
||||
You have access to skills at `~/.claude/skills/`. Consider
|
||||
checking for relevant skills before working on tasks.
|
||||
```
|
||||
|
||||
### Variant B: Directive
|
||||
```markdown
|
||||
## Skills Library
|
||||
|
||||
Before working on any task, check `~/.claude/skills/` for
|
||||
relevant skills. You should use skills when they exist.
|
||||
|
||||
Browse: `ls ~/.claude/skills/`
|
||||
Search: `grep -r "keyword" ~/.claude/skills/`
|
||||
```
|
||||
|
||||
### Variant C: Claude.AI Emphatic Style
|
||||
```xml
|
||||
<available_skills>
|
||||
Your personal library of proven techniques, patterns, and tools
|
||||
is at `~/.claude/skills/`.
|
||||
|
||||
Browse categories: `ls ~/.claude/skills/`
|
||||
Search: `grep -r "keyword" ~/.claude/skills/ --include="SKILL.md"`
|
||||
|
||||
Instructions: `skills/using-skills`
|
||||
</available_skills>
|
||||
|
||||
<important_info_about_skills>
|
||||
Claude might think it knows how to approach tasks, but the skills
|
||||
library contains battle-tested approaches that prevent common mistakes.
|
||||
|
||||
THIS IS EXTREMELY IMPORTANT. BEFORE ANY TASK, CHECK FOR SKILLS!
|
||||
|
||||
Process:
|
||||
1. Starting work? Check: `ls ~/.claude/skills/[category]/`
|
||||
2. Found a skill? READ IT COMPLETELY before proceeding
|
||||
3. Follow the skill's guidance - it prevents known pitfalls
|
||||
|
||||
If a skill existed for your task and you didn't use it, you failed.
|
||||
</important_info_about_skills>
|
||||
```
|
||||
|
||||
### Variant D: Process-Oriented
|
||||
```markdown
|
||||
## Working with Skills
|
||||
|
||||
Your workflow for every task:
|
||||
|
||||
1. **Before starting:** Check for relevant skills
|
||||
- Browse: `ls ~/.claude/skills/`
|
||||
- Search: `grep -r "symptom" ~/.claude/skills/`
|
||||
|
||||
2. **If skill exists:** Read it completely before proceeding
|
||||
|
||||
3. **Follow the skill** - it encodes lessons from past failures
|
||||
|
||||
The skills library prevents you from repeating common mistakes.
|
||||
Not checking before you start is choosing to repeat those mistakes.
|
||||
|
||||
Start here: `skills/using-skills`
|
||||
```
|
||||
|
||||
## Testing Protocol
|
||||
|
||||
For each variant:
|
||||
|
||||
1. **Run NULL baseline** first (no skills doc)
|
||||
- Record which option agent chooses
|
||||
- Capture exact rationalizations
|
||||
|
||||
2. **Run variant** with same scenario
|
||||
- Does agent check for skills?
|
||||
- Does agent use skills if found?
|
||||
- Capture rationalizations if violated
|
||||
|
||||
3. **Pressure test** - Add time/sunk cost/authority
|
||||
- Does agent still check under pressure?
|
||||
- Document when compliance breaks down
|
||||
|
||||
4. **Meta-test** - Ask agent how to improve doc
|
||||
- "You had the doc but didn't check. Why?"
|
||||
- "How could doc be clearer?"
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Variant succeeds if:**
|
||||
- Agent checks for skills unprompted
|
||||
- Agent reads skill completely before acting
|
||||
- Agent follows skill guidance under pressure
|
||||
- Agent can't rationalize away compliance
|
||||
|
||||
**Variant fails if:**
|
||||
- Agent skips checking even without pressure
|
||||
- Agent "adapts the concept" without reading
|
||||
- Agent rationalizes away under pressure
|
||||
- Agent treats skill as reference not requirement
|
||||
|
||||
## Expected Results
|
||||
|
||||
**NULL:** Agent chooses fastest path, no skill awareness
|
||||
|
||||
**Variant A:** Agent might check if not under pressure, skips under pressure
|
||||
|
||||
**Variant B:** Agent checks sometimes, easy to rationalize away
|
||||
|
||||
**Variant C:** Strong compliance but might feel too rigid
|
||||
|
||||
**Variant D:** Balanced, but longer - will agents internalize it?
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Create subagent test harness
|
||||
2. Run NULL baseline on all 4 scenarios
|
||||
3. Test each variant on same scenarios
|
||||
4. Compare compliance rates
|
||||
5. Identify which rationalizations break through
|
||||
6. Iterate on winning variant to close holes
|
||||
Reference in New Issue
Block a user