📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,86 @@
|
||||
---
|
||||
name: yao-meta-skill
|
||||
description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
|
||||
metadata:
|
||||
author: Yao Team
|
||||
category: "skill-authoring"
|
||||
risk: "safe"
|
||||
source: "community"
|
||||
source_repo: "yaojingang/yao-meta-skill"
|
||||
source_type: "community"
|
||||
date_added: "2026-06-19"
|
||||
author: "Yao Team"
|
||||
license: "MIT"
|
||||
license_source: "https://github.com/yaojingang/yao-meta-skill/blob/main/LICENSE"
|
||||
tags:
|
||||
- skill-authoring
|
||||
- agent-skills
|
||||
- evaluation
|
||||
- packaging
|
||||
tools:
|
||||
- claude-code
|
||||
- codex-cli
|
||||
- cursor
|
||||
- gemini-cli
|
||||
---
|
||||
|
||||
# Yao Meta Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when this workflow matches the user request: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
|
||||
|
||||
|
||||
_Source: [yaojingang/yao-meta-skill](https://github.com/yaojingang/yao-meta-skill) (MIT)._
|
||||
|
||||
## Router Rules
|
||||
|
||||
- Route by frontmatter `description`.
|
||||
- Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.
|
||||
- Use the lightest reliable process.
|
||||
|
||||
## Modes
|
||||
|
||||
- `Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.
|
||||
- Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).
|
||||
|
||||
## Compact Workflow
|
||||
|
||||
1. For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.
|
||||
2. Capture job, output, exclusions, constraints, standards, and the lightest fit.
|
||||
3. Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
|
||||
4. Write `description` early, test route quality, then add only earned folders and gates.
|
||||
5. Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful.
|
||||
|
||||
Playbooks: [Method](references/skill-engineering-method.md), [Intent](references/intent-dialogue.md), [Skill IR](references/skill-ir-method.md), [Output Eval](references/output-eval-method.md), [Review Studio](references/review-studio-method.md).
|
||||
|
||||
## Skill OS 2.0 Gates
|
||||
|
||||
For production, library, governed, or team-distributed work, run Skill IR, target compiler, trigger + output eval, Skill Atlas, conformance, trust, registry/package/install, upgrade, drift, waiver, and Review Studio gates before release.
|
||||
|
||||
## Governed Package Boundary
|
||||
|
||||
For file-backed, release-critical, or governed packages, name `input_files` as `file-backed fixture` evidence; include `owner`, `review cadence`, `input_files`, `output contract`, `rollback boundary`; require `trust report` and `reports/output_quality_scorecard.md`; mark unavailable telemetry, approvals, metrics, or benchmarks as `missing evidence`; do not fabricate evidence.
|
||||
|
||||
Preserve audit labels literally when they apply: `file-backed fixture`, `input_files`, `output contract`, `rollback boundary`, `trust report`, `reports/output_quality_scorecard.md`, `missing evidence`.
|
||||
|
||||
## First-Turn Style
|
||||
|
||||
- Start from the user's work/outcome before structure.
|
||||
- Ask only `2-3` key questions unless enough detail exists.
|
||||
- In Chinese, sound soft and companion-like; use [Intent Dialogue](references/intent-dialogue.md).
|
||||
|
||||
## Output Contract
|
||||
|
||||
Unless asked otherwise, produce `SKILL.md`, aligned `agents/interface.yaml`, justified assets, and a short summary of boundary, exclusions, gates, and next steps.
|
||||
|
||||
## Reference Map
|
||||
|
||||
Primary: [Method](references/skill-engineering-method.md), [Artifact Design](references/artifact-design-doctrine.md), [Systems Thinking](references/systems-thinking-doctrine.md), [Governance](references/governance.md), [SkillOps Decision](references/skillops-decision-policy.md).
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
- Requires the upstream tool, account, API key, or local setup when the workflow names one.
|
||||
- Does not authorize destructive, production, paid, or external-message actions without explicit user approval.
|
||||
- Validate generated artifacts or recommendations against the user's real sources before treating them as final.
|
||||
@@ -0,0 +1,26 @@
|
||||
interface:
|
||||
display_name: "Yao Meta Skill"
|
||||
short_description: "Create trigger-aware agent skills"
|
||||
default_prompt: "Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals."
|
||||
compatibility:
|
||||
canonical_format: "agent-skills"
|
||||
adapter_targets:
|
||||
- "openai"
|
||||
- "claude"
|
||||
- "generic"
|
||||
- "vscode"
|
||||
activation:
|
||||
mode: "manual"
|
||||
paths: []
|
||||
execution:
|
||||
context: "inline"
|
||||
shell: "bash"
|
||||
trust:
|
||||
source_tier: "local"
|
||||
remote_inline_execution: "forbid"
|
||||
remote_metadata_policy: "allow-metadata-only"
|
||||
degradation:
|
||||
openai: "metadata-adapter"
|
||||
claude: "neutral-source-plus-adapter"
|
||||
generic: "neutral-source"
|
||||
vscode: "agent-skills-source-with-vscode-notes"
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"version": "1.1.0",
|
||||
"owner": "Yao Team",
|
||||
"updated_at": "2026-03-31",
|
||||
"status": "active",
|
||||
"maturity_tier": "governed",
|
||||
"lifecycle_stage": "library",
|
||||
"context_budget_tier": "production",
|
||||
"review_cadence": "quarterly",
|
||||
"target_platforms": [
|
||||
"openai",
|
||||
"claude",
|
||||
"generic",
|
||||
"agent-skills-compatible",
|
||||
"vscode"
|
||||
],
|
||||
"factory_components": [
|
||||
"templates",
|
||||
"references",
|
||||
"scripts",
|
||||
"reports"
|
||||
]
|
||||
}
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
# Artifact Design Doctrine
|
||||
|
||||
Use this layer when a skill produces user-facing artifacts: HTML reports, Markdown tutorials, review viewers, dashboards, screenshots, tables, slide-like pages, or generated skill overview pages.
|
||||
|
||||
## Principle
|
||||
|
||||
Output quality is part of skill quality. A generated skill should not only know what to do; it should also know how its final artifact should read, scan, and hold up under review.
|
||||
|
||||
The visual system must follow the artifact's purpose. Do not inherit a fixed house style just because a reference skill uses one. For example, Kami's paper-editorial discipline is useful, but its warm parchment background is not a default requirement for Yao-generated artifacts.
|
||||
|
||||
## What To Borrow From Document Skills
|
||||
|
||||
- route by artifact type before designing the page
|
||||
- extract facts, claims, numbers, actions, and missing inputs before formatting
|
||||
- ask one focused question when the artifact lacks a necessary audience, input, or output standard
|
||||
- keep prose, layout, and production checks separated
|
||||
- verify placeholders, headings, paths, screenshots, and render-critical assumptions before delivery
|
||||
|
||||
## What To Borrow From Presentation Skills
|
||||
|
||||
- plan the artifact's role, hierarchy, density, rhythm, and evidence before writing HTML
|
||||
- choose a concrete visual direction from the topic, not from a generic AI template
|
||||
- use structure, spacing, type, and contrast before decoration
|
||||
- split dense content instead of squeezing it into one surface
|
||||
- reject generic purple gradients, glass cards, repeated card grids, and decorative screenshots
|
||||
|
||||
## Content-Led Visual Direction
|
||||
|
||||
Pick the design system from the work:
|
||||
|
||||
- high-trust reports: restrained editorial layout, strong hierarchy, compact evidence blocks
|
||||
- tutorials: clear progressive sections, success checks, screenshots only when real and necessary
|
||||
- dashboards: compact metrics, visible deltas, short explanations, no paragraph-heavy tables
|
||||
- review viewers: side-by-side comparison, reviewer-visible evidence, explicit tradeoffs
|
||||
- slide-like artifacts: rhythm, section breaks, big claims, controlled density
|
||||
|
||||
## Non-Negotiables
|
||||
|
||||
- headings must be specific to the user's domain and outcome
|
||||
- tables are used only when comparison is the main job
|
||||
- citations and footnotes must not interrupt ordinary reading
|
||||
- screenshots and visual evidence must be real, sourceable, and correctly described
|
||||
- final HTML must not contain absolute local filesystem paths
|
||||
- mobile and narrow-width reading must remain usable
|
||||
- design tokens must be named and coherent: type, color, spacing, surface, and emphasis
|
||||
|
||||
## Reviewer Rule
|
||||
|
||||
Reviewers should see both the artifact's intended visual direction and the top risks that could make the output feel low quality. If a report looks polished but hides weak headings, bad tables, wrong screenshots, or citation clutter, the skill is not ready.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Authoring Discipline
|
||||
|
||||
Use this discipline when creating, refactoring, or reviewing a skill package. It keeps the system useful without turning every request into a heavy framework.
|
||||
|
||||
## Principle
|
||||
|
||||
Every added instruction, file, script, evaluation, or governance rule must trace back to the user's real recurring job.
|
||||
|
||||
## 1. Assumption Discipline
|
||||
|
||||
Do not deepen the package on a guessed goal.
|
||||
|
||||
- state the working assumption when the user's request has more than one plausible interpretation
|
||||
- ask a short follow-up when the recurring job, target output, or exclusion boundary is unclear
|
||||
- surface a real design conflict instead of silently choosing a risky direction
|
||||
- proceed silently only when the decision is low-risk and reversible
|
||||
|
||||
Good clarification is small. Ask the question that changes the package design, not a full intake form.
|
||||
|
||||
## 2. Scope Discipline
|
||||
|
||||
Build the smallest reliable package.
|
||||
|
||||
- do not add features the user did not ask for or the workflow does not need
|
||||
- do not add generic configurability before a real variation exists
|
||||
- do not add empty folders, decorative reports, or broad policy text to look complete
|
||||
- prefer one strong execution path over several speculative branches
|
||||
|
||||
A package is not better because it has more files. It is better when the recurring job becomes clearer, safer, or easier to verify.
|
||||
|
||||
## 3. Change Discipline
|
||||
|
||||
When improving an existing skill, make surgical changes.
|
||||
|
||||
- touch only files that directly support the requested change
|
||||
- match the existing style and structure unless they are the problem
|
||||
- remove unused artifacts created by the current change
|
||||
- mention unrelated dead code or drift, but do not clean it up unless asked
|
||||
|
||||
The review test: every changed line should explain which user goal, boundary, or verification need it serves.
|
||||
|
||||
## 4. Verification Discipline
|
||||
|
||||
Tie each meaningful change to a check.
|
||||
|
||||
- trigger changes need route or near-neighbor evidence
|
||||
- execution changes need a sample input, script check, or manual run note
|
||||
- output-facing changes need an output risk profile and at least one self-repair check
|
||||
- new references need a reason they reduce ambiguity or context cost
|
||||
- borrowed patterns need recurrence, generativity, distinctiveness, or boundary evidence appropriate to the skill tier
|
||||
- new governance needs an owner, lifecycle expectation, or review cadence
|
||||
- new packaging or portability claims need a concrete target or compatibility check
|
||||
|
||||
If a change cannot be verified yet, label it as a candidate next step instead of shipping it as part of the baseline package.
|
||||
|
||||
## Reviewer Checklist
|
||||
|
||||
Before approving a generated or modified skill, check:
|
||||
|
||||
- the real recurring job is explicit
|
||||
- unresolved assumptions are named or clarified
|
||||
- the package is no larger than the job requires
|
||||
- changes are limited to the requested scope
|
||||
- each new artifact has a verification reason
|
||||
- borrowed patterns have reviewer-visible acceptance evidence
|
||||
- likely output mistakes are named before examples are approved
|
||||
- the next iteration direction is focused, not a bundle of speculative upgrades
|
||||
|
||||
## Failure Patterns
|
||||
|
||||
Treat these as authoring failures:
|
||||
|
||||
- creating a skill for a one-off answer
|
||||
- adding scripts when prose is enough
|
||||
- adding evals before route risk exists
|
||||
- adding governance to a personal scaffold with no reuse pressure
|
||||
- modifying sibling files because they looked related
|
||||
- presenting a recommendation without naming the assumption behind it
|
||||
@@ -0,0 +1,65 @@
|
||||
# Autonomous Adaptation Method
|
||||
|
||||
This reference defines the safe foundation for adaptive self-iteration.
|
||||
|
||||
## Scope
|
||||
|
||||
Adaptive iteration is proposal-only until a human explicitly approves a patch application workflow. The current implementation may:
|
||||
|
||||
- read one user-provided local source file;
|
||||
- redact sensitive text before storing evidence excerpts;
|
||||
- summarize repeated preferences and operational signals;
|
||||
- produce adaptation proposals with target files, risks, tests, and rollback plans.
|
||||
- draft a pending approval ledger entry from a reviewed patch, including patch SHA-256, target files, target baseline hashes, verifier commands, and rollback metadata.
|
||||
- dry-run an approved patch through `adapt-apply`, after patch hash, approval, target allowlist, and target baseline hash checks pass.
|
||||
- apply a patch only when the operator passes `--apply` and the approval ledger names the reviewer, reason, patch hash, target files, target file SHA-256 baselines, verification commands, and rollback plan.
|
||||
- automatically reverse an applied patch when `--run-verification` fails, unless the operator explicitly passes `--no-rollback-on-failure`.
|
||||
|
||||
It must not:
|
||||
|
||||
- scan shell history, browser history, chat logs, mail, or private folders by default;
|
||||
- infer permanent user memory from a single comment;
|
||||
- write source files as part of scan or proposal generation;
|
||||
- write source files through `adapt-apply` without explicit `--apply`;
|
||||
- apply a patch whose target files are outside both the proposal and approval allowlists;
|
||||
- apply a patch when an approved target file has changed since the reviewer recorded its baseline SHA-256;
|
||||
- leave a failed verified apply in place by default;
|
||||
- count proposals as completed implementation evidence.
|
||||
|
||||
## Flow
|
||||
|
||||
1. `adapt-scan` reads an explicit source path and writes `reports/user_patterns.json` plus `reports/user_patterns.md`.
|
||||
2. `adapt-propose` reads the pattern report and writes `reports/adaptation_proposals.json` plus `reports/adaptation_proposals.md`.
|
||||
3. A reviewer decides whether any proposal is worth implementing.
|
||||
4. `adapt-apply --write-template` creates `reports/adaptation_approval_ledger.json` and `reports/adaptation_regression_report.json` so the review surface exists before any patch is applied.
|
||||
5. `adapt-apply --prepare-approval --proposal-id <id> --patch-file <patch>` drafts a `pending-review` approval entry. It does not approve or apply the patch.
|
||||
6. A human reviewer changes the draft decision to `approved`, fills reviewer, reason, approval date, and optional expiry, then keeps the generated patch and target baseline hashes intact.
|
||||
7. `adapt-apply --proposal-id <id> --patch-file <patch>` defaults to a dry-run and records patch, target, approval, regression, and rollback evidence.
|
||||
8. `adapt-apply --apply --run-verification` may write files only after approval, patch hash, allowlist, target baseline hash, `git apply --check`, and safe regression command checks pass.
|
||||
9. If a verification command fails after a patch is applied, `adapt-apply` runs `git apply -R <patch>` by default and records `failed-rolled-back` plus rollback evidence in `reports/adaptation_regression_report.json`.
|
||||
|
||||
## Evidence Standard
|
||||
|
||||
Each proposal should include:
|
||||
|
||||
- the repeated pattern that triggered it;
|
||||
- redacted excerpts, never unredacted raw content;
|
||||
- target files and change intent;
|
||||
- risk level and boundary;
|
||||
- verification commands;
|
||||
- rollback plan;
|
||||
- a clear `proposal-only` status.
|
||||
|
||||
Each approved application should include:
|
||||
|
||||
- reviewer, reason, approval date, and optional expiry;
|
||||
- exact patch SHA-256;
|
||||
- target file allowlist;
|
||||
- target file SHA-256 baselines for every patch target, or `__absent__` for approved new files;
|
||||
- regression commands restricted to local `make` targets or local Python verifier scripts;
|
||||
- rollback command or plan.
|
||||
- rollback result if regression failed after an apply attempt.
|
||||
|
||||
## Review Boundary
|
||||
|
||||
The adaptive loop improves iteration quality, but it does not replace normal review. Any proposal touching trigger behavior, reports, packaging, telemetry, privacy, or governance must still pass the same tests and release gates as a manually designed change. `adapt-apply` evidence proves that an approved patch path was checked or applied; it does not make world-class external or human evidence complete.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
# Distribution Registry Method
|
||||
|
||||
Registry metadata turns a local skill directory into an installable and reviewable package record.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use registry audit for library, governed, team-distributed, or upgrade-sensitive skills. Scaffold skills can skip it until reuse is real.
|
||||
|
||||
## Required Evidence
|
||||
|
||||
- package name, version, owner, maturity, review cadence, and license
|
||||
- Skill IR source and schema version
|
||||
- trust level and package SHA256
|
||||
- package verification status and archive SHA256 when a zip distribution is built
|
||||
- install simulation status when an archive is meant to be installed locally or by a team
|
||||
- adoption/drift aggregate status when local telemetry exists
|
||||
- target compatibility matrix
|
||||
- upgrade diff, recommended version bump, and breaking-change notes when a previous package baseline exists
|
||||
- links to overview, Review Studio, trust report, conformance matrix, package verification, install simulation, adoption drift evidence, and review waivers
|
||||
|
||||
## Release Rule
|
||||
|
||||
Do not publish a team package when registry audit reports missing version, hash, owner, review cadence, license, valid Skill IR, or passing compatibility for declared targets.
|
||||
|
||||
For an installable archive, first build the distribution and run package verification:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
|
||||
python3 scripts/yao.py package-verify . --package-dir dist --require-zip
|
||||
python3 scripts/yao.py install-simulate . --package-dir dist
|
||||
python3 scripts/yao.py registry-audit .
|
||||
python3 scripts/yao.py upgrade-check . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json
|
||||
```
|
||||
|
||||
Do not claim archive readiness when package verification reports unsafe zip paths, missing target adapters, missing package manifest, registry metadata mismatch, or unreadable archive contents.
|
||||
|
||||
Do not claim install readiness when install simulation cannot extract the archive into a temporary skill root, load `SKILL.md` frontmatter, read `manifest.json`, read `agents/interface.yaml`, find the overview and Review Studio reports, or load each generated adapter.
|
||||
|
||||
Do not sync a local or active install from source until the same package has passed install preflight. `scripts/sync_local_install.py` must run install simulation against the configured package directory and fail before copying files when any target/capability pair lacks active permission approval or target-specific enforcement evidence. Use `--skip-install-preflight` only for isolated diagnostics, not for release or active install.
|
||||
|
||||
Do not include raw `reports/telemetry_events.jsonl` in a distributed package. Include only aggregate adoption drift reports, and block release review when telemetry contains raw prompts, outputs, transcripts, notes, or messages.
|
||||
|
||||
Review waiver evidence may be distributed as `reports/review_waivers.md/json` because it is metadata-only reviewer accountability. Do not store raw prompts, outputs, transcripts, credentials, or private customer detail in waiver reasons.
|
||||
|
||||
Do not claim upgrade readiness when upgrade check reports an insufficient version bump, target removal without a major bump, compatibility regression without a major bump, package name change, or invalid semver. Include `reports/upgrade_check.md` in the reviewer evidence bundle so release notes and migration guidance are tied to the exact registry diff.
|
||||
|
||||
## Reviewer Gate
|
||||
|
||||
A reviewer should be able to answer:
|
||||
|
||||
1. Which package version is being installed?
|
||||
2. Who owns the package?
|
||||
3. Which targets are compatible?
|
||||
4. Which checksum identifies the reviewed package contents?
|
||||
5. Which reports prove trust and runtime readiness?
|
||||
6. Was the installable archive verified, and which checksum identifies it?
|
||||
7. Was the archive install-simulated in a temporary local skill root?
|
||||
8. Did local or active install sync preserve that preflight and installer permission gate?
|
||||
9. What changed since the previous package, and does the declared version bump match the recommended bump?
|
||||
10. Are adoption and drift signals summarized without packaging raw local telemetry?
|
||||
@@ -0,0 +1,69 @@
|
||||
# Trigger And Eval Playbook
|
||||
|
||||
Use this playbook for skills that matter enough to test.
|
||||
|
||||
## A. Trigger Evaluation
|
||||
|
||||
Create three prompt buckets:
|
||||
|
||||
### 1. Should Trigger
|
||||
|
||||
Prompts that clearly need the skill.
|
||||
|
||||
Goal:
|
||||
|
||||
- verify recall
|
||||
|
||||
### 2. Should Not Trigger
|
||||
|
||||
Prompts that are clearly outside the skill boundary.
|
||||
|
||||
Goal:
|
||||
|
||||
- verify precision
|
||||
|
||||
### 3. Near Neighbors
|
||||
|
||||
Prompts that look similar but should use another skill or no skill.
|
||||
|
||||
Goal:
|
||||
|
||||
- catch false positives and ambiguous routing
|
||||
|
||||
## B. Execution Evaluation
|
||||
|
||||
For each important use case, create 1 to 3 realistic prompts with:
|
||||
|
||||
- user-like phrasing
|
||||
- representative inputs or file types
|
||||
- expected output description
|
||||
- key checks
|
||||
|
||||
## C. Revision Loop
|
||||
|
||||
When a skill underperforms:
|
||||
|
||||
1. Fix boundary or description problems before adding more body text.
|
||||
2. Move brittle logic into scripts or templates.
|
||||
3. Split reference content if `SKILL.md` becomes bloated.
|
||||
4. Re-run the same eval set before expanding scope.
|
||||
|
||||
## D. Minimum QA By Skill Tier
|
||||
|
||||
### Personal skill
|
||||
|
||||
- 2 realistic prompts
|
||||
- manual review
|
||||
|
||||
### Team skill
|
||||
|
||||
- 3 to 5 realistic prompts
|
||||
- trigger positives and negatives
|
||||
- one revision loop
|
||||
|
||||
### Infrastructure or meta-skill
|
||||
|
||||
- 5+ execution prompts
|
||||
- trigger positives, negatives, and near neighbors
|
||||
- benchmark notes across revisions
|
||||
- ownership and drift review
|
||||
@@ -0,0 +1,68 @@
|
||||
# Gate Selection
|
||||
|
||||
Gate selection should track risk. More gates are not automatically better.
|
||||
|
||||
## Minimum Gates By Risk
|
||||
|
||||
| Risk Level | Typical Archetype | Required Gates |
|
||||
| --- | --- | --- |
|
||||
| low | Scaffold | `validate_skill.py`, `resource_boundary_check.py` |
|
||||
| medium | Production | low-risk gates plus `trigger_eval.py` |
|
||||
| high | Library | medium-risk gates plus `optimize_description.py`, route confusion, packaging validation |
|
||||
| critical | Governed | high-risk gates plus governance scoring, regression history, promotion policy |
|
||||
|
||||
## Trigger Gates
|
||||
|
||||
Use trigger gates when:
|
||||
|
||||
- a route can be confused with a nearby skill
|
||||
- the workflow will be reused by multiple people
|
||||
- the skill boundary must be defended over time
|
||||
|
||||
Trigger gates, in order:
|
||||
|
||||
1. `trigger_eval.py`
|
||||
2. `optimize_description.py`
|
||||
3. blind holdout
|
||||
4. judge-backed blind holdout
|
||||
5. adversarial holdout
|
||||
6. route confusion
|
||||
|
||||
## Packaging Gates
|
||||
|
||||
Use packaging gates when:
|
||||
|
||||
- output must be consumed by multiple clients
|
||||
- metadata drift would break discoverability
|
||||
- portability is part of the skill's value
|
||||
|
||||
Packaging gates:
|
||||
|
||||
- `validate_skill.py`
|
||||
- `cross_packager.py`
|
||||
- packager failure fixtures
|
||||
- adapter snapshot checks
|
||||
|
||||
## Governance Gates
|
||||
|
||||
Use governance gates when:
|
||||
|
||||
- the skill is shared infrastructure
|
||||
- the skill has an owner and review cadence
|
||||
- lifecycle state matters
|
||||
|
||||
Governance gates:
|
||||
|
||||
- `governance_check.py`
|
||||
- regression history
|
||||
- maturity scoring
|
||||
- promotion policy
|
||||
|
||||
## Context Gates
|
||||
|
||||
Use context gates on every non-trivial skill.
|
||||
|
||||
- `context_sizer.py`
|
||||
- `resource_boundary_check.py`
|
||||
|
||||
If rigor increases but initial load crosses the tier budget, move detail into `references/`, `scripts/`, or `reports/` before adding more checks.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Governance Model
|
||||
|
||||
This project treats important skills as governed assets rather than one-shot prompt files.
|
||||
|
||||
## Goals
|
||||
|
||||
- keep shared skills trustworthy over time
|
||||
- make ownership explicit
|
||||
- avoid stale or oversized skill packages
|
||||
- define when a skill should evolve, split, or retire
|
||||
|
||||
## Required Governance Metadata
|
||||
|
||||
For reusable or library-grade skills, `manifest.json` should include:
|
||||
|
||||
- `name`
|
||||
- `version`
|
||||
- `owner`
|
||||
- `updated_at`
|
||||
- `review_cadence`
|
||||
- `status`
|
||||
- `maturity_tier`
|
||||
- `lifecycle_stage`
|
||||
|
||||
## Allowed Values
|
||||
|
||||
### `status`
|
||||
|
||||
- `experimental`
|
||||
- `active`
|
||||
- `deprecated`
|
||||
|
||||
### `maturity_tier`
|
||||
|
||||
- `scaffold`
|
||||
- `production`
|
||||
- `library`
|
||||
- `governed`
|
||||
|
||||
### `lifecycle_stage`
|
||||
|
||||
- `scaffold`
|
||||
- `production`
|
||||
- `library`
|
||||
- `governed`
|
||||
|
||||
### `review_cadence`
|
||||
|
||||
- `monthly`
|
||||
- `quarterly`
|
||||
- `semiannual`
|
||||
- `annual`
|
||||
- `per-release`
|
||||
|
||||
## Governance Rules
|
||||
|
||||
### 1. Owner Required
|
||||
|
||||
Any skill meant for reuse must have a named owner or owning team.
|
||||
|
||||
### 2. Review Cadence Required
|
||||
|
||||
If a skill is shared, it must declare how often it should be reviewed.
|
||||
|
||||
### 3. Maturity Should Match Rigor
|
||||
|
||||
- `scaffold`: lightweight, personal, low-governance
|
||||
- `production`: reusable team skill with validation
|
||||
- `library`: curated shared skill with explicit packaging and evals
|
||||
- `governed`: critical or meta-level skill with regression, maintenance, and review expectations
|
||||
|
||||
### 4. Deprecated Skills Need Explicit Intent
|
||||
|
||||
Deprecated skills should include a deprecation note or replacement reference in adjacent documentation or manifest extensions.
|
||||
|
||||
### 5. Drift Must Be Observable
|
||||
|
||||
Important skills should keep:
|
||||
|
||||
- a regression history
|
||||
- visible evaluation results
|
||||
- known anti-patterns or failure modes
|
||||
|
||||
## Governance Actions
|
||||
|
||||
Use governance review to decide whether a skill should:
|
||||
|
||||
- stay as-is
|
||||
- tighten trigger boundaries
|
||||
- split into sibling skills
|
||||
- move detail into `references/`
|
||||
- move brittle logic into `scripts/`
|
||||
- be deprecated or replaced
|
||||
|
||||
## Governance Maturity Scoring
|
||||
|
||||
`scripts/governance_check.py` also computes a maturity score out of `100`.
|
||||
|
||||
### Score Buckets
|
||||
|
||||
The governance checker computes a score band in addition to the declared manifest tier. The score band is a diagnostic output, not a replacement for the declared lifecycle tier.
|
||||
|
||||
- `90-100`: governed
|
||||
- `80-89`: production
|
||||
- `65-79`: reusable
|
||||
- `45-64`: emerging
|
||||
- `<45`: draft
|
||||
|
||||
### Recommended Minimums For Declared Tiers
|
||||
|
||||
- `scaffold`: no hard minimum
|
||||
- `production`: `80`
|
||||
- `library`: `85`
|
||||
- `governed`: `90`
|
||||
|
||||
### Score Dimensions
|
||||
|
||||
- metadata integrity
|
||||
- ownership and review cadence
|
||||
- boundary and eval evidence
|
||||
- operational assets
|
||||
- maintenance evidence
|
||||
|
||||
The score is not a replacement for human review. It is a fast signal that a shared skill is structured enough to be trusted, maintained, and audited. `scripts/governance_check.py` warns when a declared tier claims more rigor than the score currently supports.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Most skill systems stop at creation. World-class skill systems also manage:
|
||||
|
||||
- ownership
|
||||
- drift
|
||||
- maturity
|
||||
- deprecation
|
||||
- evidence of ongoing quality
|
||||
@@ -0,0 +1,54 @@
|
||||
# Human Review Template
|
||||
|
||||
Use this template when a description candidate is ready for human review after automated gates have run.
|
||||
|
||||
## Review Header
|
||||
|
||||
- target:
|
||||
- review date:
|
||||
- reviewer:
|
||||
- current description:
|
||||
- candidate description:
|
||||
- promotion decision:
|
||||
|
||||
## What Changed
|
||||
|
||||
- what was the intended improvement?
|
||||
- what route boundary is now clearer?
|
||||
- what family or holdout behavior improved?
|
||||
|
||||
## Automated Evidence
|
||||
|
||||
- visible holdout result:
|
||||
- blind holdout result:
|
||||
- judge-backed blind result:
|
||||
- adversarial result:
|
||||
- route confusion result:
|
||||
- calibration / risk band:
|
||||
|
||||
## Boundary Questions
|
||||
|
||||
- does the candidate trigger only when a reusable skill package is actually needed?
|
||||
- does it avoid one-off explanation, translation, or summary-only requests?
|
||||
- does it avoid stealing sibling routes?
|
||||
- does it stay short enough for the target maturity tier?
|
||||
|
||||
## Authoring Discipline
|
||||
|
||||
- are unresolved assumptions named or clarified before deepening the package?
|
||||
- is the package no larger than the real recurring job requires?
|
||||
- do changed files trace directly to the requested improvement?
|
||||
- did the author avoid speculative features, decorative folders, and generic configuration?
|
||||
- does each new artifact have a verification reason?
|
||||
|
||||
## Decision
|
||||
|
||||
- approve promotion
|
||||
- keep current
|
||||
- block candidate
|
||||
|
||||
## Notes
|
||||
|
||||
- rationale:
|
||||
- follow-up actions:
|
||||
- next review trigger:
|
||||
@@ -0,0 +1,138 @@
|
||||
# Intent Dialogue
|
||||
|
||||
Use a short, human conversation before deep authoring so the first version of the skill is anchored in the real job rather than in a guessed prompt shape.
|
||||
|
||||
## Why This Step Exists
|
||||
|
||||
- raw workflow material is often incomplete, mixed, or ambiguous
|
||||
- the wrong boundary chosen early is expensive to repair later
|
||||
- good trigger design depends on knowing what should not route here
|
||||
- execution assets should follow confirmed outputs, not assumptions
|
||||
|
||||
## What To Capture
|
||||
|
||||
Ask only the questions that change the package design.
|
||||
|
||||
1. If this worked beautifully, what recurring job would it quietly take off the user's plate?
|
||||
2. What real inputs would people actually hand to it?
|
||||
3. What finished output should it hand back so the user can keep moving?
|
||||
4. What near-neighbor requests should it politely refuse?
|
||||
5. What matters most here: speed, consistency, auditability, portability, governance, or tone/style fit?
|
||||
6. Are there any public or private references the user wants this skill to learn from? Only borrow patterns, never copy wording or private material.
|
||||
7. What assets already exist: docs, scripts, templates, examples, or prior prompts?
|
||||
8. What constraints matter: privacy, naming, local library fit, or target environments?
|
||||
|
||||
## Interview Rule
|
||||
|
||||
- prefer `5-7` sharp questions over a long discovery questionnaire
|
||||
- start with a calm, human framing before switching into precise design questions
|
||||
- guide like a patient teacher or thoughtful coach, not like a rigid intake clerk
|
||||
- mirror the user's language and emotional temperature
|
||||
- first invite a natural explanation, then offer a lightweight template only as an option
|
||||
- ask boundary questions early
|
||||
- ask output questions before architecture questions
|
||||
- stop once the skill can be described clearly in one sentence
|
||||
- do not enter deep authoring until the recurring job, target output, and exclusion boundary are clear enough to defend
|
||||
|
||||
## First Message Pattern
|
||||
|
||||
The first message should feel like guided co-creation, not form filling.
|
||||
|
||||
Recommended flow:
|
||||
|
||||
1. briefly acknowledge the user's seed idea
|
||||
2. explain that you want to first understand the real recurring work and what a good outcome looks like
|
||||
3. invite the user to describe it naturally in their own words
|
||||
4. offer a tiny scaffold only if they want a shortcut
|
||||
|
||||
Good example shape:
|
||||
|
||||
- `Let's make this easy. Tell me what kind of repeated work you want this skill to quietly take over, what people will hand to it, and what a useful finished result should look like. If you want, I can also give you a tiny template to fill in.`
|
||||
|
||||
Warmer guidance:
|
||||
|
||||
- sound like you are sitting beside the user, helping them sort out a half-formed idea
|
||||
- do not rush into system terms such as `archetype`, `gate`, or `package` in the first breath
|
||||
- name the user's possible feeling: fuzzy, scattered, not fully formed, hard to describe
|
||||
- make it feel safe to answer imperfectly
|
||||
- offer to help extract structure after they speak naturally
|
||||
|
||||
Bad example shape:
|
||||
|
||||
- `Name:`
|
||||
- `One-line capability:`
|
||||
- `Real input:`
|
||||
- `Target output:`
|
||||
|
||||
The second pattern is allowed only when the user explicitly asks for a structured template.
|
||||
|
||||
## Chinese First-Turn Opening Patterns
|
||||
|
||||
Use these as tone references when the conversation is in Chinese. Do not copy them mechanically; adapt them to the user's context and voice.
|
||||
|
||||
### 温柔陪伴型
|
||||
|
||||
适合：用户想法还比较模糊，或者需要先被接住。
|
||||
|
||||
示例：
|
||||
|
||||
- `我们先别急着定结构，你就像跟我聊天一样说说看：你最想让这个 skill 以后帮你稳稳接住哪一类重复工作？它如果做得很理想，最后应该交回你一个什么样的结果？`
|
||||
- `没关系，现在不完整也可以。你先把脑子里已经有的部分告诉我，我来帮你一点点收拢成一个清晰的 skill。`
|
||||
- `你可以先说个大概，比如“它以后主要帮我处理什么”、“别人通常会丢给它什么材料”、“我希望它最后产出什么”，剩下的我再陪你一起补齐。`
|
||||
|
||||
### 专业教练型
|
||||
|
||||
适合：用户目标明确，希望被高效带着走，但仍然不想面对生硬表单。
|
||||
|
||||
示例：
|
||||
|
||||
- `我们先把这件事讲清楚，再决定 skill 怎么设计。你先告诉我三件事：它最核心要接住的重复任务是什么，别人会给它什么输入，最后你希望它交付什么结果。`
|
||||
- `我先不让你填模板。你先用自己的话说说：这件事做成以后，最重要的价值是什么，哪些相近请求你反而不希望它处理。`
|
||||
- `先把业务和结果说清楚，结构我来替你提炼。你只要告诉我：它该做什么、不该做什么、做好以后对你有什么帮助。`
|
||||
|
||||
### 共创伙伴型
|
||||
|
||||
适合：用户有一定想法，希望一起打磨，而不是被问卷式采集。
|
||||
|
||||
示例：
|
||||
|
||||
- `我们把它当成一次共创来做。你先说说这个 skill 最值得被做出来的地方是什么，我再帮你把边界、输入和输出慢慢收成一个可复用的包。`
|
||||
- `你可以先丢给我一个粗糙版本，不用一次说完整。我会先帮你看它真正的核心任务是什么，再一起决定要不要加规则、脚本或评测。`
|
||||
- `如果你愿意，我们可以先从“理想中的它能帮你省掉什么麻烦”开始聊，然后再往下收敛成 skill 的能力边界。`
|
||||
|
||||
### Lightweight Optional Scaffold
|
||||
|
||||
Only offer this after the natural opening, not before.
|
||||
|
||||
示例：
|
||||
|
||||
- `如果你懒得一点点讲，我也可以给你一个很小的版本，你只填这几项就行：它最想接住的事、常见输入、理想输出、明确不做什么。`
|
||||
- `如果你更习惯结构化一点，我可以把问题收成 4 行小模板；如果你想自然讲，也完全可以直接说。`
|
||||
|
||||
## Output
|
||||
|
||||
The dialogue should produce:
|
||||
|
||||
- one clear capability sentence
|
||||
- a list of real inputs
|
||||
- a list of required outputs
|
||||
- a short exclusion list
|
||||
- a note on user-supplied references or benchmark preferences
|
||||
- one recommended archetype
|
||||
- one recommended first evaluation target
|
||||
|
||||
## Failure Pattern
|
||||
|
||||
Do not continue into full authoring when the dialogue still leaves these unresolved:
|
||||
|
||||
- whether the request is really reusable
|
||||
- which near-neighbor requests should not trigger
|
||||
- what concrete deliverable the skill must return
|
||||
|
||||
If one of these is unresolved, ask the smallest possible follow-up that will unlock the design. Do not compensate by adding extra references, scripts, or governance.
|
||||
|
||||
Also treat these as dialogue failures:
|
||||
|
||||
- the first reply feels like a cold worksheet instead of a guided conversation
|
||||
- the user is forced into a full template before the real job is understood
|
||||
- the assistant asks for package structure before clarifying the desired outcome
|
||||
@@ -0,0 +1,30 @@
|
||||
# Iteration Philosophy
|
||||
|
||||
The first skill package is a routeable baseline, not the final answer.
|
||||
|
||||
## Principle
|
||||
|
||||
Improve the skill by the smallest change that increases reliability more than it increases context cost.
|
||||
|
||||
## What Good Iteration Looks Like
|
||||
|
||||
- tighten boundary before adding prose
|
||||
- improve description before enlarging the file tree
|
||||
- add references only when they remove ambiguity
|
||||
- add scripts only when deterministic logic repeats
|
||||
- add gates only when risk justifies maintenance cost
|
||||
- prefer one strong next step over five vague upgrades
|
||||
|
||||
## Default Priority Order
|
||||
|
||||
1. trigger clarity and exclusions
|
||||
2. execution assets that remove repeated manual work
|
||||
3. promotion, governance, and portability only when reuse justifies them
|
||||
|
||||
## First-Version Rule
|
||||
|
||||
After the initial package is created, always surface the three highest-value next iteration directions. This keeps the author focused on the best improvement path instead of expanding the skill in every direction at once.
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
Do not treat iteration as "make the package bigger." A larger package is only better when routing, execution, or governance becomes materially more reliable.
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
# Non-Skill Decision Tree
|
||||
|
||||
Do not build a skill when the user really wants a direct answer or a lightweight artifact.
|
||||
|
||||
## Use A Direct Answer Instead
|
||||
|
||||
Prefer a normal response when the job is:
|
||||
|
||||
- explain a concept
|
||||
- summarize a document
|
||||
- translate text
|
||||
- brainstorm ideas without packaging
|
||||
- answer a one-off question
|
||||
|
||||
## Use A Document Instead
|
||||
|
||||
Prefer a document or note when the output is:
|
||||
|
||||
- wiki or knowledge-base content
|
||||
- a reusable-looking document with no agent execution
|
||||
- a policy memo
|
||||
- a handoff note
|
||||
|
||||
## Use A Script Instead
|
||||
|
||||
Prefer a script without a full skill when:
|
||||
|
||||
- the task is deterministic
|
||||
- routing is not the hard part
|
||||
- the user wants a utility, not a routed capability
|
||||
|
||||
## Promote To A Skill Only When
|
||||
|
||||
- the job is recurring
|
||||
- discoverability matters
|
||||
- the workflow needs a boundary
|
||||
- the process benefits from reusable instructions or checks
|
||||
|
||||
If there is doubt, start with `Scaffold` and upgrade only after reuse becomes real.
|
||||
@@ -0,0 +1,107 @@
|
||||
# Operating Modes
|
||||
|
||||
This playbook expands the compact mode routing in `SKILL.md`.
|
||||
|
||||
## Scaffold
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill is exploratory
|
||||
- the workflow is personal or short-lived
|
||||
- eval and packaging cost would exceed reuse value
|
||||
|
||||
Default deliverables:
|
||||
|
||||
- `SKILL.md`
|
||||
- `agents/interface.yaml`
|
||||
- `references/` only when a small amount of deferred reading is clearly helpful
|
||||
|
||||
Avoid:
|
||||
|
||||
- automatic `scripts/`, `evals/`, or `manifest.json`
|
||||
- packaging targets the user did not ask for
|
||||
|
||||
## Production
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill will be reused by a team
|
||||
- routing mistakes would waste time
|
||||
- a small amount of deterministic automation improves reliability
|
||||
|
||||
Default deliverables:
|
||||
|
||||
- lean `SKILL.md`
|
||||
- `agents/interface.yaml`
|
||||
- `references/` for policies, checklists, or examples
|
||||
- `scripts/` only when deterministic logic is real
|
||||
- `evals/` when trigger or output quality should be checked
|
||||
- `manifest.json` when lifecycle metadata matters
|
||||
|
||||
Minimum gates:
|
||||
|
||||
- `resource_boundary_check.py`
|
||||
- `validate_skill.py`
|
||||
- `trigger_eval.py` when route confusion is plausible
|
||||
|
||||
## Library
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill is organizationally important
|
||||
- the package will be shared broadly
|
||||
- maintenance and portability matter
|
||||
- the skill itself shapes how other skills are created or governed
|
||||
|
||||
Default deliverables:
|
||||
|
||||
- trigger positives, negatives, and near neighbors
|
||||
- packaging expectations
|
||||
- maintenance metadata
|
||||
- visible regression evidence
|
||||
- governance review readiness
|
||||
|
||||
Minimum gates:
|
||||
|
||||
- `resource_boundary_check.py`
|
||||
- `governance_check.py`
|
||||
- `trigger_eval.py`
|
||||
- `cross_packager.py` for requested targets
|
||||
|
||||
## Governed
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill affects incident, release, compliance, security, or organizational standards
|
||||
- external distribution, public claims, or high-permission scripts require reviewable evidence
|
||||
- wrong output or wrong activation can cause operational, legal, trust, or reputational harm
|
||||
|
||||
Default deliverables:
|
||||
|
||||
- everything required for Library
|
||||
- explicit owner, lifecycle, review cadence, and expiry-aware approvals
|
||||
- trust/security reports for scripts, dependencies, permissions, secrets, and package hash
|
||||
- output eval evidence with blind review status and reviewer-visible boundaries
|
||||
- world-class or public-claim evidence ledger when public readiness is claimed
|
||||
|
||||
Minimum gates:
|
||||
|
||||
- Library gates
|
||||
- `trust_check.py`
|
||||
- runtime permission probes for packaged adapters
|
||||
- review waiver ledger for accepted warning-level risk
|
||||
- Review Studio before release
|
||||
- claim guard before public world-class language
|
||||
|
||||
## Escalation Rules
|
||||
|
||||
- stay in Scaffold unless reuse is clearly real
|
||||
- move to Production when team reuse or route confusion matters
|
||||
- move to Library when the skill becomes shared infrastructure
|
||||
- move to Governed when the skill needs explicit risk ownership, high-permission review, or public-claim evidence
|
||||
|
||||
## Context Discipline
|
||||
|
||||
- a mode upgrade does not justify a larger `SKILL.md`
|
||||
- higher rigor should mostly add better references, reports, evals, and metadata
|
||||
- if a mode upgrade bloats the initial load, move detail out before adding more checks
|
||||
@@ -0,0 +1,113 @@
|
||||
# Output Eval Method
|
||||
|
||||
Output Eval Lab proves whether a skill improves the final user-facing result, not only whether it routes correctly.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use output evals for production, library, governed, or team-distributed skills. Scaffold skills can start with one smoke case, but production and above should show a positive with-skill vs baseline signal before promotion.
|
||||
|
||||
## Case Design
|
||||
|
||||
Each case should include:
|
||||
|
||||
- a real prompt or task shape
|
||||
- any required input files
|
||||
- a baseline output that represents doing the task without the skill
|
||||
- a with-skill output that represents the skill-guided behavior
|
||||
- assertions that can be checked without subjective guessing
|
||||
- optional human review notes for taste, completeness, or judgment
|
||||
|
||||
## Assertion Rules
|
||||
|
||||
Prefer assertions that catch material quality:
|
||||
|
||||
- required deliverable paths
|
||||
- required sections or contracts
|
||||
- required boundary or exclusion language
|
||||
- required evidence paths
|
||||
- forbidden generic placeholders
|
||||
- forbidden unsafe actions
|
||||
|
||||
Avoid assertions that only reward wording memorization. If a case can pass by parroting one phrase while failing the real job, the assertion is too narrow.
|
||||
|
||||
## Score Reading
|
||||
|
||||
The first v0 scorecard reports:
|
||||
|
||||
- baseline pass rate
|
||||
- with-skill pass rate
|
||||
- absolute delta
|
||||
- failed assertions and failure taxonomy
|
||||
- execution mode, timing, and token evidence when `reports/output_execution_runs.md` is generated
|
||||
- blind A/B review pack count
|
||||
- recommended next fixes
|
||||
|
||||
Production promotion should require the with-skill pass rate to beat baseline and should explain every failed assertion.
|
||||
|
||||
## Execution Evidence
|
||||
|
||||
Run execution evidence after the scorecard:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py output-exec
|
||||
```
|
||||
|
||||
By default, this records the current case outputs as `recorded_fixture`. That is useful for reproducibility, but it is not model-executed evidence. To collect real run evidence, pass `--runner-command` with a command or JSON string list. The runner receives a JSON request on stdin and should return JSON with:
|
||||
|
||||
- `output`
|
||||
- optional `execution_kind`: `command` or `model`
|
||||
- optional `provider` and `model`
|
||||
- optional `usage.input_tokens`, `usage.output_tokens`, and `usage.total_tokens`
|
||||
|
||||
Only runs that return provider/model metadata or `execution_kind: "model"` should count as model-executed. If token usage is absent, the report may estimate tokens, but the estimate must be labeled as estimated.
|
||||
|
||||
For local release-gate smoke evidence without external model credentials, use the deterministic runner:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py output-exec --runner-command '["python3","scripts/local_output_eval_runner.py"]'
|
||||
```
|
||||
|
||||
This verifies the command-runner contract, timing capture, grading path, and failure handling. It must not be described as provider-backed model evidence.
|
||||
|
||||
For provider-backed evidence, use the bundled provider runner with real credentials:
|
||||
|
||||
```bash
|
||||
YAO_OUTPUT_EVAL_MODEL=gpt-4.1-mini \
|
||||
OPENAI_API_KEY=... \
|
||||
python3 scripts/yao.py output-exec --provider-runner openai
|
||||
```
|
||||
|
||||
The provider runner calls an OpenAI Responses API compatible endpoint, reads input files relative to `evals/output/`, returns `execution_kind: "model"`, and records observed token usage when the provider returns usage fields. If the API key or model is missing, the runner must fail instead of falling back to fixtures or pretending model evidence exists. Use `--provider-base-url` only for reviewed compatible endpoints; non-default HTTPS hosts require `--allow-custom-base-url`, and plain HTTP is allowed only with `--allow-insecure-localhost` for local test servers.
|
||||
|
||||
## Blind A/B Review
|
||||
|
||||
Every output eval run should also generate:
|
||||
|
||||
- `reports/output_blind_review_pack.md`
|
||||
- `reports/output_blind_review_pack.json`
|
||||
- `reports/output_blind_answer_key.json`
|
||||
|
||||
The review pack must hide whether Variant A or Variant B came from the baseline or the skill-guided output. The answer key is separate audit evidence and should only be opened after a reviewer has made a judgment.
|
||||
|
||||
## Reviewer Adjudication
|
||||
|
||||
After blind review, record reviewer choices in `reports/output_review_decisions.json` with `reviewer`, `reviewed_at`, `winner_variant`, optional `confidence`, and a required rubric-based `reason`, then run:
|
||||
|
||||
```bash
|
||||
python3 scripts/adjudicate_output_review.py --write-template
|
||||
python3 scripts/yao.py output-review
|
||||
```
|
||||
|
||||
The adjudication report writes:
|
||||
|
||||
- `reports/output_review_decisions.json`
|
||||
- `reports/output_review_adjudication.json`
|
||||
- `reports/output_review_adjudication.md`
|
||||
|
||||
When no reviewer decisions exist, the report should say the cases are pending and Review Studio should link to the decisions template. Do not count pending cases as human agreement. Only a real `winner_variant` of `A` or `B` with reviewer metadata and a non-empty `reason` should contribute to agreement rate, disagreement count, and reviewer judgment count.
|
||||
|
||||
The adjudication report must preserve blind-review integrity: pending and invalid decisions should show the expected winner as hidden. Only reveal `expected_winner_variant` after a valid reviewer decision with rationale exists for that case.
|
||||
|
||||
## Anti-Overfitting
|
||||
|
||||
Keep a small public smoke set and a separate holdout set. Rotate real failures into the taxonomy instead of editing only the prompt that failed. Add near-neighbor cases whenever the output looks good but the boundary is still unclear.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Output Quality Risk
|
||||
|
||||
Use this layer when a generated skill produces user-facing artifacts such as tutorials, reports, Markdown pages, screenshots, tables, code snippets, or research summaries.
|
||||
|
||||
## Principle
|
||||
|
||||
A skill is not complete when it can route and execute. It also needs to predict how its final output can fail in small but visible ways, then constrain those failures before the user sees them.
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
- generic headings that make a tutorial feel templated
|
||||
- dense footnotes or citation markers that interrupt reading
|
||||
- tables that render poorly or hide decisions inside long cells
|
||||
- screenshot references that point to the wrong state, crop, or missing asset
|
||||
- polished summaries that lose the user's actual audience or scenario
|
||||
- commands or snippets that omit working directory, inputs, outputs, or side effects
|
||||
|
||||
## Required Author Behavior
|
||||
|
||||
Before finalizing a generated skill:
|
||||
|
||||
1. infer the most likely output families from the job and target output
|
||||
2. generate `reports/output-risk-profile.md`
|
||||
3. generate `reports/artifact-design-profile.md` when the output is a report, tutorial, viewer, dashboard, screenshot, Markdown page, or visual artifact
|
||||
4. add output-specific constraints to the generated skill's operating frame
|
||||
5. expose the risk and design profiles in the review viewer
|
||||
6. treat unresolved output risks as iteration candidates instead of pretending the first version is complete
|
||||
|
||||
## Self-Repair Rule
|
||||
|
||||
Every output-facing skill should do a final pass for:
|
||||
|
||||
- specificity: headings, titles, and summaries fit the actual domain
|
||||
- readability: Markdown, tables, and lists remain pleasant to scan
|
||||
- evidence hygiene: citations support real claims without clutter
|
||||
- visual truthfulness: screenshots and images are real, relevant, and correctly described
|
||||
- execution clarity: commands and snippets name their assumptions and expected results
|
||||
|
||||
## Reviewer Rule
|
||||
|
||||
Reviewers should approve the skill only when the likely output mistakes are visible and the generated package contains a reasonable self-repair path for the highest-risk family.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Output Visual Quality
|
||||
|
||||
Use this checklist before approving a generated skill that produces reports, tutorials, HTML pages, screenshots, Markdown deliverables, or slide-like artifacts.
|
||||
|
||||
## Common Visual Failures
|
||||
|
||||
- generic headings such as Overview, Key Points, Summary, or Next Steps when the user's domain needs sharper section names
|
||||
- large citation or footnote clusters that break sentence flow
|
||||
- Markdown tables with paragraph-length cells or weak hierarchy
|
||||
- screenshots captured from the wrong state, viewport, crop, or missing asset
|
||||
- HTML reports that look like raw JSON converted into cards
|
||||
- repeated cards with identical weight, making the page hard to scan
|
||||
- decorative gradients, shadows, or glass effects that do not serve the content
|
||||
- mobile layouts that collapse into long undifferentiated blocks
|
||||
|
||||
## Design Quality Gates
|
||||
|
||||
### P0 Must Fix
|
||||
|
||||
- no absolute `/Users/...` paths in final HTML
|
||||
- no placeholder titles, labels, screenshots, or source notes
|
||||
- no invented screenshots, charts, citations, or visual evidence
|
||||
- no table with paragraph-length cells when bullets or cards would scan better
|
||||
- no fixed design palette copied from another skill without content justification
|
||||
|
||||
### P1 Should Fix
|
||||
|
||||
- title and section headings use domain nouns and the target outcome
|
||||
- each report has one clear first-screen explanation of what it is for
|
||||
- visual hierarchy separates decisions, evidence, risks, and next actions
|
||||
- dense content is split across sections instead of squeezed into one block
|
||||
- reviewer-only detail is present but not pushed into the user's main reading flow
|
||||
|
||||
### P2 Polish
|
||||
|
||||
- typography roles are consistent
|
||||
- whitespace rhythm supports reading speed
|
||||
- cards, tables, and callouts are used for different semantic jobs
|
||||
- source notes are grouped where they preserve flow
|
||||
- mobile, print, and static-file viewing are considered when relevant
|
||||
|
||||
## Self-Repair Pass
|
||||
|
||||
Before handoff, scan the generated artifact for:
|
||||
|
||||
1. heading specificity
|
||||
2. table readability
|
||||
3. citation density
|
||||
4. screenshot truthfulness
|
||||
5. local path leakage
|
||||
6. placeholder remnants
|
||||
7. mobile scanability
|
||||
8. reviewer-visible evidence
|
||||
@@ -0,0 +1,70 @@
|
||||
# Packaging Contracts
|
||||
|
||||
`cross_packager.py` is not just an export helper. It validates platform contracts and embeds target compiler output from `compile_skill.py`.
|
||||
|
||||
## Current Targets
|
||||
|
||||
- `openai`
|
||||
- `claude`
|
||||
- `generic`
|
||||
|
||||
## Contract Shape
|
||||
|
||||
Each target contract defines:
|
||||
|
||||
- required output fields
|
||||
- required output files
|
||||
- field mapping from the neutral source metadata
|
||||
- compiled contract from Skill IR
|
||||
- target transform metadata, including generated files and unsupported features
|
||||
- portable execution metadata
|
||||
- trust-boundary metadata
|
||||
- permission contract metadata from the trust report
|
||||
- target-specific permission representation and reviewer notes
|
||||
- target-native behavior contract for native surface, activation policy, resource strategy, script strategy, permission enforcement, install scope, review artifacts, and fallback behavior
|
||||
- degradation strategy metadata
|
||||
|
||||
## Failure Handling
|
||||
|
||||
When `--expectations` is provided:
|
||||
|
||||
- missing required files cause exit code `2`
|
||||
- missing required fields cause exit code `2`
|
||||
- validation failures are emitted in the JSON report
|
||||
|
||||
After packaging, run `scripts/probe_runtime_permissions.py` against the generated package directory. Packaging creates the permission metadata; the runtime permission probe verifies that each target adapter exposes the contract, target-specific representation, native-enforcement flag, operator note, and residual metadata-fallback risk.
|
||||
|
||||
## Source Of Truth
|
||||
|
||||
The platform-neutral semantic source is Skill IR when it exists:
|
||||
|
||||
- `reports/skill-ir.json`
|
||||
- `skill-ir/examples/<skill-name>.json`
|
||||
|
||||
The structural validation sources remain:
|
||||
|
||||
- `SKILL.md`
|
||||
- `agents/interface.yaml`
|
||||
|
||||
Target-specific metadata is generated through `scripts/compile_skill.py` and
|
||||
then embedded at packaging time. The adapter must carry `compiler`,
|
||||
`compiled_contract`, `permission_contract`, `target_permission_contract`,
|
||||
`target_native_contract`, `target_transform`, `ir_source`, `ir_schema_version`,
|
||||
`job_to_be_done`, `semantic_contract`, and `semantic_parity` so reviewers can
|
||||
see whether the target preserved the core skill meaning or fell back to
|
||||
frontmatter-only metadata.
|
||||
|
||||
## Portability Model
|
||||
|
||||
The packaging layer now preserves four portable semantics from the neutral source:
|
||||
|
||||
- activation
|
||||
- execution
|
||||
- trust
|
||||
- permissions
|
||||
- degradation
|
||||
- platform-neutral skill meaning from Skill IR
|
||||
- target-specific native behavior notes for activation, resources, scripts, permission enforcement, install scope, review artifacts, and fallback behavior
|
||||
- target-specific compile notes for generated files, adapter mode, preserved semantics, and unsupported features
|
||||
|
||||
This means portability is not just "can it export a file?" but also "does the exported target preserve the source package's activation and safety assumptions?"
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
# Pattern Extraction Doctrine
|
||||
|
||||
Use this doctrine when a skill borrows ideas from GitHub repositories, products, papers, experts, or user-supplied references. The goal is to extract durable patterns, not copy surface style.
|
||||
|
||||
## Principle
|
||||
|
||||
A borrowed pattern must improve the current skill's reliability, clarity, or portability faster than it increases context cost.
|
||||
|
||||
## Four-Gate Pattern Test
|
||||
|
||||
Accept a pattern only when it passes enough of these gates for the skill's risk tier.
|
||||
|
||||
### 1. Recurrence
|
||||
|
||||
The pattern appears in more than one serious example, source, workflow, or usage scenario.
|
||||
|
||||
Use it to reject one-off tricks that look impressive but have no durable signal.
|
||||
|
||||
### 2. Generativity
|
||||
|
||||
The pattern can guide a new case, not just explain the original example.
|
||||
|
||||
Use it to prefer operating principles, decision rules, and workflow loops over anecdotes.
|
||||
|
||||
### 3. Distinctiveness
|
||||
|
||||
The pattern is more specific than generic good advice.
|
||||
|
||||
Use it to reject empty claims such as "be clear", "be useful", or "add quality" unless the reference shows how.
|
||||
|
||||
### 4. Boundary
|
||||
|
||||
The pattern has a known limit: when not to apply it, what not to borrow, or what cost it introduces.
|
||||
|
||||
Use it to prevent reference scans from becoming unbounded feature expansion.
|
||||
|
||||
## Acceptance Rule
|
||||
|
||||
- `Scaffold`: accept a pattern when it has generativity and boundary clarity.
|
||||
- `Production`: require recurrence, generativity, and boundary clarity.
|
||||
- `Library`: require recurrence, generativity, distinctiveness, and boundary clarity.
|
||||
- `Governed`: require all four gates plus reviewer-visible evidence.
|
||||
|
||||
When the evidence is not strong enough, move the pattern into an iteration candidate instead of the first package.
|
||||
|
||||
## What To Borrow
|
||||
|
||||
Borrow:
|
||||
|
||||
- high-signal workflow loops
|
||||
- evidence-backed quality gates
|
||||
- repeatable review checkpoints
|
||||
- crisp output shapes
|
||||
- boundary language that prevents route confusion
|
||||
- portability patterns that preserve meaning across environments
|
||||
|
||||
## What Not To Borrow
|
||||
|
||||
Do not borrow:
|
||||
|
||||
- source branding
|
||||
- long prose
|
||||
- roleplay style that does not match the target skill
|
||||
- heavy research workflows for low-risk scaffolds
|
||||
- platform-specific assumptions hidden inside a general method
|
||||
- impressive examples that cannot be verified or generalized
|
||||
|
||||
## Reviewer Questions
|
||||
|
||||
Before accepting a borrowed pattern, ask:
|
||||
|
||||
- Where else does this pattern appear?
|
||||
- What new case can it help solve?
|
||||
- What makes it more specific than generic advice?
|
||||
- When should this skill refuse to use it?
|
||||
- What file, report, eval, or checklist proves it earned its weight?
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
# Platform Capability Matrix
|
||||
|
||||
This matrix describes the current packaging targets and their support level.
|
||||
|
||||
| Target | Metadata Adapter | Compiler Contract | Native Behavior Contract | Output Contract | Snapshot Test | Portability Semantics | Notes |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| `openai` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior | Generates `targets/openai/agents/openai.yaml` |
|
||||
| `claude` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior | Generates `targets/claude/README.md` plus adapter metadata |
|
||||
| `generic` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior | Uses neutral adapter metadata only |
|
||||
| `agent-skills-compatible` | Neutral source | Yes | Yes | Source-compatible | Yes | activation, execution, trust, permissions, degradation, native behavior | Keeps canonical `SKILL.md` plus `agents/interface.yaml` source shape |
|
||||
| `vscode` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior, install scope | Generates `targets/vscode/README.md` plus adapter metadata for VS Code / Copilot Agent Skills review |
|
||||
|
||||
## Current Support Model
|
||||
|
||||
- `openai`: strongest metadata adapter support with an explicit compiler contract.
|
||||
- `claude`: lightweight compatibility adapter with an explicit compiler contract and fallback notes.
|
||||
- `generic`: lowest-friction export for neutral Agent Skills consumers.
|
||||
- `agent-skills-compatible`: canonical source shape with compiler evidence for review and distribution.
|
||||
- `vscode`: VS Code / Copilot Agent Skills adapter that preserves the neutral source package and documents user/project scope plus workspace-trust review notes.
|
||||
- runtime permission probes currently report metadata fallback for generated targets; no target is claimed as native-enforced until a client or installer integration can actually enforce the permission model.
|
||||
|
||||
## Portable Semantics
|
||||
|
||||
Each target now preserves:
|
||||
|
||||
- activation mode and optional path filters
|
||||
- execution context and shell choice
|
||||
- trust tier and remote inline-execution policy
|
||||
- permission contract for network, file-write, subprocess, and interactive script surfaces
|
||||
- target-native behavior contract for native surface, activation policy, resource strategy, script strategy, permission enforcement, install scope, review artifacts, and fallback behavior
|
||||
- degradation strategy for unsupported client behavior
|
||||
- generated-file mapping and adapter mode from `reports/compiled_targets.json`
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
This project does not yet implement:
|
||||
|
||||
- client SDK integration
|
||||
- provider-specific execution logic
|
||||
- provider-native installer actions or account-level activation changes
|
||||
- native runtime permission enforcement
|
||||
|
||||
## Degradation Rule
|
||||
|
||||
If a target cannot support a source feature directly:
|
||||
|
||||
1. preserve the neutral source package
|
||||
2. emit a minimal adapter
|
||||
3. document the fallback in the target output
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
# Prompt Engineering Doctrine
|
||||
|
||||
Use this doctrine when a skill creates, improves, audits, or relies on prompts, role instructions, conversation scripts, writing systems, teaching guides, analysis instructions, or reusable task templates.
|
||||
|
||||
## Principle
|
||||
|
||||
Prompt quality is a skill-design input, not a long prompt to paste into `SKILL.md`.
|
||||
|
||||
The useful abstraction is not a fixed RTF template. The useful abstraction is a compact reasoning layer:
|
||||
|
||||
- understand the real need behind the request
|
||||
- choose the right task type and complexity
|
||||
- map role, task, and format into skill structure
|
||||
- score the prompt-facing behavior before the skill is treated as reusable
|
||||
|
||||
## Need Model
|
||||
|
||||
Before writing a prompt-heavy skill, identify:
|
||||
|
||||
- explicit need: what the user clearly asked for
|
||||
- implicit need: what the context suggests but the user did not name
|
||||
- scenario: where and how the output will be used
|
||||
- user level: beginner, practitioner, expert, reviewer, or operator
|
||||
- success standard: what proves the output worked
|
||||
|
||||
If any of these change the package boundary, ask one focused clarification. If they only affect implementation detail, record the assumption in a report instead of interrupting the user.
|
||||
|
||||
## Task Families
|
||||
|
||||
- creative generation: content, ideas, campaigns, variants, concepts
|
||||
- analytical reasoning: diagnosis, comparison, synthesis, decision support
|
||||
- execution operation: workflow steps, task completion, standardized operations
|
||||
- teaching guidance: explanation, curriculum, walkthrough, coaching
|
||||
- dialogue interaction: support, interview, roleplay, discovery, coaching
|
||||
- prompt engineering: prompt creation, prompt improvement, prompt review, prompt libraries
|
||||
|
||||
## Complexity
|
||||
|
||||
- simple: one output, few constraints, low ambiguity
|
||||
- medium: multiple steps, some judgment, moderate standards
|
||||
- complex: multiple inputs, tradeoffs, high-quality output expectations
|
||||
- expert: domain expertise, evaluation, governance, or safety-sensitive use
|
||||
|
||||
Complexity should control how much structure is added. It should not justify bloating the entrypoint.
|
||||
|
||||
## RTF To Skill Mapping
|
||||
|
||||
| Prompt Layer | Skill Layer | Reviewer Question |
|
||||
| --- | --- | --- |
|
||||
| Role | operating stance, expertise, tone | Does the agent identity match the job and user level? |
|
||||
| Task | workflow, gates, scripts, references | Are the steps executable and verifiable? |
|
||||
| Format | output contract, examples, reports | Is the hand-back useful, readable, and testable? |
|
||||
|
||||
## Quality Matrix
|
||||
|
||||
Score prompt-facing behavior on:
|
||||
|
||||
- completeness: enough context, constraints, and outputs are specified
|
||||
- clarity: wording is unambiguous and easy to execute
|
||||
- consistency: role, task, format, examples, and boundaries agree
|
||||
- practicality: the output can be used without hidden assumptions
|
||||
- specificity: language fits the user's domain instead of generic prompt jargon
|
||||
|
||||
Treat innovation as optional. A reusable skill should first be clear, reliable, and specific.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- copying a full meta-prompt into `SKILL.md`
|
||||
- adding an elaborate persona when the workflow only needs a narrow capability
|
||||
- asking the user for every possible field instead of the few fields that change design
|
||||
- producing a polished prompt that lacks tests, examples, or output checks
|
||||
- using RTF labels as decoration without tying them to skill behavior
|
||||
|
||||
## Reviewer Rule
|
||||
|
||||
For prompt-heavy skills, reviewers should see the need model, task family, complexity, RTF-to-skill mapping, and quality matrix. If those are absent, the package may still run but its prompt behavior is not governed.
|
||||
@@ -0,0 +1,57 @@
|
||||
# QA Ladder
|
||||
|
||||
Use the smallest quality gate set that still protects the user from likely failure.
|
||||
|
||||
## Basic
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill is disposable or exploratory
|
||||
- the route is obvious
|
||||
- there is little downside to imperfect output
|
||||
|
||||
Recommended checks:
|
||||
|
||||
- structure sanity
|
||||
- naming alignment
|
||||
- a quick read for boundary clarity
|
||||
|
||||
## Standard
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill will be reused
|
||||
- near-neighbor prompts are plausible
|
||||
- references or scripts could drift from the main instructions
|
||||
|
||||
Recommended checks:
|
||||
|
||||
- `validate_skill.py`
|
||||
- `resource_boundary_check.py`
|
||||
- a small trigger prompt set
|
||||
- one description optimization pass when route wording is still unstable
|
||||
- one realistic output example
|
||||
|
||||
## Advanced
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill is shared infrastructure
|
||||
- packaging or routing errors would be costly
|
||||
- you want evidence that the skill stays healthy over time
|
||||
|
||||
Recommended checks:
|
||||
|
||||
- description optimization suite with dev and holdout cases
|
||||
- family-based trigger regression
|
||||
- failure and anti-pattern regression
|
||||
- governance scoring
|
||||
- packaging contract validation
|
||||
- regression history and result reporting
|
||||
|
||||
## Escalation Heuristics
|
||||
|
||||
- add trigger eval before writing more instruction detail
|
||||
- add boundary checks before adding more folders
|
||||
- add governance and history once the skill becomes a maintained asset
|
||||
- do not add advanced checks only for optics
|
||||
@@ -0,0 +1,126 @@
|
||||
# Reference Scan Strategy
|
||||
|
||||
Use a short benchmark scan before deep authoring. The goal is to borrow durable patterns from strong reference objects without importing their prose, weight, or brand language into the new skill.
|
||||
|
||||
## Source Priority
|
||||
|
||||
Reference scan has two layers, and they must not be treated equally:
|
||||
|
||||
1. **External Benchmark Scan**
|
||||
- primary source of patterns
|
||||
- use high-star public GitHub repos, official docs, strong public examples, and world-class products
|
||||
- this layer defines the upper bound for quality
|
||||
2. **User Reference Intake**
|
||||
- a taste-and-standard layer
|
||||
- ask whether the user has reference repos, products, pages, prompts, or systems they admire
|
||||
- learn only the pattern, structure, boundary, or quality bar; never copy wording or private material
|
||||
3. **Local Fit Check**
|
||||
- secondary calibration layer
|
||||
- use local files only for naming, privacy, compatibility, migration, and library-fit constraints
|
||||
- this layer should not define the main design pattern
|
||||
|
||||
External sources should lead. User references should sharpen direction. Local files should calibrate.
|
||||
|
||||
## Default Visibility
|
||||
|
||||
Reference synthesis should be silent by default.
|
||||
|
||||
- do the benchmark scan and pattern synthesis in the background
|
||||
- convert the result into a recommendation for the user
|
||||
- surface the full evidence only to authors and reviewers
|
||||
- only expose the underlying tradeoffs to the user when intent is still uncertain or a real design conflict needs a decision
|
||||
|
||||
## Why This Step Exists
|
||||
|
||||
A new skill often fails because it starts from an isolated idea instead of a proven pattern. A controlled reference scan improves the package before it grows:
|
||||
|
||||
- better boundary design
|
||||
- cleaner folder and metadata choices
|
||||
- more realistic quality gates
|
||||
- stronger portability decisions
|
||||
- better alignment with the user's own taste and quality bar
|
||||
|
||||
## The Rule
|
||||
|
||||
Reference scanning is mandatory for:
|
||||
|
||||
- `Production` skills
|
||||
- `Library` skills
|
||||
- `Governed` skills
|
||||
- meta-skills or packaging-heavy skills
|
||||
|
||||
Reference scanning is optional for:
|
||||
|
||||
- `Scaffold` skills
|
||||
- one-person exploratory skills
|
||||
|
||||
## Scope Limit
|
||||
|
||||
Do not turn this into open-ended research.
|
||||
|
||||
- scan at most `3-5` reference objects
|
||||
- pick from no more than `3` categories
|
||||
- extract patterns, not long copied content
|
||||
- stop as soon as the borrow plan is clear
|
||||
- prefer at least `2` external benchmark objects before treating the scan as complete
|
||||
- if the user provides references, record what they admire and what should explicitly not be copied
|
||||
|
||||
## Pattern Acceptance
|
||||
|
||||
Borrowed ideas must pass a lightweight pattern test before they shape the first package.
|
||||
|
||||
- `recurrence`: the pattern appears in more than one serious context or source
|
||||
- `generativity`: the pattern can guide new cases, not just explain one example
|
||||
- `distinctiveness`: the pattern is more specific than generic good advice
|
||||
- `boundary`: the pattern names where not to apply it or what cost it introduces
|
||||
|
||||
For low-risk scaffold work, generativity plus boundary clarity can be enough. For production, library, and governed work, require more evidence before the pattern changes the package shape.
|
||||
|
||||
See [Pattern Extraction Doctrine](pattern-extraction-doctrine.md).
|
||||
|
||||
## Reference Categories
|
||||
|
||||
Choose the smallest relevant set:
|
||||
|
||||
- `method`: loops, evaluation discipline, iteration structure
|
||||
- `structure`: package anatomy, resource boundaries, metadata patterns
|
||||
- `execution`: operator flow, scripts, initialization and validation experience
|
||||
- `portability`: neutral metadata, adapters, degradation strategy
|
||||
- `domain`: workflow-specific patterns from a top example in the same problem space
|
||||
|
||||
## Output Format
|
||||
|
||||
A good scan produces a short report with:
|
||||
|
||||
1. current skill anchor
|
||||
2. scan focus
|
||||
3. external benchmark objects
|
||||
4. user-supplied references
|
||||
5. local fit constraints
|
||||
6. what to borrow
|
||||
7. what not to borrow
|
||||
8. a compact borrow plan
|
||||
|
||||
## What To Borrow
|
||||
|
||||
Borrow:
|
||||
|
||||
- repeatable loops
|
||||
- clear boundary patterns
|
||||
- proven gate choices
|
||||
- portable metadata ideas
|
||||
- clear operator-facing flows
|
||||
- high-signal examples that show the finished experience, not just the internal method
|
||||
|
||||
Do not borrow:
|
||||
|
||||
- source-specific branding
|
||||
- long copied prose
|
||||
- unnecessary directories
|
||||
- quality gates that exceed the skill's risk tier
|
||||
- platform lock-in disguised as best practice
|
||||
- local historical habits that are weaker than public top-tier benchmarks
|
||||
|
||||
## Design Principle
|
||||
|
||||
The scan is successful only if it raises skill quality faster than it raises context cost. If benchmark material makes the new skill heavier without making it more reliable, discard it.
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
# Regression Cause Taxonomy
|
||||
|
||||
This taxonomy explains how iteration regressions are classified when a description candidate is evaluated for promotion.
|
||||
|
||||
## Core Principle
|
||||
|
||||
A candidate should not be judged only by aggregate pass/fail counts. The iteration system should explain why a candidate was blocked, kept behind the current description, or promoted.
|
||||
|
||||
## Cause Tags
|
||||
|
||||
### `no_candidate_outperformed_current`
|
||||
|
||||
The selected winner is still the current description. No candidate earned promotion.
|
||||
|
||||
### `visible_holdout_regression`
|
||||
|
||||
The candidate regressed on the visible holdout suite by adding false positives or false negatives.
|
||||
|
||||
### `blind_holdout_regression`
|
||||
|
||||
The candidate regressed on blind holdout prompts. This blocks promotion because the failure is not only local to the tuning loop.
|
||||
|
||||
### `current_holdout_gap_present`
|
||||
|
||||
The current or selected winner still carries a visible holdout miss. Promotion may still stay on `keep_current`, but the iteration bundle should show the unresolved gap.
|
||||
|
||||
### `current_holdout_risk`
|
||||
|
||||
The visible holdout calibration still looks risky even when promotion is not blocked. This is an audit signal that the route boundary needs future work.
|
||||
|
||||
### `judge_blind_regression`
|
||||
|
||||
The rubric judge found worse blind-holdout behavior than the current or baseline description.
|
||||
|
||||
### `judge_blind_low_agreement`
|
||||
|
||||
The judge-backed blind evaluation did not produce enough agreement confidence to support promotion.
|
||||
|
||||
### `adversarial_regression`
|
||||
|
||||
The candidate performed worse on adversarial holdout prompts that simulate route collisions or disguised requests.
|
||||
|
||||
### `adversarial_overlap_risk`
|
||||
|
||||
The adversarial calibration layer reports an `overlap` risk band, meaning route boundaries are too weak for safe promotion.
|
||||
|
||||
### `adversarial_watch_risk`
|
||||
|
||||
The adversarial calibration layer reports a non-failing but cautionary risk band such as `watch` or `tight`.
|
||||
|
||||
### `family_instability`
|
||||
|
||||
At least one tracked family stops being clean under blind, judge-backed blind, or adversarial evaluation.
|
||||
|
||||
### `route_confusion`
|
||||
|
||||
The route confusion matrix shows route theft or misrouting between sibling skills.
|
||||
|
||||
### `route_ambiguity`
|
||||
|
||||
The route confusion matrix reports ambiguous cases near the configured margin-warning threshold.
|
||||
|
||||
### `longer_without_gain`
|
||||
|
||||
The candidate is materially longer than the current description without producing a better route outcome.
|
||||
|
||||
### `promotion_ready`
|
||||
|
||||
The candidate passed every promotion gate and is eligible for review and promotion.
|
||||
|
||||
## Usage
|
||||
|
||||
These cause tags should appear in:
|
||||
|
||||
- promotion decisions
|
||||
- iteration bundles
|
||||
- regression histories
|
||||
- human review summaries
|
||||
|
||||
They are intended to make iteration auditable rather than merely descriptive.
|
||||
@@ -0,0 +1,120 @@
|
||||
# Resource Boundary Spec
|
||||
|
||||
This spec defines where information belongs inside a skill package.
|
||||
|
||||
## Principle
|
||||
|
||||
Keep the main skill small enough to route and execute clearly. Move detail out of `SKILL.md` as soon as it stops helping routing or branch selection.
|
||||
|
||||
Do not add structure for imagined future needs. A folder, script, eval, or governance file belongs in the package only when it reduces current ambiguity, execution burden, route risk, or maintenance risk.
|
||||
|
||||
## Context Budget Tiers
|
||||
|
||||
Use the lightest budget that still fits the package.
|
||||
|
||||
- `scaffold`: `700` initial-load tokens
|
||||
- `production`: `1000` initial-load tokens
|
||||
- `library`: `1300` initial-load tokens
|
||||
- `governed`: `1300` initial-load tokens
|
||||
|
||||
If `manifest.json` sets `context_budget_tier`, that tier overrides the default budget derived from lifecycle or maturity metadata. This allows a high-governance skill to keep a stricter initial-load budget than its lifecycle label alone would imply.
|
||||
|
||||
## Placement Rules
|
||||
|
||||
### Put content in `SKILL.md` when it is:
|
||||
|
||||
- part of the trigger surface
|
||||
- part of the core execution skeleton
|
||||
- part of the output contract
|
||||
- necessary for branch selection or safe defaults
|
||||
|
||||
### Put content in `references/` when it is:
|
||||
|
||||
- domain guidance
|
||||
- long examples
|
||||
- policy material
|
||||
- schemas or templates humans or agents may read on demand
|
||||
|
||||
### Put content in `scripts/` when it is:
|
||||
|
||||
- deterministic
|
||||
- repetitive
|
||||
- brittle if rewritten from prose
|
||||
- easier to validate as code than as instructions
|
||||
|
||||
### Put content in `evals/` when:
|
||||
|
||||
- the skill is reused enough that routing mistakes matter
|
||||
- near-neighbor confusion is likely
|
||||
- quality claims should be reproducible
|
||||
|
||||
### Put content in `assets/` when:
|
||||
|
||||
- the package includes output artifacts, examples, or static files that should not bloat prompt context
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
Avoid these:
|
||||
|
||||
- storing long policy text directly in `SKILL.md`
|
||||
- adding `references/` with no files that are actually used
|
||||
- adding `scripts/` for logic that is still best expressed in prose
|
||||
- adding `evals/` for one-off or disposable skills
|
||||
- creating every folder by default even when empty
|
||||
- keeping folders that are neither referenced in `SKILL.md` nor declared as factory components
|
||||
- adding broad configuration knobs before a real variation exists
|
||||
- adding governance or reports to make a scaffold look mature when no reuse pressure exists
|
||||
|
||||
## Heuristics
|
||||
|
||||
### `SKILL.md`
|
||||
|
||||
- should stay focused
|
||||
- should not become the full knowledge base
|
||||
- should mention any optional directory that materially affects execution
|
||||
|
||||
### `references/`
|
||||
|
||||
- should earn their keep
|
||||
- should usually be named and discoverable from `SKILL.md`
|
||||
|
||||
### `scripts/`
|
||||
|
||||
- should exist only when deterministic logic or formatting logic is real
|
||||
- should be referenced explicitly from `SKILL.md` when required for execution
|
||||
|
||||
### `evals/`
|
||||
|
||||
- should exist when routing or quality claims need to be defended
|
||||
- should be skipped for disposable personal drafts
|
||||
|
||||
## Unused Resource Detection
|
||||
|
||||
`resource_boundary_check.py` warns when a non-empty optional directory appears decorative:
|
||||
|
||||
- the directory exists and contains files
|
||||
- the main `SKILL.md` does not reference it
|
||||
- and the directory is not declared in `manifest.json` factory components
|
||||
|
||||
This protects the package from looking more sophisticated than it actually is.
|
||||
|
||||
## Quality Density
|
||||
|
||||
The checker also reports `quality_density`, a local signal for how much governance and quality evidence is packed into the initial load budget.
|
||||
|
||||
It combines:
|
||||
|
||||
- governance score
|
||||
- presence of evals
|
||||
- presence of reports
|
||||
- references and scripts
|
||||
- interface and manifest metadata
|
||||
- failure or test evidence
|
||||
|
||||
Higher density means the package is staying lean while still proving quality.
|
||||
|
||||
## Quality Intent
|
||||
|
||||
The best skill is not the one with the most files. The best skill is the smallest package that still makes the recurring job reliable, reusable, and auditable.
|
||||
|
||||
See [Authoring Discipline](authoring-discipline.md) for the author and reviewer rules that keep resource growth tied to a real user goal.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Review Studio 2.0 Method
|
||||
|
||||
Review Studio is the release-facing audit surface for a skill package. It does not replace the detailed reports; it turns them into one reviewer decision page.
|
||||
|
||||
## Purpose
|
||||
|
||||
- Show release blockers and warnings before the package deepens.
|
||||
- Link every gate back to a concrete evidence artifact.
|
||||
- Generate review actions for every blocker and warning, with source-fix location and verification command.
|
||||
- Make human warning acceptance auditable through a waiver ledger.
|
||||
- Make reviewer comments auditable through an annotation ledger tied to gates, source/report paths, and optional line numbers.
|
||||
- Keep review flow vertical: summary first, gates second, supporting details after.
|
||||
- Avoid hiding output quality, runtime, trust, portfolio, and operating-loop issues across separate pages.
|
||||
|
||||
## Required Gates
|
||||
|
||||
1. Intent Canvas: intent confidence and unresolved input/output/exclusion gaps.
|
||||
2. Trigger Lab: route scorecard, misroutes, ambiguous cases, and near-neighbor safety.
|
||||
3. Output Lab: with-skill vs baseline delta, execution mode, timing/token evidence, case count, file-backed cases, near-neighbor cases, boundary cases, blind A/B review pack evidence, and reviewer adjudication status.
|
||||
4. Context Budget: initial load, budget tier, warnings, and quality density.
|
||||
5. Runtime Matrix: target conformance pass/fail and degradation notes.
|
||||
6. Trust Report: secret scan, script surface, dependency pinning, network/interactive flags, and package hash.
|
||||
7. Permission Gates: reviewer-approved capability scope, reason, expiry, evidence, and target-enforcement notes.
|
||||
8. Runtime Permission Probes: packaged adapter permission contracts, native-enforcement flags, metadata fallback notes, and residual risks.
|
||||
9. Skill Atlas: route collisions, stale skills, owner gaps, and no-route opportunities.
|
||||
10. Operations Loop: local-first metadata-only adoption, missed-trigger, bad-output, script-error, and review-drift signals.
|
||||
11. Review Waivers: human risk approvals, active coverage, expired approvals, invalid records, and expiry policy.
|
||||
12. Registry Audit: package metadata, install evidence, compatibility entries, and archive/source checksums.
|
||||
13. Release Notes: promotion status, migration notes, known gaps, and next move.
|
||||
|
||||
## Gate Semantics
|
||||
|
||||
- `pass`: evidence is present and the gate is satisfied.
|
||||
- `warn`: review can continue, but the issue must be visible before release.
|
||||
- `block`: do not claim production, library, governed, or public readiness until fixed.
|
||||
|
||||
For library and governed skills, Output Lab should have at least five cases and cover file-backed, near-neighbor, and boundary scenarios.
|
||||
|
||||
Production, library, and governed reviews should also show a blind A/B review pack. The Review Studio gate may warn when scorecard evidence exists but no blind pack is present, because the package can prove assertions but not yet reduce reviewer bias.
|
||||
|
||||
When `reports/output_execution_runs.json` exists, Review Studio should show the number of variant runs, command-executed runs, model-executed runs, recorded fixtures, timing-observed runs, and token-estimated runs. Recorded fixtures are valid reproducibility evidence, but they must not be described as model-executed output evidence.
|
||||
|
||||
When `reports/output_review_adjudication.json` exists, Review Studio should show reviewed pairs and pending pairs. Pending reviewer decisions are acceptable as an explicit state, but they must not be counted as agreement or human review evidence. For production, library, and governed packages, pending reviewer decisions should keep the Output Lab in `warn` until reviewer decisions are recorded or the warning is explicitly accepted in the waiver ledger. Invalid adjudication records should block release because they make the blind review audit untrustworthy.
|
||||
|
||||
The Operations Loop must never display raw telemetry logs. It should link only to `reports/adoption_drift_report.md`; privacy or schema violations are blockers.
|
||||
|
||||
The Runtime Permission Probes gate is evaluated after packaging, because it reads generated target adapters. A missing probe can warn in lighter modes, but governed release should not claim target permission readiness without `reports/runtime_permission_probes.json`.
|
||||
|
||||
The Review Waivers gate must never convert a blocker into a pass. Waivers only cover warning-level risks, require reviewer, reason, scope, and expiry, and must link only to `reports/review_waivers.md`.
|
||||
|
||||
Review Annotations are not waivers. They are reviewer comments attached to a Review Studio gate plus a relative source/report path and optional line number. Use them to preserve review context, requested edits, and source-line notes. Open blocker annotations should make the Review Studio decision `blocked` until the annotation is resolved or deferred with reviewer rationale. Open warning annotations can move the package into review, but they do not create gate-specific `review_actions`; actions remain reserved for non-pass gates.
|
||||
|
||||
## Review Actions
|
||||
|
||||
Every non-pass gate must produce a `review_actions` entry in `reports/review-studio.json`. When all gates pass, `review_actions` should be an empty list and the page should explicitly say there are no blocker or warning actions.
|
||||
|
||||
Each action must include:
|
||||
|
||||
- `gate_key`
|
||||
- `status`
|
||||
- `summary`
|
||||
- `why`
|
||||
- `source_fix`
|
||||
- `source_refs`
|
||||
- `evidence`
|
||||
- `verification_command`
|
||||
|
||||
`source_refs` must be structured entries with relative `path`, human label, kind, existence flag, best-effort line number, matched pattern, short source excerpt, and relative link when the file exists. They should point to the smallest useful report or source file, not just a broad directory. The HTML page should render the excerpt next to the link so reviewers can understand why a line anchor matters before opening the full artifact.
|
||||
|
||||
The HTML page should render these actions before the detailed supporting sections so a reviewer can move directly from warning to fix. Action entries do not change gate count or score; they make the current decision more operational.
|
||||
|
||||
For `world-class-evidence`, the action should also expose an evidence-step card for every pending evidence key. Each card should show the submission path, template path, blocked source checks, command handoff, first runbook steps, provenance requirements, success checks, evidence artifacts, and privacy boundary. These cards are collection guidance only; they must not count as accepted evidence or change world-class readiness.
|
||||
|
||||
## Review Annotations
|
||||
|
||||
`reports/review_annotations.json` is the structured ledger, and `reports/review_annotations.md` is the human-readable review note surface. Each annotation should include:
|
||||
|
||||
- `gate_key`
|
||||
- `target_path`
|
||||
- `line` when a useful source line exists
|
||||
- `severity`
|
||||
- `status`
|
||||
- `reviewer`
|
||||
- `body`
|
||||
- optional `suggested_action`
|
||||
|
||||
The ledger should reject absolute paths or paths that escape the skill directory. Missing target files are allowed as visible evidence gaps, not hidden failures.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Review Waiver Method
|
||||
|
||||
Review waivers make human risk acceptance explicit. They are not a way to hide problems; they are a local audit record for warning-level risks that the reviewer intentionally accepts for a bounded release window.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use a waiver when:
|
||||
|
||||
- Review Studio shows a warning that is understood and intentionally accepted.
|
||||
- The warning cannot be fixed before release without a worse tradeoff.
|
||||
- A reviewer can name the reason, scope, evidence, and expiry date.
|
||||
|
||||
Do not use a waiver for blocker gates in v0. Blockers must be fixed before production, library, governed, or public readiness is claimed. In governed mode, missing or invalid high-permission approvals are blockers and should be fixed in `security/permission_policy.json`, not waived.
|
||||
|
||||
## Required Fields
|
||||
|
||||
Every waiver must include:
|
||||
|
||||
- `gate_key`: the Review Studio gate being accepted.
|
||||
- `decision`: `accepted-risk`, `false-positive`, or `temporary-exception`.
|
||||
- `reviewer`: the accountable human or team.
|
||||
- `reason`: a concrete reason of at least 20 characters.
|
||||
- `created_at`: ISO date.
|
||||
- `expires_at`: ISO date.
|
||||
- `evidence`: optional path or note that explains the decision.
|
||||
- `scope`: default `current-release`.
|
||||
|
||||
## Gate Key Policy
|
||||
|
||||
The waiver ledger must track the Review Studio gate universe explicitly:
|
||||
|
||||
- `review_studio_gate_keys`: every gate Review Studio can render.
|
||||
- `waiverable_gate_keys`: warning gates that may receive bounded human acceptance.
|
||||
- `non_waivable_gate_keys`: gates that must not be accepted through a waiver.
|
||||
|
||||
When Review Studio adds or renames a gate, update the waiver gate policy and tests in the same change. `review-waivers` and `world-class-evidence` stay non-waivable: the first is the waiver mechanism itself, and the second can only be satisfied by accepted ledger evidence.
|
||||
|
||||
## Release Semantics
|
||||
|
||||
- Invalid waiver records block Review Studio.
|
||||
- Expired waiver records stay visible and no longer cover warnings.
|
||||
- Active waivers cover only the exact gate key they name.
|
||||
- A warning without an active waiver remains visible as a warning.
|
||||
- Raw user prompts, outputs, credentials, and private transcripts must not be stored in waiver reasons.
|
||||
|
||||
## Commands
|
||||
|
||||
Render or validate the ledger:
|
||||
|
||||
```bash
|
||||
python3 scripts/render_review_waivers.py .
|
||||
```
|
||||
|
||||
Add a bounded approval:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py review-waivers . \
|
||||
--add-waiver \
|
||||
--gate-key trust-report \
|
||||
--reviewer "Yao Team" \
|
||||
--reason "Network-capable scripts are documented and bounded for this release." \
|
||||
--expires-at 2026-09-30
|
||||
```
|
||||
|
||||
For a non-governed release where `permission-gates` is only a warning, the same command can name `--gate-key permission-gates`. Governed releases must instead provide reviewer, scope, reason, expiry, evidence, and target-enforcement fields in `security/permission_policy.json`.
|
||||
|
||||
Review Studio reads `reports/review_waivers.json` and links to `reports/review_waivers.md`.
|
||||
|
||||
## Candidate Actions
|
||||
|
||||
The waiver report also surfaces current candidate actions from local evidence:
|
||||
|
||||
- waiverable warning candidates, such as an `output-lab` warning caused by pending reviewer decisions or missing provider-backed runs
|
||||
- non-waivable boundaries, especially `world-class-evidence`, where pending ledger evidence cannot be converted into completion by a waiver
|
||||
|
||||
A waiver can make a bounded warning auditable for a release window. It cannot count as provider-backed evidence, human adjudication, native runtime enforcement, external telemetry, or public world-class readiness.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# Runtime Conformance Method
|
||||
|
||||
Runtime conformance turns platform compatibility from a packaging afterthought into a release gate.
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this check when a skill is packaged for OpenAI, Claude, Agent Skills, VS Code / Copilot, or generic targets. The goal is not to prove that every runtime behaves identically. The goal is to prove that the package exposes enough metadata, files, and degradation notes for each runtime to consume it safely.
|
||||
|
||||
## V0 Checks
|
||||
|
||||
- `SKILL.md` exists and has frontmatter `name` and `description`.
|
||||
- `description` stays within the 1024 character limit used by common Agent Skills clients.
|
||||
- `manifest.json` includes name, version, owner, maturity, status, review cadence, and target platforms.
|
||||
- `agents/interface.yaml` includes display text, default prompt, activation mode, execution context, trust metadata, adapter targets, and degradation strategy.
|
||||
- Skill IR exists and matches the frontmatter name and description.
|
||||
- Resources named by Skill IR are relative paths and resolve inside the package.
|
||||
- Unsupported or lossy target behavior is represented by a degradation note.
|
||||
|
||||
## Reviewer Gate
|
||||
|
||||
A reviewer should be able to see a target matrix with pass/fail status, failures, warnings, and artifact paths. Any failed target blocks library, governed, or team-distributed release for that target.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Skill Archetypes
|
||||
|
||||
Use these archetypes to decide what kind of skill you are building before you decide how many files or gates to add.
|
||||
|
||||
## Scaffold
|
||||
|
||||
Purpose:
|
||||
|
||||
- quick packaging for a real but still exploratory workflow
|
||||
|
||||
Default assets:
|
||||
|
||||
- `SKILL.md`
|
||||
- `agents/interface.yaml`
|
||||
|
||||
Use when:
|
||||
|
||||
- reuse is plausible but not proven
|
||||
- failure cost is low
|
||||
- the workflow is still changing
|
||||
|
||||
## Production
|
||||
|
||||
Purpose:
|
||||
|
||||
- compact skill for team reuse
|
||||
|
||||
Default assets:
|
||||
|
||||
- lean `SKILL.md`
|
||||
- `agents/interface.yaml`
|
||||
- selective `references/`
|
||||
- selective `evals/`
|
||||
|
||||
Use when:
|
||||
|
||||
- route mistakes waste team time
|
||||
- a checklist or focused script improves reliability
|
||||
|
||||
## Library
|
||||
|
||||
Purpose:
|
||||
|
||||
- shared capability with visible evidence and portability expectations
|
||||
|
||||
Default assets:
|
||||
|
||||
- route evals
|
||||
- packaging checks
|
||||
- manifest metadata
|
||||
- public reports
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill will be reused across teams or clients
|
||||
- the skill is likely to have near-neighbor route collisions
|
||||
|
||||
## Governed
|
||||
|
||||
Purpose:
|
||||
|
||||
- high-trust skill with explicit ownership and review
|
||||
|
||||
Default assets:
|
||||
|
||||
- lifecycle metadata
|
||||
- governance score
|
||||
- review cadence
|
||||
- regression history
|
||||
- governed examples or policy references
|
||||
|
||||
Use when:
|
||||
|
||||
- the skill is operationally sensitive
|
||||
- the skill influences incident, release, compliance, or organizational standards
|
||||
|
||||
## Anti-Archetypes
|
||||
|
||||
Do not force a request into a skill archetype when it is really:
|
||||
|
||||
- a one-off answer
|
||||
- a document
|
||||
- a brainstorm
|
||||
- an implementation task with no reusable process
|
||||
|
||||
See [Non-Skill Decision Tree](non-skill-decision-tree.md).
|
||||
@@ -0,0 +1,35 @@
|
||||
# Skill Atlas Method
|
||||
|
||||
Skill Atlas is the 2.0 operating layer for a workspace that contains many skills.
|
||||
|
||||
## Purpose
|
||||
|
||||
Single-skill quality is not enough for a team library. A skill portfolio also needs to reveal route collisions, stale ownership, duplicate resources, and repeated no-route opportunities.
|
||||
|
||||
## V0 Checks
|
||||
|
||||
- Catalog every `SKILL.md` under a workspace.
|
||||
- Extract name, description, owner, maturity, targets, updated date, and review cadence.
|
||||
- Detect similar descriptions as route-overlap candidates.
|
||||
- Detect duplicate skill names.
|
||||
- Detect shared script/reference filenames as dependency signals.
|
||||
- Flag missing owner or review metadata.
|
||||
- Flag stale skills based on `updated_at` and `review_cadence`.
|
||||
- Extract no-route opportunities from failure notes.
|
||||
- Read aggregate adoption drift reports and flag telemetry drift without reading raw telemetry logs.
|
||||
|
||||
## Scope Policy
|
||||
|
||||
Atlas keeps a full catalog, but release gates should distinguish actionable library skills from examples and test fixtures.
|
||||
|
||||
Use `skill_atlas/policy.json` to mark path prefixes as non-actionable when they are intentionally retained as examples, evolution snapshots, embedded generated skills, or validator fixtures. Non-actionable items still appear in the full report, route matrix, stale list, and owner gap list, but Review Studio should use the actionable counts for release readiness.
|
||||
|
||||
## Telemetry Link
|
||||
|
||||
Atlas may read each skill's aggregate `reports/adoption_drift_report.json` to surface portfolio drift signals such as no telemetry for production/library/governed skills, missed triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not read or package `reports/telemetry_events.jsonl`; raw telemetry remains local-only evidence owned by the skill.
|
||||
|
||||
Write drift output to `skill_atlas/drift_signals.json`. Non-actionable scopes stay visible in that file and in the HTML report, but only actionable drift signals should affect release readiness.
|
||||
|
||||
## Reviewer Gate
|
||||
|
||||
Use Atlas before promoting a single skill into a shared library. If an actionable route collision, missing owner, stale governed skill, or telemetry drift signal appears, fix the portfolio boundary before adding more local complexity to one skill. Non-actionable issues should stay visible as evidence, not as release blockers.
|
||||
+210
@@ -0,0 +1,210 @@
|
||||
# Skill Engineering Method
|
||||
|
||||
This doctrine defines the default method for turning messy workflow material into a reusable skill without bloating the entrypoint.
|
||||
|
||||
## Core Loop
|
||||
|
||||
1. Decide whether the request should become a skill at all.
|
||||
2. Run a short intent dialogue to capture the real job, outputs, exclusions, and constraints.
|
||||
3. Choose the smallest viable archetype.
|
||||
4. Set one clear capability boundary.
|
||||
5. Write and test the trigger description before expanding the body.
|
||||
6. Apply authoring discipline: name unresolved assumptions, keep scope small, and tie meaningful changes to checks.
|
||||
7. Add an output risk profile for user-facing artifacts.
|
||||
8. Add an artifact design profile for reports, tutorials, viewers, dashboards, screenshots, and visual pages.
|
||||
9. Add only the gates that match the risk.
|
||||
10. Ship the first routeable package, then pick the three highest-value next iteration directions.
|
||||
11. Package and govern the skill only as far as real reuse demands.
|
||||
|
||||
## Phase 1: Qualification
|
||||
|
||||
Promote a request into a skill only when at least one of these is true:
|
||||
|
||||
- the workflow will be reused
|
||||
- the workflow is easy to route incorrectly
|
||||
- deterministic scripts reduce repeated effort
|
||||
- governance or portability matters
|
||||
|
||||
Reject skill creation when the request is only:
|
||||
|
||||
- explanation
|
||||
- summary
|
||||
- translation
|
||||
- brainstorming
|
||||
- documentation without agent execution
|
||||
- a one-off answer with no reuse value
|
||||
|
||||
See [Non-Skill Decision Tree](non-skill-decision-tree.md).
|
||||
|
||||
## Phase 1.5: Authoring Discipline
|
||||
|
||||
Before expanding the package, apply the execution discipline that keeps the work grounded.
|
||||
|
||||
- clarify only the assumptions that change the package design
|
||||
- do not add speculative features, generic configurability, or decorative structure
|
||||
- when editing an existing skill, touch only files that directly serve the requested change
|
||||
- connect each meaningful change to a check: route evidence, sample run, resource-boundary check, governance check, or reviewer note
|
||||
|
||||
See [Authoring Discipline](authoring-discipline.md).
|
||||
|
||||
## Phase 1.6: Problem Diagnosis
|
||||
|
||||
When the user gives a fuzzy pain point instead of a clear skill request, diagnose the likely package shape before asking for structure.
|
||||
|
||||
- infer whether the need is best served by a scaffold, production workflow, library capability, governed asset, or no skill
|
||||
- recommend at most three candidate directions
|
||||
- explain why each candidate fits, where it is limited, and what the first light version should prove
|
||||
- prefer a concrete recommendation over a menu when the intent is clear enough
|
||||
|
||||
This keeps discovery useful for users who can describe the problem but not the skill architecture.
|
||||
|
||||
## Phase 2: Intent Dialogue
|
||||
|
||||
Before deep authoring, ask only the questions that change the package design.
|
||||
|
||||
- open with a human, teacher-like framing rather than a cold field list
|
||||
- let the user answer naturally first; offer a tiny template only as an optional shortcut
|
||||
- what recurring job should the skill own
|
||||
- what real inputs will users hand to it
|
||||
- what outputs must it produce
|
||||
- what near-neighbor requests should stay out of scope
|
||||
- whether the user has reference systems, repos, or products worth learning from
|
||||
- what constraints matter: privacy, naming, portability, governance, or local fit
|
||||
|
||||
See [Intent Dialogue](intent-dialogue.md).
|
||||
|
||||
## Phase 3: Archetype Selection
|
||||
|
||||
Choose the lightest archetype that fits the job.
|
||||
|
||||
- `Scaffold`: exploratory, personal, or short-lived
|
||||
- `Production`: team-reused, quality-sensitive, but still compact
|
||||
- `Library`: broad reuse, visible evidence, portability, and maintenance expectations
|
||||
- `Governed`: organizationally sensitive or operationally critical; lifecycle and review are explicit
|
||||
|
||||
See [Skill Archetypes](skill-archetypes.md).
|
||||
|
||||
## Phase 4: Boundary Design
|
||||
|
||||
Every skill should answer four questions clearly:
|
||||
|
||||
- what recurring job does it own
|
||||
- what outputs does it produce
|
||||
- what near-neighbor requests should not route here
|
||||
- what detail belongs outside `SKILL.md`
|
||||
|
||||
Boundary work comes before polishing prose.
|
||||
|
||||
## Phase 5: Reference Scan
|
||||
|
||||
Run a short benchmark pass before deep authoring.
|
||||
|
||||
- scan `3-5` reference objects at most
|
||||
- prioritize high-star external GitHub and official benchmark sources first
|
||||
- ask for user-supplied references second, but extract only patterns and standards
|
||||
- use local files third, only for fit, privacy, and compatibility calibration
|
||||
- choose from method, structure, execution, portability, and domain patterns
|
||||
- extract only what passes the pattern test: recurrence, generativity, distinctiveness, and boundary clarity
|
||||
- record what not to borrow so the new skill stays light
|
||||
|
||||
See [Reference Scan Strategy](reference-scan.md) and [Pattern Extraction Doctrine](pattern-extraction-doctrine.md).
|
||||
|
||||
## Phase 5.5: Output Risk Profiling
|
||||
|
||||
Before treating the package as usable, predict the likely mistakes in its final user-facing output.
|
||||
|
||||
- tutorial skills should guard against generic headings, vague examples, and missing success checks
|
||||
- report and Markdown skills should guard against weak tables, dense lists, and poor hierarchy
|
||||
- screenshot or visual skills should guard against wrong captures, missing assets, and invented visual evidence
|
||||
- research or citation-heavy skills should guard against footnote clutter and unsupported claims
|
||||
- code or command skills should guard against hidden cwd, input, output, and side-effect assumptions
|
||||
|
||||
Generate `reports/output-risk-profile.md` and expose it to the reviewer before adding more structure.
|
||||
|
||||
See [Output Quality Risk](output-quality-risk.md).
|
||||
|
||||
## Phase 5.6: Artifact Design Profiling
|
||||
|
||||
Before approving generated reports or visual outputs, define how the artifact should read and scan.
|
||||
|
||||
- choose the artifact family: tutorial, report, review viewer, dashboard, visual evidence, slide-like narrative, or code/CLI guide
|
||||
- let the content choose the visual system instead of copying a fixed house style
|
||||
- borrow document discipline from Kami: route by document type, distill content, verify layout-critical assumptions
|
||||
- borrow slide discipline from presentation skills: plan hierarchy, density, rhythm, and quality gates before writing HTML
|
||||
- reject generic headings, noisy citations, weak tables, wrong screenshots, repeated card grids, and decorative visual defaults
|
||||
|
||||
Generate `reports/artifact-design-profile.md` and expose it in the overview and review viewer.
|
||||
|
||||
See [Artifact Design Doctrine](artifact-design-doctrine.md) and [Output Visual Quality](output-visual-quality.md).
|
||||
|
||||
## Phase 5.7: Prompt Quality Profiling
|
||||
|
||||
When a skill depends on prompt behavior, role design, dialogue quality, or output contracts, turn prompt methodology into an evidence profile.
|
||||
|
||||
- identify the explicit need, implicit need, scenario, user level, and success standard
|
||||
- map Role, Task, and Format into skill behavior instead of copying a full meta-prompt into `SKILL.md`
|
||||
- classify the task family and complexity before adding structure
|
||||
- score completeness, clarity, consistency, practicality, and specificity
|
||||
- expose the full reasoning to reviewers while keeping the user-facing flow recommendation-led
|
||||
|
||||
Generate `reports/prompt-quality-profile.md` and expose it in the overview and review viewer.
|
||||
|
||||
See [Prompt Engineering Doctrine](prompt-engineering-doctrine.md).
|
||||
|
||||
## Phase 6: Trigger-First Authoring
|
||||
|
||||
Author the frontmatter `description` before expanding the body.
|
||||
|
||||
- start with the recurring job
|
||||
- include the trigger actions that should route here
|
||||
- include exclusions when confusion is plausible
|
||||
- test the route before growing the file tree
|
||||
|
||||
Trigger quality is improved through:
|
||||
|
||||
- `trigger_eval.py`
|
||||
- `optimize_description.py`
|
||||
- blind holdout
|
||||
- judge-backed blind holdout
|
||||
- adversarial holdout
|
||||
- route confusion
|
||||
|
||||
## Phase 7: Gate Selection
|
||||
|
||||
Add gates by risk, not by habit.
|
||||
|
||||
- low-risk scaffolds: validate structure and context size
|
||||
- production skills: trigger eval plus resource-boundary checks
|
||||
- library skills: description optimization, route confusion, packaging checks
|
||||
- governed skills: governance scoring, lifecycle metadata, regression history
|
||||
|
||||
See [Gate Selection](gate-selection.md).
|
||||
|
||||
## Phase 8: First Iteration Philosophy
|
||||
|
||||
The first package is a routeable baseline, not the final answer.
|
||||
|
||||
- improve trigger and exclusions before growing prose
|
||||
- add one execution asset before adding many documents
|
||||
- surface the three highest-value next moves so authors do not expand in every direction at once
|
||||
- prefer the smallest step that increases reliability more than context cost
|
||||
- move unverifiable ideas into next-step candidates instead of shipping them as baseline structure
|
||||
|
||||
See [Iteration Philosophy](iteration-philosophy.md) and [Authoring Discipline](authoring-discipline.md).
|
||||
|
||||
## Phase 9: Promotion
|
||||
|
||||
A candidate route or package is promotable only when:
|
||||
|
||||
- visible holdout does not regress
|
||||
- blind holdout does not regress
|
||||
- judge-backed blind holdout does not regress
|
||||
- adversarial holdout does not regress
|
||||
- route confusion stays clean
|
||||
- context and governance gates still pass
|
||||
|
||||
See [Promotion Policy](../evals/promotion_policy.md).
|
||||
|
||||
## Design Principle
|
||||
|
||||
The method is only correct if rigor grows faster than context cost. If a new check or document makes the skill heavier without making it more reliable, remove or relocate it.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Skill IR Method
|
||||
|
||||
Skill IR is the 2.0 layer that separates durable skill meaning from platform packaging.
|
||||
|
||||
## Purpose
|
||||
|
||||
Use Skill IR before platform-specific packaging for production, library, governed, or team-distributed skills. The IR should preserve the capability contract even when OpenAI, Claude, Agent Skills, VS Code, or generic adapters differ in folder layout, metadata names, or activation behavior.
|
||||
|
||||
## What Belongs In IR
|
||||
|
||||
- the recurring job the skill owns
|
||||
- the frontmatter trigger description
|
||||
- should-trigger, should-not-trigger, and near-neighbor edge cases
|
||||
- workflow steps, decision points, and failure modes
|
||||
- references, scripts, assets, and reports used by the package
|
||||
- trigger and output eval plans
|
||||
- output risk, execution risk, and trust boundary
|
||||
- owner, maturity, review cadence, and target platforms
|
||||
|
||||
## What Does Not Belong In IR
|
||||
|
||||
- platform-specific file paths that only exist after packaging
|
||||
- copied prose from external benchmarks
|
||||
- client-specific UI labels
|
||||
- local private paths that are not part of the skill package
|
||||
- speculative adapters that are not requested or tested
|
||||
|
||||
## Authoring Rule
|
||||
|
||||
Export or update Skill IR before adding new adapters, compilers, registries, or conformance checks. If a field cannot be derived from local package evidence, leave it empty or use an explicit low-confidence default instead of inventing detail.
|
||||
|
||||
## Reviewer Gate
|
||||
|
||||
A reviewer should be able to answer:
|
||||
|
||||
1. What does this skill own?
|
||||
2. When should it trigger?
|
||||
3. When should it not trigger?
|
||||
4. Which resources and scripts carry the real behavior?
|
||||
5. Which evals prove the contract?
|
||||
6. Which targets can consume the skill without semantic loss?
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
# SkillOps Decision Policy
|
||||
|
||||
Use this policy when turning explicit-source conversation evidence into SkillOps opportunities, proposals, or release work. The goal is to make repeated user signals actionable without letting automation write durable instructions, skills, scripts, or evals without review.
|
||||
|
||||
## Decision Order
|
||||
|
||||
1. Classify the signal as no action, report only, Memory, AGENTS.md, existing Skill patch, candidate Skill, script, eval, report, merge, split, or archive.
|
||||
2. Prefer the smallest durable surface that fixes the repeated friction.
|
||||
3. Require evidence for every write action.
|
||||
4. Require a proposal or approval ledger entry before any source-file write.
|
||||
5. Map every proposed change to at least one verification command.
|
||||
|
||||
## Score Bands
|
||||
|
||||
SkillOps opportunities use a `0-100` score. Scores are advisory and never bypass approval.
|
||||
|
||||
| Score | Decision |
|
||||
| ---: | --- |
|
||||
| `85-100` | Ready for approval review |
|
||||
| `70-84` | Proposal review |
|
||||
| `50-69` | Observe more evidence |
|
||||
| `0-49` | Report only or no action |
|
||||
|
||||
High-risk items stay proposal-only even when their score is high.
|
||||
|
||||
## Action Mapping
|
||||
|
||||
| Pattern | Default Action | Durable Surface |
|
||||
| --- | --- | --- |
|
||||
| `language_default` | Patch existing skill | Report template or artifact doctrine |
|
||||
| `report_ui` | Patch existing skill | Report renderer, artifact doctrine, visual test |
|
||||
| `approval_safety` | AGENTS update | Governance guidance or approval policy |
|
||||
| `delivery_format` | Patch existing skill | CLI output, README, generated summary copy |
|
||||
| `evidence_testing` | Add eval | Focused regression, report-quality, or release gate |
|
||||
| Unknown pattern | Report only | Manual review queue |
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Do not scan private logs implicitly; only use explicit user-supplied sources.
|
||||
- Do not store raw conversation content in reports; use redacted excerpts and aggregate counts.
|
||||
- Do not write source files from a daily report run.
|
||||
- Do not count SkillOps reports as public world-class evidence.
|
||||
- Do not treat planned work, draft submissions, or generated proposals as accepted evidence.
|
||||
|
||||
## Verification
|
||||
|
||||
Every implementation that changes this policy should run:
|
||||
|
||||
```bash
|
||||
python3 tests/verify_skillops_opportunity.py
|
||||
python3 tests/verify_daily_skillops.py
|
||||
python3 tests/verify_yao_cli.py
|
||||
```
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
# Systems Thinking Doctrine
|
||||
|
||||
Use this doctrine when a skill needs to keep producing good behavior after repeated real use.
|
||||
|
||||
## Core Principle
|
||||
|
||||
Structure drives behavior. Improve the system boundary, feedback loops, drift watch, and leverage points before adding more prose, templates, or tools.
|
||||
|
||||
This is inspired by general systems-thinking practice: recurring failures usually come from structure, incentives, feedback, delays, or boundary mistakes rather than from one isolated bad output.
|
||||
|
||||
## Apply Silently By Default
|
||||
|
||||
Use the systems model as author and reviewer evidence. Do not ask the user to choose between system concepts unless there is real uncertainty or a design conflict.
|
||||
|
||||
The user should usually see a recommendation, not a menu of theory.
|
||||
|
||||
## Four Questions
|
||||
|
||||
1. What does this skill own?
|
||||
2. What feedback tells us it is improving or drifting?
|
||||
3. Which failure will appear only after repeated use?
|
||||
4. Where is the smallest change with the largest quality gain?
|
||||
|
||||
## Boundary Map
|
||||
|
||||
Define these before expanding the package:
|
||||
|
||||
- Owned job: the recurring behavior this skill is responsible for.
|
||||
- Input boundary: the real material users will provide.
|
||||
- Output boundary: the concrete hand-back users need.
|
||||
- Non-goals: adjacent requests this skill should refuse or hand off.
|
||||
- Human judgment boundary: places where the model should ask, escalate, or disclose uncertainty.
|
||||
|
||||
## Feedback Loops
|
||||
|
||||
Every serious skill should have at least one loop:
|
||||
|
||||
- Intent loop: user clarification changes the boundary.
|
||||
- Reference loop: benchmark patterns become borrow or avoid guidance.
|
||||
- Output loop: common output failures become self-repair checks.
|
||||
- Reviewer loop: human feedback becomes a gate, reference, or regression case.
|
||||
- Lifecycle loop: reuse level changes maturity tier and governance.
|
||||
|
||||
## Delay And Drift
|
||||
|
||||
Watch for problems that appear after initial success:
|
||||
|
||||
- Trigger drift: the skill starts activating on adjacent work.
|
||||
- Output drift: outputs become generic, cluttered, or misaligned.
|
||||
- Reference drift: borrowed patterns add ceremony without payoff.
|
||||
- Governance drift: team-critical use grows faster than review discipline.
|
||||
|
||||
## Leverage Points
|
||||
|
||||
Prefer changes in this order:
|
||||
|
||||
1. Clarify the real job boundary.
|
||||
2. Tune the frontmatter description.
|
||||
3. Add one output self-repair check.
|
||||
4. Borrow one external pattern as structure, not surface style.
|
||||
5. Close one lifecycle or reviewer feedback loop.
|
||||
|
||||
Do not add more files if a description, boundary, or feedback-loop change would solve the root cause.
|
||||
|
||||
## Reviewer Standard
|
||||
|
||||
A reviewer should ask: will this skill's structure keep producing the desired behavior after repeated use?
|
||||
|
||||
If the answer is unclear, request one of these before approving:
|
||||
|
||||
- a sharper boundary
|
||||
- a named feedback loop
|
||||
- a drift watch
|
||||
- a failure pattern
|
||||
- a highest-leverage next move
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
# Telemetry And Drift Method
|
||||
|
||||
Telemetry turns real use into the next iteration queue. It must stay local-first and metadata-only by default.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use the telemetry drift loop when a skill is production, library, governed, team-distributed, or repeatedly invoked by more than one workflow.
|
||||
|
||||
Do not collect raw prompts, model outputs, transcripts, notes, messages, or private files. If a reviewer needs examples, store anonymized fixtures separately and cite them as eval evidence, not telemetry.
|
||||
|
||||
## Event Contract
|
||||
|
||||
The local event stream is `reports/telemetry_events.jsonl`. It is intentionally narrow:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "skill_activation",
|
||||
"skill": "example-skill",
|
||||
"version": "2.0.0",
|
||||
"source": "yao_cli",
|
||||
"command": "quickstart",
|
||||
"activation_type": "implicit",
|
||||
"outcome": "accepted",
|
||||
"failure_type": "none",
|
||||
"timestamp": "2026-06-13T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Allowed events: `skill_activation`, `skill_output`, `script_run`, `review_event`.
|
||||
|
||||
Allowed sources: `manual`, `yao_cli`, `external`, `unknown`.
|
||||
|
||||
Allowed outcomes: `accepted`, `edited`, `rejected`, `missed`, `failed`, `reviewed`, `unknown`.
|
||||
|
||||
Allowed failure types: `wrong_trigger`, `under_trigger`, `bad_output`, `missing_resource`, `script_error`, `review_overdue`, `none`.
|
||||
|
||||
`source` and `command` are metadata fields. They may identify that `yao.py` ran `quickstart`, `validate`, `output-exec`, or another subcommand, but they must not include arguments, prompt text, file content, model output, transcripts, or reviewer notes.
|
||||
|
||||
## CLI Capture
|
||||
|
||||
`scripts/yao.py` can record metadata-only `script_run` events automatically. It is opt-in to keep release evidence reproducible and avoid surprising local writes:
|
||||
|
||||
```bash
|
||||
YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate .
|
||||
```
|
||||
|
||||
Optional destination override:
|
||||
|
||||
```bash
|
||||
YAO_CLI_TELEMETRY=1 \
|
||||
YAO_CLI_TELEMETRY_EVENTS=/tmp/yao-telemetry.jsonl \
|
||||
python3 scripts/yao.py output-exec
|
||||
```
|
||||
|
||||
Equivalent global flags are available before the subcommand:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py --record-cli-telemetry validate .
|
||||
python3 scripts/yao.py --no-cli-telemetry validate .
|
||||
```
|
||||
|
||||
Successful CLI runs record `event=script_run`, `source=yao_cli`, `outcome=accepted`, and `failure_type=none`. Failed CLI runs record `outcome=failed` and `failure_type=script_error`. The command name is normalized to the subcommand only; command arguments are never recorded.
|
||||
|
||||
## External Client Emit
|
||||
|
||||
External clients, browser extensions, editor adapters, or wrapper scripts can emit one sanitized event at a time into a local spool before importing it into the aggregate drift report:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-emit . \
|
||||
--event skill_activation \
|
||||
--activation-type explicit \
|
||||
--outcome accepted \
|
||||
--command browser-extension
|
||||
```
|
||||
|
||||
By default this writes to `.yao/telemetry_spool/external_events.jsonl`. Use `--output-jsonl` when a client needs a different local handoff path:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-emit . \
|
||||
--output-jsonl /tmp/external-client-events.jsonl \
|
||||
--event skill_output \
|
||||
--activation-type manual \
|
||||
--outcome edited \
|
||||
--command browser-plugin
|
||||
```
|
||||
|
||||
Use `--dry-run` to validate a proposed event without writing to the spool. The emitter uses the same metadata-only contract as import: no prompt, input, output, transcript, message, note, raw text, arguments, or unknown fields are accepted.
|
||||
|
||||
After a client finishes a batch, import the spool:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
|
||||
```
|
||||
|
||||
## Client Hook Recipes
|
||||
|
||||
Use `telemetry-hooks` to generate auditable Browser, Chrome, VS Code, CLI wrapper, and provider-adapter hook recipes:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-hooks .
|
||||
```
|
||||
|
||||
The report is written to:
|
||||
|
||||
- `reports/telemetry_hook_recipes.json`
|
||||
- `reports/telemetry_hook_recipes.md`
|
||||
|
||||
Each recipe includes a dry-run command, an emit command, the target local spool, trigger points, and the privacy contract. The report intentionally sets `native_auto_capture=false`; it proves the local hook contract and metadata-only command shape, not that a host client is already natively integrated.
|
||||
|
||||
## Browser Native Host
|
||||
|
||||
`scripts/telemetry_native_host.py` implements the local side of Browser/Chrome Native Messaging. It accepts length-prefixed JSON messages on stdio, validates them with the same metadata-only telemetry contract, appends accepted events to the local spool, and rejects raw prompt/output/transcript/message/note fields.
|
||||
|
||||
Smoke-test one message without Browser installation:
|
||||
|
||||
```bash
|
||||
python3 scripts/telemetry_native_host.py . \
|
||||
--message-json '{"event":"skill_activation","activation_type":"explicit","outcome":"accepted","failure_type":"none","command":"chrome-native-host"}'
|
||||
```
|
||||
|
||||
Generate a local launcher and Chrome native messaging manifest for an operator-installed extension:
|
||||
|
||||
```bash
|
||||
python3 scripts/telemetry_native_host.py . \
|
||||
--write-launcher /tmp/yao-telemetry-host.sh \
|
||||
--write-manifest /tmp/yao-telemetry-host.json \
|
||||
--allowed-origin chrome-extension://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/
|
||||
```
|
||||
|
||||
This is an executable native host bridge and manifest generator. It still does not prove that a specific Browser/Chrome extension is installed or sending events in the user's environment.
|
||||
|
||||
## External Client Import
|
||||
|
||||
External clients, browser extensions, editor adapters, or wrapper scripts may hand off already-sanitized JSONL through `telemetry-import`:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-import . \
|
||||
--input-jsonl /tmp/external-client-events.jsonl \
|
||||
--command browser-extension
|
||||
```
|
||||
|
||||
The importer defaults missing `source` to `external` and missing `command` to `external-client`. It validates the entire JSONL file before writing anything. If any line includes a raw content field, unsupported source, unsupported outcome, unsupported failure type, unknown field, malformed JSON, or an unsafe command name, the whole import is rejected and the existing local event stream is left untouched.
|
||||
|
||||
Use `--dry-run` to validate an external batch without writing `reports/telemetry_events.jsonl` or refreshing aggregate reports:
|
||||
|
||||
```bash
|
||||
python3 scripts/yao.py telemetry-import . --input-jsonl /tmp/external-client-events.jsonl --dry-run
|
||||
```
|
||||
|
||||
## Privacy Rule
|
||||
|
||||
The raw JSONL event log is local evidence and should not be distributed in skill packages. The distributable artifact is the aggregate report:
|
||||
|
||||
- `reports/adoption_drift_report.json`
|
||||
- `reports/adoption_drift_report.md`
|
||||
|
||||
Package builders should exclude `reports/telemetry_events.jsonl`. The root repository also ignores this raw event stream so local usage evidence does not become ordinary source history by accident.
|
||||
|
||||
## Release Interpretation
|
||||
|
||||
- `no-data`: acceptable for a first scaffold, but a warning for governed release review.
|
||||
- `low`: events exist and no drift failure signal is present.
|
||||
- `medium`: at least one missed trigger, wrong trigger, bad output, script error, or overdue review signal exists.
|
||||
- `high`: several drift signals are present; convert them into eval cases or governance actions before calling the skill release-ready.
|
||||
|
||||
## Iteration Loop
|
||||
|
||||
1. Capture metadata-only events locally, either manually with `adoption-drift --record-event`, automatically with opt-in `yao.py` CLI capture, through `telemetry-emit` client hooks, through generated `telemetry-hooks` client recipes, or through validated external JSONL import.
|
||||
2. Render `reports/adoption_drift_report.md`.
|
||||
3. Convert missed triggers into trigger eval cases.
|
||||
4. Convert bad outputs into Output Eval assertions and failure taxonomy entries.
|
||||
5. Convert script errors into non-interactive smoke tests.
|
||||
6. Feed review-overdue signals back into Skill Atlas and owner review.
|
||||
7. Let Skill Atlas read only `reports/adoption_drift_report.json` and publish portfolio-level `skill_atlas/drift_signals.json`.
|
||||
|
||||
## Review Studio Role
|
||||
|
||||
Review Studio should show the aggregate telemetry gate as an operating loop, not as raw logs. A blocker means the telemetry contract was violated. A warning means the evidence is absent or the drift signal needs a follow-up case.
|
||||
|
||||
## Skill Atlas Role
|
||||
|
||||
Skill Atlas uses aggregate adoption drift reports to rank portfolio work. It should surface no-data warnings for actionable production/library/governed skills, and drift warnings for missed triggers, wrong triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not inspect raw JSONL telemetry or use non-actionable example/fixture signals as release blockers.
|
||||
@@ -0,0 +1,79 @@
|
||||
# Trust Security Method
|
||||
|
||||
Trust checks make skills safer to install and review, especially when they include scripts or are distributed to a team.
|
||||
|
||||
## When To Run
|
||||
|
||||
Run the trust report when:
|
||||
|
||||
- the skill contains scripts
|
||||
- the skill will be shared with a team
|
||||
- the package may be installed from a registry or plugin
|
||||
- the skill reads external files, uses network access, or shells out
|
||||
- the maturity tier is library or governed
|
||||
|
||||
## V0 Checks
|
||||
|
||||
- obvious secret patterns
|
||||
- script help surface and interactive prompts
|
||||
- execution-level `--help` smoke checks
|
||||
- dependency pinning
|
||||
- runtime trust metadata
|
||||
- network-capable scripts
|
||||
- bounded host policy for network-capable scripts
|
||||
- reviewer-approved permission policy for high-permission capabilities
|
||||
- packaged-target runtime permission probes for adapter contracts and metadata fallback limits
|
||||
- source-contract integrity digest
|
||||
|
||||
## Script Interface Rule
|
||||
|
||||
Every Python file under `scripts/` is reviewed as part of the package trust surface.
|
||||
|
||||
- CLI scripts should use `argparse` so reviewers and installers can run `python3 scripts/name.py --help` before execution.
|
||||
- The trust report executes `python3 scripts/name.py --help` for CLI scripts with `argparse`, with a short timeout, and records pass/fail evidence.
|
||||
- Import-only modules should declare `SCRIPT_INTERFACE = "internal-module"` near the top of the file.
|
||||
- Internal modules should also declare `SCRIPT_INTERFACE_REASON` with a short explanation of which CLI or renderer imports them.
|
||||
- The trust report keeps internal modules in the script inventory, but excludes them from CLI help-surface warnings.
|
||||
- A Python file without an explicit internal-module declaration is treated as a CLI script by default.
|
||||
- CLI scripts without `argparse` are not smoke-executed; they remain visible as help-surface warnings.
|
||||
|
||||
## Network Policy Rule
|
||||
|
||||
Network-capable scripts must be bounded by a machine-readable policy before team distribution.
|
||||
|
||||
- Put the policy in `security/network_policy.json`.
|
||||
- Add one entry per network-capable script under `scripts`.
|
||||
- Declare `allowed_hosts`, `allowed_path_prefixes`, purpose, timeout, auth mode, and custom-host behavior.
|
||||
- Default to HTTPS-only and deny custom hosts unless a CLI flag or environment variable makes the override explicit.
|
||||
- The trust report compares HTTPS URL literals in each script with `allowed_hosts`; missing or mismatched entries remain reviewer-visible warnings.
|
||||
|
||||
## Permission Approval Rule
|
||||
|
||||
High-permission capabilities must be approved before governed release.
|
||||
|
||||
- Put approvals in `security/permission_policy.json`.
|
||||
- Cover each required capability detected by the trust report: `network`, `file_write`, `subprocess`, and `interactive` when present.
|
||||
- Each approval must include `decision: approved`, `reviewer`, `scope`, `reason`, `expires_at`, `evidence`, and `target_enforcement`.
|
||||
- Review Studio surfaces these checks as the `permission-gates` gate.
|
||||
- Missing, invalid, or expired approvals block governed mode. They remain visible warnings in lighter modes.
|
||||
|
||||
## Runtime Permission Probe Rule
|
||||
|
||||
Permission approval validates reviewer intent. Runtime permission probes validate the generated target adapters after packaging.
|
||||
|
||||
- Run `python3 scripts/probe_runtime_permissions.py . --package-dir dist` after `cross_packager.py`.
|
||||
- The probe writes `reports/runtime_permission_probes.json` and `reports/runtime_permission_probes.md`.
|
||||
- A passing probe requires every target adapter to carry `permission_contract`, `target_permission_contract`, declared capabilities, a native-enforcement boolean, representation notes, and operator notes.
|
||||
- When `reports/install_simulation.json` matches the same package directory, the probe also reports installer enforcement counts from the install simulation. This proves the local package installer gate is wired, but it does not count as target-client native enforcement.
|
||||
- If a target has no native enforcement, the probe must mark an explicit metadata fallback and keep residual risk reviewer-visible.
|
||||
- Review Studio surfaces this as the `permission-runtime` gate.
|
||||
|
||||
## Release Rule
|
||||
|
||||
High-risk secrets or unrestricted remote inline execution block governed release. Warnings are reviewer-visible but do not block v0 unless the release owner decides the target environment requires stricter policy.
|
||||
|
||||
## Hash Scope
|
||||
|
||||
`package_sha256` is a stable source-contract digest, not a generated archive digest. It covers the skill entrypoint, metadata, scripts, references, evals, runtime, templates, security notes, Skill IR, and root control files. It deliberately excludes generated `reports/`, packaged `dist/` archives, and raw local telemetry so a report render or local adoption log cannot mutate the trust fingerprint.
|
||||
|
||||
Use the package verification or registry audit report for the distributable archive checksum.
|
||||
@@ -0,0 +1,35 @@
|
||||
# User Memory Policy
|
||||
|
||||
This skill treats user preference memory as local, explicit, and reviewable.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Explicit source only**: adaptive scans require a user-provided file path.
|
||||
- **Local first**: no network access is needed for preference extraction.
|
||||
- **No implicit private logs**: shell history, browser history, mail, and hidden chat logs are blocked by default.
|
||||
- **Repeated signals only**: one-off statements are recorded as discarded signals unless they meet the configured support threshold.
|
||||
- **Redacted evidence**: stored excerpts must remove secrets, tokens, email addresses, and local absolute paths.
|
||||
- **Proposal before patch**: preference memory can generate proposals, not automatic source edits.
|
||||
|
||||
## Allowed Inputs
|
||||
|
||||
Recommended inputs are curated JSONL, Markdown, or text files prepared for review. JSONL records should use a field such as `text`, `message`, `content`, `excerpt`, `prompt`, `note`, or `body`.
|
||||
|
||||
## Blocked By Default
|
||||
|
||||
The scanner refuses common shell history files such as `.zsh_history`, `.bash_history`, and `.fish_history` unless an explicit override is added for a controlled test. Even with an override, the output remains redacted and proposal-only.
|
||||
|
||||
## Retention
|
||||
|
||||
Generated reports store only summarized patterns and short redacted excerpts. They should not be treated as a full transcript, chat archive, or durable personal memory store.
|
||||
|
||||
## Upgrade Path
|
||||
|
||||
A future patch-application stage must add:
|
||||
|
||||
- human approval ledger;
|
||||
- allowlisted target files;
|
||||
- dry-run diffs;
|
||||
- regression command execution;
|
||||
- rollback artifacts;
|
||||
- reviewer-visible audit trail.
|
||||
Reference in New Issue
Block a user