📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,206 @@
|
||||
---
|
||||
name: session-analysis
|
||||
description: This skill should be used when analyzing conversation patterns, identifying frustration or success signals, or when "analyze conversation", "what went wrong", or "patterns" are mentioned.
|
||||
metadata:
|
||||
version: "2.0.0"
|
||||
---
|
||||
|
||||
# Conversation Analysis
|
||||
|
||||
Signal extraction → pattern detection → behavioral insights.
|
||||
|
||||
<when_to_use>
|
||||
|
||||
- User requests conversation analysis
|
||||
- Identifying frustration, success, or workflow patterns
|
||||
- Extracting user preferences and requirements
|
||||
- Understanding task evolution and iterations
|
||||
|
||||
NOT for: real-time monitoring, content generation, single message analysis
|
||||
|
||||
</when_to_use>
|
||||
|
||||
<signal_taxonomy>
|
||||
|
||||
| Type | Subtype | Indicators |
|
||||
|------|---------|------------|
|
||||
| Success | Explicit Praise | "Perfect!", "Exactly what I needed", exclamation marks |
|
||||
| Success | Continuation | "Now do the same for...", building on prior work |
|
||||
| Success | Adoption | User implements suggestion without modification |
|
||||
| Success | Acceptance | "Looks good", "Ship it", "Merge this" |
|
||||
| Frustration | Correction | "No, I meant...", "That's wrong", "Do X instead" |
|
||||
| Frustration | Reversion | User undoes agent changes, "Go back" |
|
||||
| Frustration | Repetition | Same request 2+ times, escalating specificity |
|
||||
| Frustration | Explicit | "This isn't working", "Why did you...", accusatory tone |
|
||||
| Workflow | Sequence | "First...", "Then...", "Finally...", numbered lists |
|
||||
| Workflow | Transition | "Now that X is done, let's Y", stage changes |
|
||||
| Workflow | Tool Chain | Recurring tool usage patterns (Read → Edit → Bash) |
|
||||
| Workflow | Context Switch | Abrupt topic changes, no transition language |
|
||||
| Request | Prohibition | "Don't use X", "Never do Y", "Avoid Z" |
|
||||
| Request | Requirement | "Always check...", "Make sure to...", "You must..." |
|
||||
| Request | Preference | "I prefer...", "It's better to...", comparative language |
|
||||
| Request | Conditional | "If X then Y", "When A, do B", situational rules |
|
||||
|
||||
Confidence levels:
|
||||
- High (0.8–1.0): Explicit keywords match taxonomy, no ambiguity, strong context
|
||||
- Medium (0.5–0.79): Implicit signal, partial context, minor ambiguity
|
||||
- Low (0.2–0.49): Ambiguous language, weak context, borderline classification
|
||||
|
||||
</signal_taxonomy>
|
||||
|
||||
<stages>
|
||||
|
||||
Load the **maintain-tasks** skill for stage tracking. Stages advance only, never regress.
|
||||
|
||||
| Stage | Trigger | activeForm |
|
||||
|-------|---------|------------|
|
||||
| Parse Input | Session start | "Parsing input" |
|
||||
| Extract Signals | Scope validated | "Extracting signals" |
|
||||
| Detect Patterns | Signals extracted | "Detecting patterns" |
|
||||
| Synthesize Report | Patterns detected | "Synthesizing report" |
|
||||
|
||||
Task format:
|
||||
|
||||
```text
|
||||
- Parse Input { scope description }
|
||||
- Extract Signals { from N messages }
|
||||
- Detect Patterns { category focus }
|
||||
- Synthesize Report { output format }
|
||||
```
|
||||
|
||||
Edge cases:
|
||||
- Small scope (<5 messages): Skip Extract Signals, jump to Synthesize
|
||||
- Re-analysis: Resume at Detect Patterns
|
||||
- Narrow focus (single signal type): Skip Detect Patterns
|
||||
|
||||
Workflow:
|
||||
- Start: Create Parse Input `in_progress`
|
||||
- Transition: Mark current `completed`, add next `in_progress`
|
||||
- After delivery: Mark Synthesize Report `completed`
|
||||
|
||||
</stages>
|
||||
|
||||
<workflow>
|
||||
|
||||
1. Define Scope
|
||||
- Message range (all, recent N, date range)
|
||||
- Actors (user only, agent only, both)
|
||||
- Exclusions (system messages, tool outputs, code blocks)
|
||||
- Mark Parse Input `completed`, create Extract Signals `in_progress`
|
||||
|
||||
2. Extract Signals
|
||||
- Scan messages for signal keywords
|
||||
- Match against taxonomy
|
||||
- Assign confidence (high/medium/low)
|
||||
- Record: type, subtype, message_id, timestamp, quote, context
|
||||
- Mark Extract Signals `completed`, create Detect Patterns `in_progress`
|
||||
|
||||
3. Detect Patterns
|
||||
- Group signals by type/subtype
|
||||
- Find clusters (3+ related signals)
|
||||
- Identify evolution (signal changes over time)
|
||||
- Track repetition (recurring themes)
|
||||
- Spot correlations (tool chains, workflows)
|
||||
- Mark Detect Patterns `completed`, create Synthesize Report `in_progress`
|
||||
|
||||
4. Output
|
||||
- Generate JSON with signals, patterns, summary
|
||||
- Include confidence, recommendations, action items
|
||||
- Append `△ Caveats` if gaps exist
|
||||
- Mark Synthesize Report `completed`
|
||||
|
||||
</workflow>
|
||||
|
||||
<pattern_detection>
|
||||
|
||||
Behavioral patterns from signal clusters:
|
||||
|
||||
| Pattern | Detection | Confidence |
|
||||
|---------|-----------|------------|
|
||||
| Repetition | Same signal 3+ times | Strong: 5+ signals |
|
||||
| Evolution | Signal type changes over time | Moderate: 3-4 signals |
|
||||
| Preferences | Consistent request signals | Strong: across sessions |
|
||||
| Tool Chains | Recurring tool sequences (5+ times) | High: frequent use |
|
||||
| Problem Areas | Clustered frustration signals | Strong: 3+ in same topic |
|
||||
|
||||
Temporal patterns:
|
||||
- Escalation: Increasing frustration/stronger requirements
|
||||
- De-escalation: Frustration → success transition
|
||||
- Cyclical: Same issue recurs across sessions
|
||||
|
||||
</pattern_detection>
|
||||
|
||||
<output_format>
|
||||
|
||||
JSON structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"analysis": {
|
||||
"scope": {
|
||||
"message_count": N,
|
||||
"date_range": "YYYY-MM-DD to YYYY-MM-DD",
|
||||
"actors": ["user", "agent"]
|
||||
},
|
||||
"signals": [
|
||||
{
|
||||
"type": "success|frustration|workflow|request",
|
||||
"subtype": "specific_subtype",
|
||||
"message_id": "msg_123",
|
||||
"timestamp": "ISO8601",
|
||||
"quote": "exact text",
|
||||
"confidence": "high|medium|low",
|
||||
"context": "brief explanation"
|
||||
}
|
||||
],
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "repetition|evolution|preference|tool_chain",
|
||||
"category": "success|frustration|workflow|request",
|
||||
"description": "pattern summary",
|
||||
"occurrences": N,
|
||||
"confidence": "strong|moderate|weak",
|
||||
"first_seen": "ISO8601",
|
||||
"last_seen": "ISO8601",
|
||||
"recommendation": "actionable next step"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total_signals": N,
|
||||
"by_type": { "success": N, "frustration": N, ... },
|
||||
"key_insights": ["insight 1", "insight 2"],
|
||||
"action_items": ["item 1", "item 2"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</output_format>
|
||||
|
||||
<rules>
|
||||
|
||||
ALWAYS:
|
||||
- Create Parse Input at session start
|
||||
- Update todos at stage transitions
|
||||
- Include confidence levels for all signals
|
||||
- Support patterns with 2+ signals minimum
|
||||
- Mark Synthesize Report `completed` after delivery
|
||||
- Apply recency weighting (recent overrides old)
|
||||
|
||||
NEVER:
|
||||
- Skip stage transitions
|
||||
- Extract low-confidence signals without marking them
|
||||
- Claim patterns from single occurrences
|
||||
- Regress stages
|
||||
- Deliver without marking final stage complete
|
||||
- Over-interpret neutral language
|
||||
|
||||
</rules>
|
||||
|
||||
<references>
|
||||
|
||||
- [signal-patterns.md](references/signal-patterns.md) — extended taxonomy, edge cases
|
||||
- [extraction-techniques.md](references/extraction-techniques.md) — regex, heuristics
|
||||
- [sample-analysis.md](examples/sample-analysis.md) — complete walkthrough
|
||||
|
||||
</references>
|
||||
+633
@@ -0,0 +1,633 @@
|
||||
# Sample Conversation Analysis
|
||||
|
||||
A complete worked example showing the conversation analysis process from raw messages to structured output.
|
||||
|
||||
## Sample Conversation Excerpt
|
||||
|
||||
This conversation shows a user working with an AI agent to implement authentication for a web application.
|
||||
|
||||
---
|
||||
|
||||
**Message 1** (User, 2025-11-28 09:00:00)
|
||||
|
||||
```
|
||||
I need to add authentication to the app. Use JWT tokens and make sure to hash passwords with bcrypt.
|
||||
```
|
||||
|
||||
**Message 2** (Agent, 2025-11-28 09:01:30)
|
||||
|
||||
```
|
||||
I'll implement JWT authentication with bcrypt password hashing. Let me create the auth middleware and user model.
|
||||
[Agent implements using npm and plain JavaScript]
|
||||
```
|
||||
|
||||
**Message 3** (User, 2025-11-28 09:05:00)
|
||||
|
||||
```
|
||||
No, use Bun not npm. And I prefer TypeScript over JavaScript for all new code.
|
||||
```
|
||||
|
||||
**Message 4** (Agent, 2025-11-28 09:06:00)
|
||||
|
||||
```
|
||||
I'll switch to Bun and TypeScript. Let me recreate the implementation.
|
||||
[Agent reimplements with Bun and TypeScript]
|
||||
```
|
||||
|
||||
**Message 5** (User, 2025-11-28 09:10:00)
|
||||
|
||||
```
|
||||
Good, but you forgot to add input validation. Always validate user input before processing.
|
||||
```
|
||||
|
||||
**Message 6** (Agent, 2025-11-28 09:11:00)
|
||||
|
||||
```
|
||||
Adding input validation for email and password fields.
|
||||
[Agent adds validation]
|
||||
```
|
||||
|
||||
**Message 7** (User, 2025-11-28 09:15:00)
|
||||
|
||||
```
|
||||
Perfect! Now apply this same pattern to the password reset endpoint.
|
||||
```
|
||||
|
||||
**Message 8** (Agent, 2025-11-28 09:16:00)
|
||||
|
||||
```
|
||||
I'll implement password reset using the same auth pattern with JWT and validation.
|
||||
[Agent implements password reset]
|
||||
```
|
||||
|
||||
**Message 9** (User, 2025-11-28 09:20:00)
|
||||
|
||||
```
|
||||
Excellent work. This is exactly what I needed. Let's move on to the API rate limiting next.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Analysis
|
||||
|
||||
### Step 1: Scope Definition
|
||||
|
||||
**Scope parameters**:
|
||||
- Message range: Messages 1-9 (complete excerpt)
|
||||
- Actors: User and Agent (both included)
|
||||
- Exclusions: Agent's implementation details (code blocks) excluded from signal extraction
|
||||
- Time range: 2025-11-28 09:00:00 to 09:20:00 (20-minute session)
|
||||
|
||||
**Scope summary**: Analyzing 5 user messages over a 20-minute authentication implementation session.
|
||||
|
||||
### Step 2: Signal Extraction
|
||||
|
||||
#### Message 1 Analysis
|
||||
|
||||
**Text**: "I need to add authentication to the app. Use JWT tokens and make sure to hash passwords with bcrypt."
|
||||
|
||||
**Signals detected**:
|
||||
|
||||
1. **Request (requirement)**
|
||||
- Quote: "Use JWT tokens"
|
||||
- Confidence: High (imperative language)
|
||||
- Context: Initial requirement setting
|
||||
|
||||
2. **Request (requirement)**
|
||||
- Quote: "make sure to hash passwords with bcrypt"
|
||||
- Confidence: High ("make sure" = requirement language)
|
||||
- Context: Security requirement
|
||||
|
||||
**No frustration**: First message, no prior context to contradict.
|
||||
|
||||
#### Message 3 Analysis
|
||||
|
||||
**Text**: "No, use Bun not npm. And I prefer TypeScript over JavaScript for all new code."
|
||||
|
||||
**Signals detected**:
|
||||
|
||||
1. **Frustration (correction)**
|
||||
- Quote: "No, use Bun not npm"
|
||||
- Confidence: High (explicit negation "No")
|
||||
- Context: Agent used npm, user correcting to Bun
|
||||
|
||||
2. **Request (preference)**
|
||||
- Quote: "I prefer TypeScript over JavaScript for all new code"
|
||||
- Confidence: High (explicit "prefer" language)
|
||||
- Context: Agent used JavaScript, user stating preference
|
||||
|
||||
**Pattern note**: Two corrections in one message suggest agent didn't check project preferences.
|
||||
|
||||
#### Message 5 Analysis
|
||||
|
||||
**Text**: "Good, but you forgot to add input validation. Always validate user input before processing."
|
||||
|
||||
**Signals detected**:
|
||||
|
||||
1. **Success (weak praise)**
|
||||
- Quote: "Good"
|
||||
- Confidence: Low (weak praise, qualified by "but")
|
||||
- Context: Acknowledging partial success
|
||||
|
||||
2. **Frustration (correction)**
|
||||
- Quote: "you forgot to add input validation"
|
||||
- Confidence: Medium (pointing out omission)
|
||||
- Context: Expected validation wasn't included
|
||||
|
||||
3. **Request (requirement)**
|
||||
- Quote: "Always validate user input before processing"
|
||||
- Confidence: High ("Always" = absolute requirement)
|
||||
- Context: Establishing security requirement
|
||||
|
||||
**Pattern note**: "Good, but..." pattern indicates mixed success—implementation direction correct but incomplete.
|
||||
|
||||
#### Message 7 Analysis
|
||||
|
||||
**Text**: "Perfect! Now apply this same pattern to the password reset endpoint."
|
||||
|
||||
**Signals detected**:
|
||||
|
||||
1. **Success (explicit praise)**
|
||||
- Quote: "Perfect!"
|
||||
- Confidence: High (superlative + exclamation)
|
||||
- Context: Validation added, implementation now complete
|
||||
|
||||
2. **Success (continuation)**
|
||||
- Quote: "Now apply this same pattern to the password reset endpoint"
|
||||
- Confidence: High (explicit extension request without corrections)
|
||||
- Context: User wants to replicate successful pattern
|
||||
|
||||
**Pattern note**: Transition from frustration (Message 5) to success (Message 7) shows agent resolved issues.
|
||||
|
||||
#### Message 9 Analysis
|
||||
|
||||
**Text**: "Excellent work. This is exactly what I needed. Let's move on to the API rate limiting next."
|
||||
|
||||
**Signals detected**:
|
||||
|
||||
1. **Success (explicit praise)**
|
||||
- Quote: "Excellent work"
|
||||
- Confidence: High (superlative praise)
|
||||
- Context: Password reset implementation successful
|
||||
|
||||
2. **Success (fulfillment)**
|
||||
- Quote: "This is exactly what I needed"
|
||||
- Confidence: High (explicit satisfaction language)
|
||||
- Context: Complete feature meets requirements
|
||||
|
||||
3. **Workflow (stage transition)**
|
||||
- Quote: "Let's move on to the API rate limiting next"
|
||||
- Confidence: High (completion + new direction)
|
||||
- Context: Auth complete, moving to new feature
|
||||
|
||||
**Pattern note**: Strong success signal followed by stage transition indicates task completion and satisfaction.
|
||||
|
||||
### Step 3: Signal Classification
|
||||
|
||||
**Grouping by type**:
|
||||
|
||||
**Success signals** (5 total):
|
||||
- Message 5: "Good" (weak praise, low confidence)
|
||||
- Message 7: "Perfect!" (explicit praise, high confidence)
|
||||
- Message 7: "apply this same pattern" (continuation, high confidence)
|
||||
- Message 9: "Excellent work" (explicit praise, high confidence)
|
||||
- Message 9: "This is exactly what I needed" (fulfillment, high confidence)
|
||||
|
||||
**Frustration signals** (3 total):
|
||||
- Message 3: "No, use Bun not npm" (correction, high confidence)
|
||||
- Message 3: "I prefer TypeScript over JavaScript" (correction/preference, high confidence)
|
||||
- Message 5: "you forgot to add input validation" (correction, medium confidence)
|
||||
|
||||
**Request signals** (4 total):
|
||||
- Message 1: "Use JWT tokens" (requirement, high confidence)
|
||||
- Message 1: "make sure to hash passwords with bcrypt" (requirement, high confidence)
|
||||
- Message 3: "I prefer TypeScript over JavaScript for all new code" (preference, high confidence)
|
||||
- Message 5: "Always validate user input before processing" (requirement, high confidence)
|
||||
|
||||
**Workflow signals** (1 total):
|
||||
- Message 9: "Let's move on to the API rate limiting next" (stage transition, high confidence)
|
||||
|
||||
### Step 4: Pattern Detection
|
||||
|
||||
#### Pattern 1: Initial Misconfiguration
|
||||
|
||||
**Type**: Frustration cluster (early)
|
||||
|
||||
**Description**: Agent didn't check project preferences before starting, leading to immediate corrections.
|
||||
|
||||
**Evidence**:
|
||||
- Message 3: Two corrections (Bun, TypeScript)
|
||||
- Timing: Within 5 minutes of starting
|
||||
- Resolution: Agent corrected both issues
|
||||
|
||||
**Confidence**: High (2 explicit corrections in single message)
|
||||
|
||||
**Recommendation**: Agent should check project preferences (package manager, language) before implementing.
|
||||
|
||||
#### Pattern 2: Security Requirements
|
||||
|
||||
**Type**: Request pattern (requirements)
|
||||
|
||||
**Description**: User emphasized security best practices throughout conversation.
|
||||
|
||||
**Evidence**:
|
||||
- Message 1: "hash passwords with bcrypt" (cryptographic requirement)
|
||||
- Message 5: "Always validate user input" (input validation requirement)
|
||||
- Consistency: Both messages establish security requirements
|
||||
|
||||
**Confidence**: Strong (2+ consistent security-focused requirements)
|
||||
|
||||
**Recommendation**: Internalize security-first approach for auth features. Always include: password hashing, input validation, secure token handling.
|
||||
|
||||
#### Pattern 3: Technology Preferences
|
||||
|
||||
**Type**: Request pattern (preferences)
|
||||
|
||||
**Description**: User has strong preferences for specific technologies.
|
||||
|
||||
**Evidence**:
|
||||
- Message 3: "use Bun not npm" (package manager preference)
|
||||
- Message 3: "I prefer TypeScript over JavaScript for all new code" (language preference)
|
||||
- Explicitness: User stated preferences clearly when agent used wrong tools
|
||||
|
||||
**Confidence**: Strong (explicit preference statements)
|
||||
|
||||
**Recommendation**: Add to project memory: "Use Bun for package management" and "Use TypeScript for all new code".
|
||||
|
||||
#### Pattern 4: Success After Correction
|
||||
|
||||
**Type**: Evolution (frustration → success)
|
||||
|
||||
**Description**: Agent successfully resolved issues and delivered satisfactory implementation.
|
||||
|
||||
**Evidence**:
|
||||
- Messages 3, 5: Corrections/frustration
|
||||
- Messages 7, 9: Explicit praise and satisfaction
|
||||
- Progression: Agent incorporated feedback and improved
|
||||
|
||||
**Confidence**: Moderate (clear evolution but single session)
|
||||
|
||||
**Recommendation**: The feedback loop worked. Agent responded well to corrections.
|
||||
|
||||
#### Pattern 5: Pattern Replication Request
|
||||
|
||||
**Type**: Success (continuation)
|
||||
|
||||
**Description**: User wants successful patterns applied to similar features.
|
||||
|
||||
**Evidence**:
|
||||
- Message 7: "apply this same pattern to the password reset endpoint"
|
||||
- Context: After successful auth implementation with JWT, TypeScript, validation
|
||||
- Pattern components: JWT tokens, TypeScript, input validation, bcrypt hashing
|
||||
|
||||
**Confidence**: Strong (explicit continuation request)
|
||||
|
||||
**Recommendation**: Create reusable auth pattern template with these components for future auth endpoints.
|
||||
|
||||
### Step 5: Structured Output
|
||||
|
||||
See JSON output below.
|
||||
|
||||
## Final JSON Output
|
||||
|
||||
```json
|
||||
{
|
||||
"analysis": {
|
||||
"scope": {
|
||||
"message_count": 5,
|
||||
"date_range": "2025-11-28 09:00:00 to 2025-11-28 09:20:00",
|
||||
"duration_minutes": 20,
|
||||
"actors": ["user", "agent"],
|
||||
"exclusions": ["code blocks", "agent implementation details"]
|
||||
},
|
||||
"signals": [
|
||||
{
|
||||
"type": "request",
|
||||
"subtype": "requirement",
|
||||
"message_id": "msg_001",
|
||||
"timestamp": "2025-11-28T09:00:00Z",
|
||||
"quote": "Use JWT tokens",
|
||||
"confidence": "high",
|
||||
"context": "Initial authentication requirement"
|
||||
},
|
||||
{
|
||||
"type": "request",
|
||||
"subtype": "requirement",
|
||||
"message_id": "msg_001",
|
||||
"timestamp": "2025-11-28T09:00:00Z",
|
||||
"quote": "make sure to hash passwords with bcrypt",
|
||||
"confidence": "high",
|
||||
"context": "Security requirement for password storage"
|
||||
},
|
||||
{
|
||||
"type": "frustration",
|
||||
"subtype": "correction",
|
||||
"message_id": "msg_003",
|
||||
"timestamp": "2025-11-28T09:05:00Z",
|
||||
"quote": "No, use Bun not npm",
|
||||
"confidence": "high",
|
||||
"context": "Agent used npm, user correcting to Bun"
|
||||
},
|
||||
{
|
||||
"type": "request",
|
||||
"subtype": "preference",
|
||||
"message_id": "msg_003",
|
||||
"timestamp": "2025-11-28T09:05:00Z",
|
||||
"quote": "I prefer TypeScript over JavaScript for all new code",
|
||||
"confidence": "high",
|
||||
"context": "Language preference correction"
|
||||
},
|
||||
{
|
||||
"type": "success",
|
||||
"subtype": "weak_praise",
|
||||
"message_id": "msg_005",
|
||||
"timestamp": "2025-11-28T09:10:00Z",
|
||||
"quote": "Good",
|
||||
"confidence": "low",
|
||||
"context": "Qualified approval (Good, but...)"
|
||||
},
|
||||
{
|
||||
"type": "frustration",
|
||||
"subtype": "correction",
|
||||
"message_id": "msg_005",
|
||||
"timestamp": "2025-11-28T09:10:00Z",
|
||||
"quote": "you forgot to add input validation",
|
||||
"confidence": "medium",
|
||||
"context": "Missing security feature"
|
||||
},
|
||||
{
|
||||
"type": "request",
|
||||
"subtype": "requirement",
|
||||
"message_id": "msg_005",
|
||||
"timestamp": "2025-11-28T09:10:00Z",
|
||||
"quote": "Always validate user input before processing",
|
||||
"confidence": "high",
|
||||
"context": "Absolute security requirement"
|
||||
},
|
||||
{
|
||||
"type": "success",
|
||||
"subtype": "explicit_praise",
|
||||
"message_id": "msg_007",
|
||||
"timestamp": "2025-11-28T09:15:00Z",
|
||||
"quote": "Perfect!",
|
||||
"confidence": "high",
|
||||
"context": "Implementation now complete with validation"
|
||||
},
|
||||
{
|
||||
"type": "success",
|
||||
"subtype": "continuation",
|
||||
"message_id": "msg_007",
|
||||
"timestamp": "2025-11-28T09:15:00Z",
|
||||
"quote": "Now apply this same pattern to the password reset endpoint",
|
||||
"confidence": "high",
|
||||
"context": "Extension request to replicate successful pattern"
|
||||
},
|
||||
{
|
||||
"type": "success",
|
||||
"subtype": "explicit_praise",
|
||||
"message_id": "msg_009",
|
||||
"timestamp": "2025-11-28T09:20:00Z",
|
||||
"quote": "Excellent work",
|
||||
"confidence": "high",
|
||||
"context": "Password reset implementation successful"
|
||||
},
|
||||
{
|
||||
"type": "success",
|
||||
"subtype": "fulfillment",
|
||||
"message_id": "msg_009",
|
||||
"timestamp": "2025-11-28T09:20:00Z",
|
||||
"quote": "This is exactly what I needed",
|
||||
"confidence": "high",
|
||||
"context": "Complete satisfaction with auth implementation"
|
||||
},
|
||||
{
|
||||
"type": "workflow",
|
||||
"subtype": "stage_transition",
|
||||
"message_id": "msg_009",
|
||||
"timestamp": "2025-11-28T09:20:00Z",
|
||||
"quote": "Let's move on to the API rate limiting next",
|
||||
"confidence": "high",
|
||||
"context": "Auth complete, transitioning to new feature"
|
||||
}
|
||||
],
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "frustration_cluster",
|
||||
"category": "initial_misconfiguration",
|
||||
"description": "Agent didn't check project preferences before starting, leading to immediate corrections for package manager (Bun) and language (TypeScript)",
|
||||
"occurrences": 2,
|
||||
"confidence": "strong",
|
||||
"first_seen": "2025-11-28T09:05:00Z",
|
||||
"last_seen": "2025-11-28T09:05:00Z",
|
||||
"affected_messages": ["msg_003"],
|
||||
"recommendation": "Always check project configuration before implementing. Specifically: check package.json for package manager, check for TypeScript config to determine language choice."
|
||||
},
|
||||
{
|
||||
"pattern_type": "requirement_pattern",
|
||||
"category": "security_requirements",
|
||||
"description": "User consistently emphasized security best practices: password hashing, input validation, secure token handling",
|
||||
"occurrences": 3,
|
||||
"confidence": "strong",
|
||||
"first_seen": "2025-11-28T09:00:00Z",
|
||||
"last_seen": "2025-11-28T09:10:00Z",
|
||||
"affected_messages": ["msg_001", "msg_005"],
|
||||
"recommendation": "Internalize security-first approach for auth features. Standard auth checklist: (1) bcrypt password hashing, (2) input validation on all user inputs, (3) secure JWT token handling, (4) HTTPS enforcement"
|
||||
},
|
||||
{
|
||||
"pattern_type": "preference_pattern",
|
||||
"category": "technology_stack",
|
||||
"description": "User has strong preferences: Bun for package management, TypeScript for all new code",
|
||||
"occurrences": 2,
|
||||
"confidence": "strong",
|
||||
"first_seen": "2025-11-28T09:05:00Z",
|
||||
"last_seen": "2025-11-28T09:05:00Z",
|
||||
"affected_messages": ["msg_003"],
|
||||
"recommendation": "Add to project memory: 'Package manager: Bun (never use npm)' and 'Language: TypeScript for all new code (never use plain JavaScript)'"
|
||||
},
|
||||
{
|
||||
"pattern_type": "evolution",
|
||||
"category": "frustration_to_success",
|
||||
"description": "Agent successfully resolved early issues and delivered satisfactory implementation after corrections",
|
||||
"occurrences": 4,
|
||||
"confidence": "moderate",
|
||||
"first_seen": "2025-11-28T09:05:00Z",
|
||||
"last_seen": "2025-11-28T09:20:00Z",
|
||||
"affected_messages": ["msg_003", "msg_005", "msg_007", "msg_009"],
|
||||
"recommendation": "Positive feedback loop. Agent incorporated user corrections and improved. Continue this pattern of responsive iteration."
|
||||
},
|
||||
{
|
||||
"pattern_type": "continuation",
|
||||
"category": "pattern_replication",
|
||||
"description": "User requested successful auth pattern be applied to related endpoints (password reset)",
|
||||
"occurrences": 1,
|
||||
"confidence": "moderate",
|
||||
"first_seen": "2025-11-28T09:15:00Z",
|
||||
"last_seen": "2025-11-28T09:15:00Z",
|
||||
"affected_messages": ["msg_007"],
|
||||
"recommendation": "Create reusable auth pattern template: JWT authentication + TypeScript + input validation + bcrypt hashing. Apply this template to all auth-related endpoints."
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total_signals": 12,
|
||||
"by_type": {
|
||||
"success": 5,
|
||||
"frustration": 3,
|
||||
"request": 4,
|
||||
"workflow": 1
|
||||
},
|
||||
"by_confidence": {
|
||||
"high": 10,
|
||||
"medium": 1,
|
||||
"low": 1
|
||||
},
|
||||
"sentiment_trend": "negative_to_positive",
|
||||
"key_insights": [
|
||||
"Agent initially missed project preferences (Bun, TypeScript) leading to early corrections",
|
||||
"User prioritizes security: always hash passwords, validate input, use secure tokens",
|
||||
"Technology preferences: Bun (not npm), TypeScript (not JavaScript)",
|
||||
"Agent successfully incorporated feedback and delivered satisfactory solution",
|
||||
"Final implementation met all requirements and user requested pattern replication"
|
||||
],
|
||||
"action_items": [
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "memory_update",
|
||||
"action": "Add to project memory: 'Package manager: Bun (never npm)'"
|
||||
},
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "memory_update",
|
||||
"action": "Add to project memory: 'Language: TypeScript for all new code'"
|
||||
},
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "security_template",
|
||||
"action": "Create auth pattern template: JWT + TypeScript + validation + bcrypt"
|
||||
},
|
||||
{
|
||||
"priority": "medium",
|
||||
"category": "workflow_improvement",
|
||||
"action": "Add pre-implementation checklist: (1) Check package manager, (2) Check language preference, (3) Review security requirements"
|
||||
},
|
||||
{
|
||||
"priority": "low",
|
||||
"category": "documentation",
|
||||
"action": "Document successful auth pattern for future reference"
|
||||
}
|
||||
],
|
||||
"success_rate": 0.625,
|
||||
"explanation": "5 success signals / 8 total sentiment signals (excluding workflow) = 62.5% success rate"
|
||||
},
|
||||
"metadata": {
|
||||
"analyzed_at": "2025-11-28T10:00:00Z",
|
||||
"analyzer_version": "1.0.0",
|
||||
"analysis_duration_ms": 450,
|
||||
"signal_extraction_method": "taxonomy_based",
|
||||
"pattern_detection_method": "clustering_and_temporal"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Analysis Explanation
|
||||
|
||||
### How Patterns Were Identified
|
||||
|
||||
#### 1. Initial Misconfiguration Pattern
|
||||
|
||||
**Detection method**: Single message with multiple corrections.
|
||||
|
||||
**Process**:
|
||||
1. Message 3 contains two correction signals (Bun, TypeScript)
|
||||
2. Both corrections happen within 5 minutes of task start
|
||||
3. Both relate to configuration choices agent made without checking
|
||||
4. Clustered corrections + early timing = misconfiguration pattern
|
||||
|
||||
**Key insight**: Agent should have checked project configuration before implementing.
|
||||
|
||||
#### 2. Security Requirements Pattern
|
||||
|
||||
**Detection method**: Recurring theme across multiple messages.
|
||||
|
||||
**Process**:
|
||||
1. Extract all request signals related to security
|
||||
2. Message 1: "hash passwords with bcrypt" (security)
|
||||
3. Message 5: "Always validate user input" (security)
|
||||
4. Both use requirement language ("make sure", "always")
|
||||
5. 2+ security requirements + consistent emphasis = security pattern
|
||||
|
||||
**Key insight**: User treats security as non-negotiable for auth features.
|
||||
|
||||
#### 3. Technology Preferences Pattern
|
||||
|
||||
**Detection method**: Explicit preference statements.
|
||||
|
||||
**Process**:
|
||||
1. Message 3 contains two preference/correction signals
|
||||
2. "I prefer TypeScript" = explicit preference language
|
||||
3. "use Bun not npm" = explicit tool choice
|
||||
4. Both stated as corrections = strong preferences
|
||||
5. Explicit preferences + corrections = preference pattern
|
||||
|
||||
**Key insight**: These are project-wide standards, not one-off choices.
|
||||
|
||||
#### 4. Frustration → Success Evolution
|
||||
|
||||
**Detection method**: Temporal sentiment analysis.
|
||||
|
||||
**Process**:
|
||||
1. Plot signals by timestamp and type
|
||||
2. T1 (9:05): Frustration (corrections)
|
||||
3. T2 (9:10): Mixed (weak success + correction)
|
||||
4. T3 (9:15): Success (praise + continuation)
|
||||
5. T4 (9:20): Strong success (praise + fulfillment)
|
||||
6. Monotonic improvement = evolution pattern
|
||||
|
||||
**Key insight**: Agent learned and improved through user feedback.
|
||||
|
||||
#### 5. Pattern Replication Request
|
||||
|
||||
**Detection method**: Explicit continuation signal.
|
||||
|
||||
**Process**:
|
||||
1. Message 7: "apply this same pattern to..."
|
||||
2. Explicit reference to successful approach
|
||||
3. Request to extend to similar feature
|
||||
4. High confidence continuation signal = replication pattern
|
||||
|
||||
**Key insight**: Successful patterns should be templated for reuse.
|
||||
|
||||
### Confidence Scoring Rationale
|
||||
|
||||
**High confidence signals** (10/12):
|
||||
- Explicit signal keywords ("Perfect!", "No", "Always", "I prefer")
|
||||
- Clear context supporting classification
|
||||
- No ambiguity in intent
|
||||
|
||||
**Medium confidence signals** (1/12):
|
||||
- "you forgot to add input validation" = correction but softer language
|
||||
- Implicit criticism rather than explicit negation
|
||||
|
||||
**Low confidence signals** (1/12):
|
||||
- "Good" = weak praise, especially with qualifier "but"
|
||||
- Could be polite rather than genuinely satisfied
|
||||
|
||||
### Action Items Prioritization
|
||||
|
||||
**High priority**: Project configuration issues that caused early corrections. These should be permanently fixed via memory updates.
|
||||
|
||||
**Medium priority**: Process improvements to prevent similar issues in future implementations.
|
||||
|
||||
**Low priority**: Documentation and knowledge capture for team benefit.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Always check configuration first**: Package manager and language preferences should be verified before writing code.
|
||||
|
||||
2. **Security is non-negotiable**: For auth features, security requirements (hashing, validation, tokens) should be included by default.
|
||||
|
||||
3. **Explicit preferences become requirements**: When users state preferences (especially with corrections), treat them as project standards.
|
||||
|
||||
4. **Successful patterns should be templated**: When a pattern gets explicit praise and continuation requests, create a reusable template.
|
||||
|
||||
5. **Feedback loops work**: Evolution from frustration to success shows that agent can learn and improve through user corrections.
|
||||
+854
@@ -0,0 +1,854 @@
|
||||
# Extraction Techniques
|
||||
|
||||
Technical methods for extracting signals from conversation history, including regex patterns, heuristics, and context analysis.
|
||||
|
||||
## Signal Detection Patterns
|
||||
|
||||
### Success Signals
|
||||
|
||||
#### Explicit Praise Detection
|
||||
|
||||
**Regex patterns**:
|
||||
|
||||
```regex
|
||||
# High confidence praise
|
||||
\b(perfect|excellent|exactly|amazing|brilliant|outstanding|superb)\b!*
|
||||
|
||||
# Medium confidence praise
|
||||
\b(great|good|nice|wonderful|fantastic)\b
|
||||
|
||||
# Superlatives
|
||||
\b(best|ideal|optimal|precisely)\s+(what|how|where)\s+
|
||||
|
||||
# Enthusiastic patterns
|
||||
!{2,}|🎉|👍|✅
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Check for exclamation marks: 1 = medium confidence, 2+ = high confidence
|
||||
2. Count positive adjectives in message: 2+ = high enthusiasm
|
||||
3. Check for "exactly what I needed" or similar fulfillment language
|
||||
4. Verify no contradictory signals in same message (e.g., "good but...")
|
||||
|
||||
**Context checks**:
|
||||
|
||||
```
|
||||
If praise detected:
|
||||
- Check previous agent message: What did agent do?
|
||||
- Check next user message: Did user continue or correct?
|
||||
- Score based on continuation vs. correction
|
||||
```
|
||||
|
||||
#### Continuation Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Explicit continuation
|
||||
\bnow\s+(do|apply|use|add|implement)\s+
|
||||
\bapply\s+this\s+(to|pattern|approach)\s+
|
||||
\bnext[,\s]+(let's|do|add)
|
||||
|
||||
# Extension language
|
||||
\b(also|additionally|furthermore|moreover)\s+
|
||||
\bsame\s+(for|with|to)\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Check if message references "this", "that", "same" without corrections
|
||||
2. Verify previous agent message exists (continuation requires prior context)
|
||||
3. Check for negation words: "now do X instead" is correction, not continuation
|
||||
4. Look for expansion keywords: "also", "and", "too", "as well"
|
||||
|
||||
**Context checks**:
|
||||
|
||||
```
|
||||
If continuation suspected:
|
||||
- Extract referenced prior work (parse "this", "that", "same")
|
||||
- Check for corrections between reference and current message
|
||||
- If corrections exist: Not continuation (likely correction signal)
|
||||
- If no corrections: Continuation (confidence based on explicitness)
|
||||
```
|
||||
|
||||
#### Adoption Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
This is primarily behavioral, not linguistic. Requires analyzing agent suggestions vs. user actions.
|
||||
|
||||
```
|
||||
Detection algorithm:
|
||||
1. Extract agent's suggestions from previous messages
|
||||
2. Check user's next message or code changes
|
||||
3. If user implements suggestion without modification:
|
||||
- Adoption signal (high confidence)
|
||||
4. If user asks clarifying questions then implements:
|
||||
- Adoption signal (medium confidence)
|
||||
5. If user modifies before implementing:
|
||||
- Partial adoption (low confidence) or correction signal
|
||||
```
|
||||
|
||||
**Code comparison heuristic**:
|
||||
|
||||
```javascript
|
||||
// Pseudocode for adoption detection
|
||||
function detectAdoption(agentMessage, userResponse) {
|
||||
const agentSuggestions = extractSuggestions(agentMessage);
|
||||
const userActions = extractActions(userResponse);
|
||||
|
||||
for (const suggestion of agentSuggestions) {
|
||||
const match = findMatchingAction(suggestion, userActions);
|
||||
if (match && match.similarity > 0.8) {
|
||||
return { signal: 'adoption', confidence: 'high' };
|
||||
} else if (match && match.similarity > 0.5) {
|
||||
return { signal: 'adoption', confidence: 'medium' };
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
### Frustration Signals
|
||||
|
||||
#### Correction Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Explicit negation
|
||||
\b(no[,\s]|wrong|incorrect|not\s+what|that's\s+not)\b
|
||||
|
||||
# Correction language
|
||||
\b(actually|instead|rather)\s+
|
||||
\bI\s+meant\s+
|
||||
\bdo\s+\w+\s+instead\b
|
||||
|
||||
# Contradiction markers
|
||||
\bdon't\s+do\s+\w+
|
||||
\bnot\s+\w+[,\s]+\w+
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Check for negation words followed by agent's previous output
|
||||
2. Look for "instead" patterns: "X instead of Y" where Y was agent's choice
|
||||
3. Check for contradiction: "I said X" where X contradicts recent agent action
|
||||
4. Verify correction vs. iteration: correction references agent error, iteration builds on success
|
||||
|
||||
**Context checks**:
|
||||
|
||||
```
|
||||
If correction suspected:
|
||||
- Extract what user is correcting (X → Y)
|
||||
- Check if agent did X in previous message
|
||||
- If yes: Correction signal (confidence based on negation strength)
|
||||
- If no: Possible misunderstanding or false positive
|
||||
```
|
||||
|
||||
#### Repetition Detection
|
||||
|
||||
**Pattern**:
|
||||
|
||||
This requires multi-message analysis.
|
||||
|
||||
```
|
||||
Detection algorithm:
|
||||
1. Extract normalized intent from each user message
|
||||
2. Build similarity matrix across messages
|
||||
3. Find clusters of high-similarity messages (>0.7 similarity)
|
||||
4. If cluster size >= 2 and spans multiple agent responses:
|
||||
- Repetition signal
|
||||
5. Check for escalation language ("again", "already told you"):
|
||||
- High confidence
|
||||
6. Otherwise: Medium confidence
|
||||
```
|
||||
|
||||
**Normalization steps**:
|
||||
|
||||
```javascript
|
||||
function normalizeIntent(message) {
|
||||
// Remove politeness/filler
|
||||
let normalized = message.toLowerCase();
|
||||
normalized = normalized.replace(/\b(please|thanks|thank you)\b/g, '');
|
||||
|
||||
// Extract core imperative
|
||||
const imperatives = normalized.match(/\b(use|do|make|add|implement|fix)\s+\w+/g);
|
||||
|
||||
// Extract prohibitions
|
||||
const prohibitions = normalized.match(/\bdon't\s+\w+/g);
|
||||
|
||||
return { imperatives, prohibitions };
|
||||
}
|
||||
|
||||
function calculateSimilarity(intent1, intent2) {
|
||||
// Check for matching imperatives or prohibitions
|
||||
// Return similarity score 0.0 - 1.0
|
||||
}
|
||||
```
|
||||
|
||||
**Escalation markers**:
|
||||
|
||||
```regex
|
||||
\b(again|once again|already|I\s+told\s+you|I\s+said|still)\b
|
||||
```
|
||||
|
||||
#### Explicit Frustration Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Direct frustration language
|
||||
\b(frustrat(ing|ed)|annoying|annoyed|confusion|confused)\b
|
||||
|
||||
# Problem statements
|
||||
\b(not\s+working|doesn't\s+work|broken|failing|fails)\b
|
||||
|
||||
# Accusatory questions
|
||||
\bwhy\s+(did|would|do)\s+you\s+
|
||||
|
||||
# Exasperation
|
||||
\bcome\s+on\b|\bseriously\b|\breally\?\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Question marks with negative tone: medium frustration
|
||||
2. Multiple question marks: high frustration
|
||||
3. All caps words: high frustration
|
||||
4. Repetition of negative words: escalating frustration
|
||||
|
||||
**Tone analysis**:
|
||||
|
||||
```javascript
|
||||
function analyzeTone(message) {
|
||||
const negativeWords = message.match(/\b(not|no|never|don't|can't|won't)\b/g);
|
||||
const negativeWordCount = negativeWords ? negativeWords.length : 0;
|
||||
|
||||
const questionMarks = message.match(/\?/g);
|
||||
const questionCount = questionMarks ? questionMarks.length : 0;
|
||||
|
||||
const capsWords = message.match(/\b[A-Z]{2,}\b/g);
|
||||
const capsCount = capsWords ? capsWords.length : 0;
|
||||
|
||||
// Frustration score
|
||||
const score = (negativeWordCount * 0.3) + (questionCount * 0.2) + (capsCount * 0.5);
|
||||
|
||||
if (score > 1.5) return 'high';
|
||||
if (score > 0.7) return 'medium';
|
||||
return 'low';
|
||||
}
|
||||
```
|
||||
|
||||
### Workflow Signals
|
||||
|
||||
#### Sequence Marker Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Ordinal markers
|
||||
\b(first|second|third|fourth|fifth)\b[,\s]
|
||||
\b(1st|2nd|3rd|4th|5th)\b[,\s]
|
||||
\bstep\s+\d+[:\s]
|
||||
|
||||
# Temporal sequence
|
||||
\b(before|after|then|next|finally)\b[,\s]
|
||||
\bonce\s+\w+[,\s]+(then|do|we)\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Count ordinal markers: 2+ = high confidence sequence
|
||||
2. Check for numbered lists (1., 2., 3.)
|
||||
3. Look for temporal connectives in order (first...then...finally)
|
||||
4. Verify sequence is prescriptive (steps to take) not descriptive (events that happened)
|
||||
|
||||
**List detection**:
|
||||
|
||||
```javascript
|
||||
function detectSequence(message) {
|
||||
// Check for numbered list
|
||||
const numberedItems = message.match(/^\d+\.\s+.+$/gm);
|
||||
if (numberedItems && numberedItems.length >= 2) {
|
||||
return { signal: 'sequence', confidence: 'high', items: numberedItems };
|
||||
}
|
||||
|
||||
// Check for ordinal markers
|
||||
const ordinals = message.match(/\b(first|second|third|then|next|finally)\b/gi);
|
||||
if (ordinals && ordinals.length >= 2) {
|
||||
return { signal: 'sequence', confidence: 'medium', markers: ordinals };
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
#### Stage Transition Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Completion + new direction
|
||||
\b(now\s+that|with\s+that|that's\s+done)\b.+\b(let's|next|moving|time\s+to)\b
|
||||
|
||||
# Explicit transitions
|
||||
\bmoving\s+on\s+to\b
|
||||
\bnext\s+up[:\s]
|
||||
\bswitching\s+to\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. Check for completion language: "done", "finished", "complete", "that's it"
|
||||
2. Check for new direction: "now", "next", "let's", "time to"
|
||||
3. Must have both completion and new direction for high confidence
|
||||
4. If only new direction: context switch, not stage transition
|
||||
|
||||
**Context checks**:
|
||||
|
||||
```
|
||||
If stage transition suspected:
|
||||
- Check if previous task mentioned in completion language
|
||||
- Verify previous task was in progress (not already complete)
|
||||
- Check if new direction is related (stage) or unrelated (context switch)
|
||||
- Related = stage transition, unrelated = context switch
|
||||
```
|
||||
|
||||
#### Tool Chain Detection
|
||||
|
||||
**Pattern**:
|
||||
|
||||
Requires analyzing agent's tool usage across multiple tasks.
|
||||
|
||||
```
|
||||
Detection algorithm:
|
||||
1. Extract tool call sequences from agent messages
|
||||
2. Group by task (task boundary = user message)
|
||||
3. Find recurring sequences:
|
||||
- Use n-gram analysis (n=2 to 5)
|
||||
- Count frequency of each n-gram
|
||||
- Filter to sequences with frequency >= 3
|
||||
4. For each recurring sequence:
|
||||
- Calculate confidence based on frequency and consistency
|
||||
- Extract as tool chain pattern
|
||||
```
|
||||
|
||||
**N-gram analysis**:
|
||||
|
||||
```javascript
|
||||
function extractToolChains(agentMessages) {
|
||||
const sequences = [];
|
||||
|
||||
for (const message of agentMessages) {
|
||||
const tools = extractToolCalls(message); // ['Read', 'Edit', 'Bash']
|
||||
sequences.push(tools);
|
||||
}
|
||||
|
||||
// Find recurring n-grams
|
||||
const ngrams = {};
|
||||
for (const seq of sequences) {
|
||||
for (let n = 2; n <= Math.min(5, seq.length); n++) {
|
||||
for (let i = 0; i <= seq.length - n; i++) {
|
||||
const gram = seq.slice(i, i + n).join(' → ');
|
||||
ngrams[gram] = (ngrams[gram] || 0) + 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Filter to frequent patterns
|
||||
const chains = Object.entries(ngrams)
|
||||
.filter(([gram, count]) => count >= 3)
|
||||
.map(([gram, count]) => ({
|
||||
chain: gram,
|
||||
frequency: count,
|
||||
confidence: count >= 5 ? 'high' : count >= 3 ? 'medium' : 'low'
|
||||
}));
|
||||
|
||||
return chains;
|
||||
}
|
||||
```
|
||||
|
||||
### Request Signals
|
||||
|
||||
#### Prohibition Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Absolute prohibitions
|
||||
\b(never|don't|do\s+not)\s+
|
||||
\bavoid\s+(using|doing)\s+
|
||||
|
||||
# Explicit constraints
|
||||
\bno\s+\w+\b
|
||||
\bwithout\s+\w+\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. "Never" = high confidence prohibition
|
||||
2. "Don't" + imperative = high confidence
|
||||
3. "Avoid" = medium confidence (softer prohibition)
|
||||
4. "No X" = context-dependent (check if X is an action or noun)
|
||||
|
||||
**Context checks**:
|
||||
|
||||
```
|
||||
If prohibition suspected:
|
||||
- Extract prohibited action/item
|
||||
- Check for exceptions: "don't X unless Y"
|
||||
- If exception: conditional signal, not absolute prohibition
|
||||
- If no exception: prohibition (confidence based on strength of negation)
|
||||
```
|
||||
|
||||
#### Requirement Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Modal verbs
|
||||
\b(must|should|need\s+to|have\s+to|always)\s+
|
||||
|
||||
# Imperatives with emphasis
|
||||
\bmake\s+sure\s+(to\s+)?
|
||||
\bensure\s+(that\s+)?
|
||||
\bremember\s+to\s+
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. "Must" / "Always" = high confidence requirement
|
||||
2. "Should" = medium confidence requirement
|
||||
3. "Make sure" = medium confidence requirement
|
||||
4. Bare imperative ("Run tests") = context-dependent
|
||||
|
||||
**Strength scoring**:
|
||||
|
||||
```javascript
|
||||
function classifyRequirement(message) {
|
||||
if (/\b(must|always|required)\b/i.test(message)) {
|
||||
return { strength: 'strong', confidence: 'high' };
|
||||
}
|
||||
if (/\b(should|need\s+to|make\s+sure)\b/i.test(message)) {
|
||||
return { strength: 'moderate', confidence: 'medium' };
|
||||
}
|
||||
if (/\b(could|might\s+want\s+to|consider)\b/i.test(message)) {
|
||||
return { strength: 'weak', confidence: 'low' };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
#### Preference Detection
|
||||
|
||||
**Patterns**:
|
||||
|
||||
```regex
|
||||
# Explicit preference
|
||||
\bI\s+prefer\s+
|
||||
\bI'd\s+rather\s+
|
||||
\bI\s+like\s+\w+\s+(better|more)\b
|
||||
|
||||
# Comparative language
|
||||
\b(better|cleaner|easier|simpler)\s+to\s+
|
||||
\bX\s+over\s+Y\b
|
||||
```
|
||||
|
||||
**Heuristic rules**:
|
||||
|
||||
1. "I prefer X" = high confidence preference
|
||||
2. "X is better" = medium confidence (could be objective claim)
|
||||
3. "I like X" = low confidence (weak preference)
|
||||
4. Check for comparison: "X over Y" or "X not Y" strengthens signal
|
||||
|
||||
**Subjectivity detection**:
|
||||
|
||||
```javascript
|
||||
function isSubjective(statement) {
|
||||
// Check for first-person markers
|
||||
const firstPerson = /\b(I|my|me)\b/i.test(statement);
|
||||
|
||||
// Check for subjective language
|
||||
const subjective = /\b(prefer|like|rather|think|believe|feel)\b/i.test(statement);
|
||||
|
||||
// Check for evaluative language
|
||||
const evaluative = /\b(better|worse|best|worst|cleaner|messier)\b/i.test(statement);
|
||||
|
||||
return firstPerson || subjective || evaluative;
|
||||
}
|
||||
```
|
||||
|
||||
## Message Boundary Detection
|
||||
|
||||
Identify where user messages begin and end, separating from agent messages and tool outputs.
|
||||
|
||||
### Actor Classification
|
||||
|
||||
```javascript
|
||||
function classifyActor(message) {
|
||||
// Check for role markers
|
||||
if (message.role === 'user') return 'user';
|
||||
if (message.role === 'assistant') return 'agent';
|
||||
|
||||
// Fallback to content analysis
|
||||
if (/<function_calls>/i.test(message.content)) return 'agent';
|
||||
if (/<function_results>/i.test(message.content)) return 'tool';
|
||||
|
||||
// Default to user for ambiguous cases
|
||||
return 'user';
|
||||
}
|
||||
```
|
||||
|
||||
### Message Filtering
|
||||
|
||||
```javascript
|
||||
function filterMessages(conversation, options = {}) {
|
||||
const {
|
||||
actors = ['user', 'agent'],
|
||||
excludeToolOutput = true,
|
||||
excludeCodeBlocks = false,
|
||||
minLength = 0,
|
||||
} = options;
|
||||
|
||||
return conversation
|
||||
.filter(msg => actors.includes(classifyActor(msg)))
|
||||
.filter(msg => !excludeToolOutput || !msg.content.includes('<function_results>'))
|
||||
.filter(msg => !excludeCodeBlocks || !msg.content.includes('```'))
|
||||
.filter(msg => msg.content.length >= minLength);
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Turn Pattern Recognition
|
||||
|
||||
Detect patterns that span multiple messages.
|
||||
|
||||
### Escalation Detection
|
||||
|
||||
```javascript
|
||||
function detectEscalation(messages) {
|
||||
// Group messages by topic
|
||||
const topics = clusterByTopic(messages);
|
||||
|
||||
for (const topic of topics) {
|
||||
// Check if frustration increases over time
|
||||
const frustrationScores = topic.messages.map(msg => {
|
||||
const signals = extractSignals(msg);
|
||||
return signals.filter(s => s.type === 'frustration').length;
|
||||
});
|
||||
|
||||
// Check for monotonic increase
|
||||
let isEscalating = true;
|
||||
for (let i = 1; i < frustrationScores.length; i++) {
|
||||
if (frustrationScores[i] <= frustrationScores[i - 1]) {
|
||||
isEscalating = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (isEscalating && frustrationScores.length >= 2) {
|
||||
return {
|
||||
pattern: 'escalation',
|
||||
topic: topic.name,
|
||||
messages: topic.messages,
|
||||
confidence: frustrationScores.length >= 3 ? 'high' : 'medium'
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
### Topic Clustering
|
||||
|
||||
```javascript
|
||||
function clusterByTopic(messages) {
|
||||
// Simple keyword-based clustering
|
||||
const clusters = [];
|
||||
|
||||
for (const msg of messages) {
|
||||
const keywords = extractKeywords(msg);
|
||||
|
||||
// Find existing cluster with matching keywords
|
||||
let matched = false;
|
||||
for (const cluster of clusters) {
|
||||
const overlap = keywords.filter(k => cluster.keywords.includes(k));
|
||||
if (overlap.length / keywords.length > 0.3) {
|
||||
cluster.messages.push(msg);
|
||||
cluster.keywords = [...new Set([...cluster.keywords, ...keywords])];
|
||||
matched = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Create new cluster if no match
|
||||
if (!matched) {
|
||||
clusters.push({
|
||||
name: keywords[0] || 'unnamed',
|
||||
keywords,
|
||||
messages: [msg]
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return clusters;
|
||||
}
|
||||
|
||||
function extractKeywords(message) {
|
||||
// Remove stop words and extract nouns/verbs
|
||||
const stopWords = new Set(['the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for']);
|
||||
|
||||
const words = message.content
|
||||
.toLowerCase()
|
||||
.replace(/[^\w\s]/g, '')
|
||||
.split(/\s+/)
|
||||
.filter(w => w.length > 3 && !stopWords.has(w));
|
||||
|
||||
// Return top 5 most frequent words
|
||||
const freq = {};
|
||||
for (const word of words) {
|
||||
freq[word] = (freq[word] || 0) + 1;
|
||||
}
|
||||
|
||||
return Object.entries(freq)
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 5)
|
||||
.map(([word]) => word);
|
||||
}
|
||||
```
|
||||
|
||||
## Context Analysis Methods
|
||||
|
||||
### Recency Weighting
|
||||
|
||||
More recent signals should carry more weight than older ones.
|
||||
|
||||
```javascript
|
||||
function applyRecencyWeight(signals, halfLifeDays = 7) {
|
||||
const now = Date.now();
|
||||
const halfLifeMs = halfLifeDays * 24 * 60 * 60 * 1000;
|
||||
|
||||
return signals.map(signal => {
|
||||
const age = now - signal.timestamp;
|
||||
const weight = Math.pow(0.5, age / halfLifeMs);
|
||||
|
||||
return {
|
||||
...signal,
|
||||
weight,
|
||||
weightedConfidence: signal.confidence * weight
|
||||
};
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Contradiction Resolution
|
||||
|
||||
When signals conflict, resolve using recency and confidence.
|
||||
|
||||
```javascript
|
||||
function resolveContradictions(signals) {
|
||||
// Group by topic
|
||||
const groups = groupByTopic(signals);
|
||||
|
||||
for (const group of groups) {
|
||||
// Sort by timestamp (newest first)
|
||||
group.signals.sort((a, b) => b.timestamp - a.timestamp);
|
||||
|
||||
// Check for contradictions
|
||||
const contradictions = findContradictions(group.signals);
|
||||
|
||||
for (const [newer, older] of contradictions) {
|
||||
if (newer.confidence >= older.confidence) {
|
||||
// Mark older signal as superseded
|
||||
older.superseded = true;
|
||||
older.supersededBy = newer.message_id;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Filter out superseded signals
|
||||
return signals.filter(s => !s.superseded);
|
||||
}
|
||||
|
||||
function findContradictions(signals) {
|
||||
const pairs = [];
|
||||
|
||||
for (let i = 0; i < signals.length; i++) {
|
||||
for (let j = i + 1; j < signals.length; j++) {
|
||||
if (areContradictory(signals[i], signals[j])) {
|
||||
pairs.push([signals[i], signals[j]]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return pairs;
|
||||
}
|
||||
|
||||
function areContradictory(signal1, signal2) {
|
||||
// Example: "Use X" vs. "Don't use X"
|
||||
if (signal1.type === 'request' && signal2.type === 'request') {
|
||||
// Extract actions
|
||||
const action1 = signal1.quote.match(/\b(use|do|make|add)\s+(\w+)/i);
|
||||
const action2 = signal2.quote.match(/\b(don't|never|avoid)\s+(use|do|make|add)?\s*(\w+)/i);
|
||||
|
||||
if (action1 && action2 && action1[2] === action2[3]) {
|
||||
return true; // Contradiction: "use X" vs. "don't use X"
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Incremental Analysis
|
||||
|
||||
For long conversations, analyze incrementally rather than re-analyzing entire history.
|
||||
|
||||
```javascript
|
||||
class IncrementalAnalyzer {
|
||||
constructor() {
|
||||
this.lastAnalyzedIndex = 0;
|
||||
this.signals = [];
|
||||
this.patterns = [];
|
||||
}
|
||||
|
||||
analyze(messages) {
|
||||
// Only analyze new messages
|
||||
const newMessages = messages.slice(this.lastAnalyzedIndex);
|
||||
|
||||
// Extract signals from new messages
|
||||
const newSignals = newMessages.flatMap(msg => extractSignals(msg));
|
||||
this.signals.push(...newSignals);
|
||||
|
||||
// Update patterns with new signals
|
||||
this.patterns = detectPatterns(this.signals);
|
||||
|
||||
// Update index
|
||||
this.lastAnalyzedIndex = messages.length;
|
||||
|
||||
return {
|
||||
signals: this.signals,
|
||||
patterns: this.patterns
|
||||
};
|
||||
}
|
||||
|
||||
reset() {
|
||||
this.lastAnalyzedIndex = 0;
|
||||
this.signals = [];
|
||||
this.patterns = [];
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Caching
|
||||
|
||||
Cache expensive operations like topic clustering and keyword extraction.
|
||||
|
||||
```javascript
|
||||
class AnalysisCache {
|
||||
constructor(ttlMs = 5 * 60 * 1000) { // 5 minute TTL
|
||||
this.cache = new Map();
|
||||
this.ttl = ttlMs;
|
||||
}
|
||||
|
||||
get(key) {
|
||||
const entry = this.cache.get(key);
|
||||
if (!entry) return null;
|
||||
|
||||
if (Date.now() - entry.timestamp > this.ttl) {
|
||||
this.cache.delete(key);
|
||||
return null;
|
||||
}
|
||||
|
||||
return entry.value;
|
||||
}
|
||||
|
||||
set(key, value) {
|
||||
this.cache.set(key, {
|
||||
value,
|
||||
timestamp: Date.now()
|
||||
});
|
||||
}
|
||||
|
||||
clear() {
|
||||
this.cache.clear();
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const cache = new AnalysisCache();
|
||||
|
||||
function extractSignalsWithCache(message) {
|
||||
const key = `signals:${message.id}`;
|
||||
const cached = cache.get(key);
|
||||
|
||||
if (cached) return cached;
|
||||
|
||||
const signals = extractSignals(message);
|
||||
cache.set(key, signals);
|
||||
|
||||
return signals;
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Graceful Degradation
|
||||
|
||||
If signal extraction fails for a message, continue with remaining messages.
|
||||
|
||||
```javascript
|
||||
function extractSignalsSafe(messages) {
|
||||
const results = {
|
||||
signals: [],
|
||||
errors: []
|
||||
};
|
||||
|
||||
for (const msg of messages) {
|
||||
try {
|
||||
const signals = extractSignals(msg);
|
||||
results.signals.push(...signals);
|
||||
} catch (error) {
|
||||
results.errors.push({
|
||||
message_id: msg.id,
|
||||
error: error.message
|
||||
});
|
||||
// Continue with next message
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
Validate extracted signals before adding to results.
|
||||
|
||||
```javascript
|
||||
function validateSignal(signal) {
|
||||
const required = ['type', 'subtype', 'message_id', 'quote', 'confidence'];
|
||||
|
||||
for (const field of required) {
|
||||
if (!(field in signal)) {
|
||||
throw new Error(`Missing required field: ${field}`);
|
||||
}
|
||||
}
|
||||
|
||||
const validTypes = ['success', 'frustration', 'workflow', 'request'];
|
||||
if (!validTypes.includes(signal.type)) {
|
||||
throw new Error(`Invalid signal type: ${signal.type}`);
|
||||
}
|
||||
|
||||
if (signal.confidence < 0 || signal.confidence > 1) {
|
||||
throw new Error(`Confidence must be 0-1, got: ${signal.confidence}`);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
```
|
||||
+511
@@ -0,0 +1,511 @@
|
||||
# Signal Patterns Reference
|
||||
|
||||
Extended taxonomy with edge cases, disambiguation guidance, and confidence scoring rubric.
|
||||
|
||||
## Success Signals
|
||||
|
||||
### Explicit Praise
|
||||
|
||||
**Core indicators**: Positive adjectives, exclamation marks, superlatives.
|
||||
|
||||
**Examples**:
|
||||
- "Perfect!"
|
||||
- "Exactly what I needed"
|
||||
- "This is great work"
|
||||
- "Love it"
|
||||
- "Well done"
|
||||
|
||||
**Edge cases**:
|
||||
- "Good enough" → Low confidence (lukewarm, not enthusiastic)
|
||||
- "That works" → Low confidence (neutral acceptance, not praise)
|
||||
- "Thanks" → Context-dependent (could be courtesy, not satisfaction)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Superlatives ("perfect", "excellent"), multiple exclamations, enthusiastic tone
|
||||
- **Medium**: Positive adjectives ("good", "nice") with neutral tone
|
||||
- **Low**: Minimal positive language, could be polite rather than satisfied
|
||||
|
||||
### Continuation
|
||||
|
||||
**Core indicators**: Building on previous work, extending scope, applying pattern elsewhere.
|
||||
|
||||
**Examples**:
|
||||
- "Now do the same for the login page"
|
||||
- "Apply this pattern to all API routes"
|
||||
- "Great, next let's handle the error cases"
|
||||
|
||||
**Edge cases**:
|
||||
- "Now try X instead" → Frustration (correction) if contradicts prior work
|
||||
- "Also do Y" → Continuation only if X succeeded; check for corrections first
|
||||
- "Next, fix the bug in Z" → Context switch, not continuation
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit reference to prior success + request to extend
|
||||
- **Medium**: Implied satisfaction + new related task
|
||||
- **Low**: Sequential tasks without confirmation of prior success
|
||||
|
||||
### Adoption
|
||||
|
||||
**Core indicators**: User implements agent's suggestion without modification or pushback.
|
||||
|
||||
**Examples**:
|
||||
- Agent: "Use TanStack Router" → User: *next message shows TanStack Router implementation*
|
||||
- Agent: "Refactor to use async/await" → User: "Done, looks cleaner"
|
||||
- Agent suggests pattern → User's code follows pattern exactly
|
||||
|
||||
**Edge cases**:
|
||||
- User modifies suggestion before implementing → Medium confidence (partial adoption)
|
||||
- User implements after asking clarifying questions → Still adoption (high confidence)
|
||||
- User implements weeks later → Weak adoption signal (confounded by time)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Immediate implementation with no modifications
|
||||
- **Medium**: Implementation after clarification or with minor adjustments
|
||||
- **Low**: Implementation significantly delayed or heavily modified
|
||||
|
||||
### Completion Acceptance
|
||||
|
||||
**Core indicators**: Approval language followed by action (merge, ship, close ticket).
|
||||
|
||||
**Examples**:
|
||||
- "Looks good, merge it"
|
||||
- "Ship it"
|
||||
- "Perfect, closing this issue"
|
||||
- "LGTM" (Looks Good To Me)
|
||||
|
||||
**Edge cases**:
|
||||
- "Looks good, but..." → Conditional acceptance, check for follow-up corrections
|
||||
- "Merge" without review → Context-dependent (could be trust or urgency, not satisfaction)
|
||||
- "Ship it" with sarcasm → Rare but check surrounding context for frustration
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit approval + action directive
|
||||
- **Medium**: Approval without action or action without explicit approval
|
||||
- **Low**: Ambiguous approval ("fine", "okay") that could indicate resignation
|
||||
|
||||
## Frustration Signals
|
||||
|
||||
### Correction
|
||||
|
||||
**Core indicators**: Negation words, contradiction of agent output, explicit redirection.
|
||||
|
||||
**Examples**:
|
||||
- "No, I meant X not Y"
|
||||
- "That's wrong, do Z instead"
|
||||
- "Actually, use A instead of B"
|
||||
|
||||
**Edge cases**:
|
||||
- "Small correction: use X" → Frustration (low confidence) if agent should have known
|
||||
- "Let's adjust to X" → Not frustration if iterating on shared work
|
||||
- "Change X to Y" → Context-dependent (correction vs. evolution)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit negation ("no", "wrong", "don't") + correction
|
||||
- **Medium**: Implicit correction ("actually", "instead") without harsh language
|
||||
- **Low**: Neutral adjustment language that could be iteration not correction
|
||||
|
||||
### Reversion
|
||||
|
||||
**Core indicators**: Request to undo agent's changes, return to previous state.
|
||||
|
||||
**Examples**:
|
||||
- "Revert that change"
|
||||
- "Go back to the original version"
|
||||
- "Undo what you just did"
|
||||
- User manually reverts agent's commit
|
||||
|
||||
**Edge cases**:
|
||||
- "Let's try the original approach" → Could be exploration, not frustration
|
||||
- Revert after testing both options → Scientific method, not frustration
|
||||
- Partial revert → Medium confidence (some parts worked, some didn't)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit revert request with no justification (implies failure)
|
||||
- **Medium**: Revert with explanation that agent's approach had issues
|
||||
- **Low**: Revert as part of A/B testing or exploration
|
||||
|
||||
### Repetition
|
||||
|
||||
**Core indicators**: Same request issued 2+ times with escalating specificity or frustration.
|
||||
|
||||
**Examples**:
|
||||
- Message 1: "Use Bun not npm"
|
||||
- Message 2: "Again, use Bun"
|
||||
- Message 3: "I already told you to use Bun!"
|
||||
|
||||
**Edge cases**:
|
||||
- Repetition after context switch → May not be frustration, could be reminder
|
||||
- Repetition with new information → Evolution, not frustration
|
||||
- Repetition across different tasks → Preference signal, not frustration
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: 3+ repetitions with escalating tone or "again"/"already told you"
|
||||
- **Medium**: 2 repetitions with no new context
|
||||
- **Low**: 2 similar requests in different contexts (could be unrelated)
|
||||
|
||||
### Explicit Frustration
|
||||
|
||||
**Core indicators**: Direct expression of dissatisfaction, questioning agent's behavior.
|
||||
|
||||
**Examples**:
|
||||
- "This isn't working"
|
||||
- "Why did you do X when I said Y?"
|
||||
- "I already told you not to..."
|
||||
- "This is frustrating"
|
||||
|
||||
**Edge cases**:
|
||||
- "Hmm, that's odd" → Confusion, not necessarily frustration
|
||||
- "Why X?" → Curiosity if neutral tone; frustration if accusatory
|
||||
- "Not quite" → Gentle correction, low frustration
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit frustration words ("frustrating", "annoying") or accusatory questions
|
||||
- **Medium**: Implied dissatisfaction ("not working", "this is wrong")
|
||||
- **Low**: Neutral problem statements without emotional language
|
||||
|
||||
## Workflow Signals
|
||||
|
||||
### Sequence Markers
|
||||
|
||||
**Core indicators**: Ordinal language, numbered lists, temporal connectives.
|
||||
|
||||
**Examples**:
|
||||
- "First, do X. Then Y. Finally Z."
|
||||
- "Step 1: A, Step 2: B, Step 3: C"
|
||||
- "Before we start, let's..."
|
||||
|
||||
**Edge cases**:
|
||||
- Single "first" without "second" → Low confidence (could be emphasis, not sequence)
|
||||
- "Then" without "first" → Continuation, not new sequence
|
||||
- Numbered list describing features (not steps) → Not workflow signal
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Multiple ordinal markers (first, second, third) or numbered steps
|
||||
- **Medium**: Single sequence marker with clear temporal relationship
|
||||
- **Low**: Ambiguous temporal language ("before", "after") without clear sequence
|
||||
|
||||
### Stage Transitions
|
||||
|
||||
**Core indicators**: Reference to completion + new direction, explicit context shift.
|
||||
|
||||
**Examples**:
|
||||
- "Now that X is done, let's work on Y"
|
||||
- "Moving on to the API layer"
|
||||
- "With that complete, next is..."
|
||||
|
||||
**Edge cases**:
|
||||
- "Let's do Y" without completion reference → Context switch, not stage transition
|
||||
- "After X, do Y" (pre-planning) → Sequence marker, not transition
|
||||
- "Y is next" → Future reference, not active transition
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit completion reference + new task
|
||||
- **Medium**: Implied completion + new direction
|
||||
- **Low**: New task without completion signal (could be interruption)
|
||||
|
||||
### Tool Chains
|
||||
|
||||
**Core indicators**: Consistent sequence of tool usage across multiple tasks.
|
||||
|
||||
**Examples**:
|
||||
- Pattern: Read → Edit → Bash (test) appears 5+ times
|
||||
- Pattern: Glob → Grep → Read appears 3+ times for search tasks
|
||||
- Pattern: Write → Bash (validate) appears 4+ times
|
||||
|
||||
**Edge cases**:
|
||||
- Same tools in different order → Not a chain, separate usage
|
||||
- Tool chain appears once → Not a pattern yet
|
||||
- Tool chain broken by user interruption → Still valid if resumes afterward
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: 5+ occurrences of same tool sequence
|
||||
- **Medium**: 3-4 occurrences with occasional variation
|
||||
- **Low**: 2 occurrences or significant variation in sequence
|
||||
|
||||
### Context Switches
|
||||
|
||||
**Core indicators**: Abrupt topic change, new file focus, no transition language.
|
||||
|
||||
**Examples**:
|
||||
- Working on auth.ts → Suddenly "Fix the database schema"
|
||||
- Discussing React components → "Now debug the API"
|
||||
- Mid-task: "Actually, let's work on something else"
|
||||
|
||||
**Edge cases**:
|
||||
- Switch after completing task → Stage transition, not context switch
|
||||
- Switch with explanation → Intentional pivot, still a switch but lower friction
|
||||
- Return to previous context → Resumption, not new switch
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Abrupt change with no transition, different domain
|
||||
- **Medium**: Change with minimal transition or related domain
|
||||
- **Low**: Change with explanation or natural task completion
|
||||
|
||||
## Request Signals
|
||||
|
||||
### Prohibition
|
||||
|
||||
**Core indicators**: Negative imperatives, explicit constraints, "don't" statements.
|
||||
|
||||
**Examples**:
|
||||
- "Don't use any types"
|
||||
- "Never use npm, always use Bun"
|
||||
- "Avoid using classes"
|
||||
|
||||
**Edge cases**:
|
||||
- "I wouldn't use X" → Preference, not prohibition (softer language)
|
||||
- "Don't do X unless Y" → Conditional prohibition
|
||||
- "Try not to X" → Soft prohibition (low confidence)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Absolute negatives ("never", "don't", "no") with no conditions
|
||||
- **Medium**: Soft negatives ("avoid", "try not to") or conditional prohibitions
|
||||
- **Low**: Implicit discouragement without explicit prohibition
|
||||
|
||||
### Requirement
|
||||
|
||||
**Core indicators**: Absolute language, modal verbs (must, should, always), imperatives.
|
||||
|
||||
**Examples**:
|
||||
- "Always run tests before committing"
|
||||
- "You must validate input"
|
||||
- "Make sure to check for errors"
|
||||
|
||||
**Edge cases**:
|
||||
- "It's good to X" → Preference, not requirement
|
||||
- "Should probably X" → Weak requirement (medium confidence)
|
||||
- "Try to X" → Suggestion, not requirement
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Absolute modal verbs ("must", "always") or strong imperatives
|
||||
- **Medium**: Soft modal verbs ("should") or qualified requirements
|
||||
- **Low**: Suggestions ("could", "might want to") without strong language
|
||||
|
||||
### Preference
|
||||
|
||||
**Core indicators**: Comparative language, subjective statements, "prefer" / "better" / "rather".
|
||||
|
||||
**Examples**:
|
||||
- "I prefer TypeScript over JavaScript"
|
||||
- "It's better to use async/await"
|
||||
- "I'd rather use functional components"
|
||||
|
||||
**Edge cases**:
|
||||
- "I like X" → Weak preference (low confidence)
|
||||
- "X is better" (stated as fact) → Could be requirement depending on tone
|
||||
- "Prefer X, but Y works too" → Flexible preference (medium confidence)
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit preference language ("prefer", "I'd rather") with comparison
|
||||
- **Medium**: Implied preference through evaluation ("better", "cleaner")
|
||||
- **Low**: Weak positive statements without comparison
|
||||
|
||||
### Conditional
|
||||
|
||||
**Core indicators**: Logical connectives (if/then, when, unless), situational rules.
|
||||
|
||||
**Examples**:
|
||||
- "If X then Y"
|
||||
- "When working on auth, always use Z"
|
||||
- "Unless A, do B"
|
||||
|
||||
**Edge cases**:
|
||||
- "Maybe if X" → Uncertain conditional (low confidence)
|
||||
- "X or Y depending on Z" → Multiple conditionals, complex rule
|
||||
- "If X" without "then" → Incomplete conditional, infer consequence from context
|
||||
|
||||
**Confidence criteria**:
|
||||
- **High**: Explicit if/then structure with clear condition and action
|
||||
- **Medium**: Implied conditional (when, unless) or incomplete structure
|
||||
- **Low**: Vague conditional ("depending on", "maybe if")
|
||||
|
||||
## Disambiguation Guidance
|
||||
|
||||
### Success vs. Frustration
|
||||
|
||||
**Ambiguous case**: "That works"
|
||||
|
||||
- **Success** if: No prior corrections, agent's first attempt, user moves on
|
||||
- **Frustration** if: After multiple attempts, lukewarm tone, user makes adjustments
|
||||
|
||||
**Rule**: Check for prior corrections. If 0-1, success. If 2+, frustration.
|
||||
|
||||
### Continuation vs. Correction
|
||||
|
||||
**Ambiguous case**: "Now do X for Y too"
|
||||
|
||||
- **Continuation** if: X succeeded for original target, Y is similar target
|
||||
- **Correction** if: X failed for original target, Y is different approach
|
||||
|
||||
**Rule**: Verify success of X before classifying as continuation.
|
||||
|
||||
### Request vs. Workflow
|
||||
|
||||
**Ambiguous case**: "Always run tests before committing"
|
||||
|
||||
- **Request (requirement)** if: User establishing new rule
|
||||
- **Workflow (sequence)** if: User describing existing process
|
||||
|
||||
**Rule**: Check if user is prescribing (request) or describing (workflow). Use tense as clue: imperative = request, present tense = workflow.
|
||||
|
||||
### Preference vs. Requirement
|
||||
|
||||
**Ambiguous case**: "Use TypeScript"
|
||||
|
||||
- **Preference** if: Soft language, alternatives mentioned, presented as opinion
|
||||
- **Requirement** if: Absolute language, no alternatives, presented as rule
|
||||
|
||||
**Rule**: Look for qualifiers. "I prefer X" = preference. "Use X" = requirement. "Always use X" = strong requirement.
|
||||
|
||||
## Confidence Scoring Rubric
|
||||
|
||||
### High Confidence (0.8 - 1.0)
|
||||
|
||||
- Explicit signal keywords match taxonomy exactly
|
||||
- No ambiguity in language or intent
|
||||
- Context strongly supports classification
|
||||
- Multiple supporting clues (tone, punctuation, surrounding messages)
|
||||
|
||||
**Example**: "Don't use npm, always use Bun" → Prohibition (high confidence)
|
||||
|
||||
### Medium Confidence (0.5 - 0.79)
|
||||
|
||||
- Implicit signal requiring some interpretation
|
||||
- Context provides partial support
|
||||
- Minor ambiguity but best classification is clear
|
||||
- Single supporting clue or mixed signals
|
||||
|
||||
**Example**: "Bun is better here" → Preference (medium confidence)
|
||||
|
||||
### Low Confidence (0.2 - 0.49)
|
||||
|
||||
- Ambiguous language with multiple possible interpretations
|
||||
- Weak or contradictory context
|
||||
- Signal requires significant inference
|
||||
- Borderline between two signal types
|
||||
|
||||
**Example**: "That's fine" → Success? Frustration? (low confidence)
|
||||
|
||||
### No Signal (< 0.2)
|
||||
|
||||
- Neutral language with no clear signal
|
||||
- Insufficient context to classify
|
||||
- Pure information exchange with no behavioral indicator
|
||||
|
||||
**Example**: "The file is at /path/to/file" → No signal
|
||||
|
||||
## Signal Combinations
|
||||
|
||||
Certain signal combinations indicate specific patterns:
|
||||
|
||||
### Success → Continuation
|
||||
|
||||
Pattern: **Positive Reinforcement Loop**
|
||||
|
||||
User satisfied with approach and wants to extend it.
|
||||
|
||||
```
|
||||
Message 1: "Perfect! This works great" (success: explicit praise)
|
||||
Message 2: "Now apply this to all the other routes" (success: continuation)
|
||||
```
|
||||
|
||||
**Recommendation**: Internalize the successful approach as a pattern to reuse.
|
||||
|
||||
### Frustration (repetition) → Success
|
||||
|
||||
Pattern: **Learning Curve Overcome**
|
||||
|
||||
Agent initially misunderstood but eventually delivered correctly.
|
||||
|
||||
```
|
||||
Message 1: "Use Bun not npm" (frustration: correction)
|
||||
Message 2: "Again, Bun not npm" (frustration: repetition)
|
||||
Message 3: "Perfect, that's the right package manager" (success: praise)
|
||||
```
|
||||
|
||||
**Recommendation**: Add the learned requirement to memory to avoid future repetition.
|
||||
|
||||
### Request (prohibition) + Frustration (repetition)
|
||||
|
||||
Pattern: **Persistent Violation**
|
||||
|
||||
Agent repeatedly violates an explicit constraint.
|
||||
|
||||
```
|
||||
Message 1: "Don't use any types" (request: prohibition)
|
||||
Message 3: "I said no any types" (frustration: repetition + request)
|
||||
Message 5: "Again, avoid any types" (frustration: repetition + request)
|
||||
```
|
||||
|
||||
**Recommendation**: Escalate to memory update or configuration change. This is a critical user requirement being violated.
|
||||
|
||||
### Workflow (tool chain) + Success (adoption)
|
||||
|
||||
Pattern: **Workflow Optimization**
|
||||
|
||||
User established efficient tool sequence and agent adopted it.
|
||||
|
||||
```
|
||||
Message 1: "Read, then Edit, then run tests" (workflow: sequence)
|
||||
Message 3: *agent follows sequence* (workflow: tool chain)
|
||||
Message 5: "Great, you've got the workflow down" (success: praise)
|
||||
```
|
||||
|
||||
**Recommendation**: Codify tool chain as a standard workflow template.
|
||||
|
||||
### Context Switch + Frustration (explicit)
|
||||
|
||||
Pattern: **Blocked Progress**
|
||||
|
||||
User abandons current task due to persistent issues.
|
||||
|
||||
```
|
||||
Message 1: "This isn't working" (frustration: explicit)
|
||||
Message 2: "Let's work on something else instead" (workflow: context switch)
|
||||
```
|
||||
|
||||
**Recommendation**: Mark the abandoned task for later review. Likely indicates a blocker or knowledge gap.
|
||||
|
||||
## Temporal Patterns
|
||||
|
||||
### Escalation Pattern
|
||||
|
||||
Signal intensity increases over time.
|
||||
|
||||
```
|
||||
T1: "Use Bun" (request: preference)
|
||||
T2: "Please use Bun, not npm" (request: requirement)
|
||||
T3: "I already told you to use Bun!" (frustration: repetition + explicit)
|
||||
```
|
||||
|
||||
**Detection**: Same topic, increasing frustration or stronger modal verbs.
|
||||
|
||||
**Recommendation**: High-priority memory update. User is emphasizing this requirement.
|
||||
|
||||
### De-escalation Pattern
|
||||
|
||||
Frustration decreases as issue resolves.
|
||||
|
||||
```
|
||||
T1: "This is broken" (frustration: explicit)
|
||||
T2: "Getting closer" (neutral)
|
||||
T3: "Perfect!" (success: praise)
|
||||
```
|
||||
|
||||
**Detection**: Same topic, decreasing frustration and increasing success signals.
|
||||
|
||||
**Recommendation**: Identify what changed between T2 and T3. This is the successful approach.
|
||||
|
||||
### Cyclical Pattern
|
||||
|
||||
Same issue recurs periodically.
|
||||
|
||||
```
|
||||
Day 1: "Use TypeScript" (request)
|
||||
Day 5: "Remember to use TypeScript" (frustration: repetition)
|
||||
Day 10: "Again, TypeScript not JavaScript" (frustration: repetition)
|
||||
```
|
||||
|
||||
**Detection**: Same request/correction across multiple sessions with time gaps.
|
||||
|
||||
**Recommendation**: Critical memory failure. Agent is not retaining this requirement across sessions.
|
||||
Reference in New Issue
Block a user