Pattern for progressively refining context retrieval to solve the subagent context problem

Iterative Retrieval Pattern

Solves the "context problem" in multi-agent workflows where subagents don't know what context they need until they start working.

The Problem

Subagents are spawned with limited context. They don't know:

Which files contain relevant code
What patterns exist in the codebase
What terminology the project uses

Standard approaches fail:

Send everything: Exceeds context limits
Send nothing: Agent lacks critical information
Guess what's needed: Often wrong

The Solution: Iterative Retrieval

A 4-phase loop that progressively refines context:

1┌─────────────────────────────────────────────┐
2│                                             │
3│   ┌──────────┐      ┌──────────┐            │
4│   │ DISPATCH │─────▶│ EVALUATE │            │
5│   └──────────┘      └──────────┘            │
6│        ▲                  │                 │
7│        │                  ▼                 │
8│   ┌──────────┐      ┌──────────┐            │
9│   │   LOOP   │◀─────│  REFINE  │            │
10│   └──────────┘      └──────────┘            │
11│                                             │
12│        Max 3 cycles, then proceed           │
13└─────────────────────────────────────────────┘

Phase 1: DISPATCH

Initial broad query to gather candidate files:

1// Start with high-level intent
2const initialQuery = {
3  patterns: ['src/**/*.ts', 'lib/**/*.ts'],
4  keywords: ['authentication', 'user', 'session'],
5  excludes: ['*.test.ts', '*.spec.ts']
6};
7
8// Dispatch to retrieval agent
9const candidates = await retrieveFiles(initialQuery);

Phase 2: EVALUATE

Assess retrieved content for relevance:

1function evaluateRelevance(files, task) {
2  return files.map(file => ({
3    path: file.path,
4    relevance: scoreRelevance(file.content, task),
5    reason: explainRelevance(file.content, task),
6    missingContext: identifyGaps(file.content, task)
7  }));
8}

Scoring criteria:

High (0.8-1.0): Directly implements target functionality
Medium (0.5-0.7): Contains related patterns or types
Low (0.2-0.4): Tangentially related
None (0-0.2): Not relevant, exclude

Phase 3: REFINE

Update search criteria based on evaluation:

1function refineQuery(evaluation, previousQuery) {
2  return {
3    // Add new patterns discovered in high-relevance files
4    patterns: [...previousQuery.patterns, ...extractPatterns(evaluation)],
5
6    // Add terminology found in codebase
7    keywords: [...previousQuery.keywords, ...extractKeywords(evaluation)],
8
9    // Exclude confirmed irrelevant paths
10    excludes: [...previousQuery.excludes, ...evaluation
11      .filter(e => e.relevance < 0.2)
12      .map(e => e.path)
13    ],
14
15    // Target specific gaps
16    focusAreas: evaluation
17      .flatMap(e => e.missingContext)
18      .filter(unique)
19  };
20}

Phase 4: LOOP

Repeat with refined criteria (max 3 cycles):

1async function iterativeRetrieve(task, maxCycles = 3) {
2  let query = createInitialQuery(task);
3  let bestContext = [];
4
5  for (let cycle = 0; cycle < maxCycles; cycle++) {
6    const candidates = await retrieveFiles(query);
7    const evaluation = evaluateRelevance(candidates, task);
8
9    // Check if we have sufficient context
10    const highRelevance = evaluation.filter(e => e.relevance >= 0.7);
11    if (highRelevance.length >= 3 && !hasCriticalGaps(evaluation)) {
12      return highRelevance;
13    }
14
15    // Refine and continue
16    query = refineQuery(evaluation, query);
17    bestContext = mergeContext(bestContext, highRelevance);
18  }
19
20  return bestContext;
21}

Practical Examples

Example 1: Bug Fix Context

1Task: "Fix the authentication token expiry bug"
2
3Cycle 1:
4  DISPATCH: Search for "token", "auth", "expiry" in src/**
5  EVALUATE: Found auth.ts (0.9), tokens.ts (0.8), user.ts (0.3)
6  REFINE: Add "refresh", "jwt" keywords; exclude user.ts
7
8Cycle 2:
9  DISPATCH: Search refined terms
10  EVALUATE: Found session-manager.ts (0.95), jwt-utils.ts (0.85)
11  REFINE: Sufficient context (2 high-relevance files)
12
13Result: auth.ts, tokens.ts, session-manager.ts, jwt-utils.ts

Example 2: Feature Implementation

1Task: "Add rate limiting to API endpoints"
2
3Cycle 1:
4  DISPATCH: Search "rate", "limit", "api" in routes/**
5  EVALUATE: No matches - codebase uses "throttle" terminology
6  REFINE: Add "throttle", "middleware" keywords
7
8Cycle 2:
9  DISPATCH: Search refined terms
10  EVALUATE: Found throttle.ts (0.9), middleware/index.ts (0.7)
11  REFINE: Need router patterns
12
13Cycle 3:
14  DISPATCH: Search "router", "express" patterns
15  EVALUATE: Found router-setup.ts (0.8)
16  REFINE: Sufficient context
17
18Result: throttle.ts, middleware/index.ts, router-setup.ts

Integration with Agents

Use in agent prompts:

1When retrieving context for this task:
21. Start with broad keyword search
32. Evaluate each file's relevance (0-1 scale)
43. Identify what context is still missing
54. Refine search criteria and repeat (max 3 cycles)
65. Return files with relevance >= 0.7

Best Practices

Start broad, narrow progressively - Don't over-specify initial queries
Learn codebase terminology - First cycle often reveals naming conventions
Track what's missing - Explicit gap identification drives refinement
Stop at "good enough" - 3 high-relevance files beats 10 mediocre ones
Exclude confidently - Low-relevance files won't become relevant

The Longform Guide - Subagent orchestration section
continuous-learning skill - For patterns that improve over time
Agent definitions in ~/.claude/agents/