# Wonder: Reason Fail-Fast on Insufficient Knowledge Graph Depth

## Query

How should mem:reason handle insufficient knowledge graph depth? The design doc proposes fail-fast detection, structured gap signals, and auto-routing to wonder for graph expansion. Open questions: What threshold defines "insufficient"? How to prevent wonder-then-reason loops? Should the pivot be opt-in or default? How do other systems handle this? What quality gate prevents wonder from polluting the graph with noise?

## Hypothesis

A pre-check after seed retrieval but before path finding can reliably distinguish "insufficient knowledge" (wrong domain) from "genuine uncertainty" (relevant but conflicting beliefs), enabling early abort with a structured pivot to wonder.

## Methodology

- 4 parallel research agents
- Agent 1: Prior art (SOAR, active learning, query abandonment, MYCIN)
- Agent 2: Fail-fast threshold design (scoring function, concrete numbers)
- Agent 3: Loop prevention and quality gates
- Agent 4: Architecture mapping to agentmemory codebase
- Sources: agentmemory belief graph (75 beliefs retrieved), web research (10 academic sources)

## Prior Beliefs

- aba0d7cb0bb6 (92%): SOAR creates substates on insufficient knowledge -- meta-reasoning context
- aacbb4bb40c4 (96%): mem:reason must always document outputs to files
- c5b87747c2e3 (95%): Fail-fast, quick iteration, deterministic execution
- eabe34e224fb (78%): Multi-hop requires reasoning across the graph, fundamentally harder
- e0a20e265bcb (78%): Graph topology predicts required traversal depth

## New Findings

### Agent 1: Prior Art

**SOAR**: Impasse detection is structural and binary, not probabilistic. Four trigger conditions: no operators proposed, tie, conflict, or operator selected but no rules fire. No numeric threshold -- it's "can vs cannot make progress." Substates recurse.

**Active learning**: Epistemic uncertainty (reducible, model lacks knowledge) vs aleatoric uncertainty (irreducible, data is ambiguous). Maps directly: "too few relevant beliefs" = epistemic (pivot to wonder), "relevant beliefs disagree" = aleatoric (report UNCERTAIN).

**IR query abandonment**: RAG systems use relevance score thresholds as primary signal. If no retrieved chunk exceeds threshold, return "I don't know." Session-level tracking (prior failed queries) improves accuracy by ~22%.

**MYCIN**: Certainty factor threshold at 0.2. Rules below threshold simply don't fire. No explicit "I can't reason" mode, but in practice insufficient evidence meant no conclusion was drawn.

Sources: SOAR Architecture Manual, Laird 2022 (arXiv:2205.03854), Springer epistemic/aleatoric (10.1007/s10994-021-05946-3), SIGIR 2014 query abandonment, MYCIN (Buchanan-Shortliffe 1984 Ch.5).

### Agent 2: Fail-Fast Metrics

**Graph Coverage Score (GCS)**:
```
GCS = 0.6 * top_relevance + 0.3 * connectivity - 0.1 * impasse_ratio
```
- top_relevance: mean of top-3 seed relevance scores
- connectivity: path_count / max_possible_paths
- impasse_ratio: impasse_count / seed_count

**Thresholds**:
- GCS < 0.15: Fail-fast. Seeds are off-topic.
- GCS 0.15-0.35: Shallow pass. Report low confidence, suggest auto-wonder.
- GCS > 0.35: Full reasoning.

**Failure case validation**: GitHub migration query scored GCS = 0.02 (top_relevance ~0.10, connectivity 0.0, impasse_ratio 0.4). Would be caught immediately.

**Key insight**: Two-axis check distinguishes insufficient knowledge from genuine uncertainty:
- Relevance >= 0.45 AND conflicts present = UNCERTAIN (genuine)
- Relevance < 0.45 = INSUFFICIENT_KNOWLEDGE (fail-fast)

### Agent 3: Loop Prevention

**Max retries**: 2 wonder-reason cycles (3 total reason attempts).

**Query broadening strategy**:
- Cycle 1: Hypernym expansion ("GitHub account migration" -> "git remote authentication and identity")
- Cycle 2: Domain broadening ("git configuration and workflow documentation")

**Lateral thinking detector**: `if count(beliefs matching tool_name) < 5: wonder about the tool itself first`. This catches "you don't know enough about the tool to reason about the problem."

**Quality gate**: Combined relevance-novelty score >= 0.45. Relevance floor 0.3, novelty floor (1 - max_similarity_to_existing) > 0.15.

**TTL**: Auto-wonder speculative beliefs get 72-hour TTL. Promoted to permanent if feedback("used") received after successful reason.

**Kill switch**: If cycle 1 wonder returns zero beliefs passing quality gate, skip cycle 2 entirely.

### Agent 4: Architecture Mapping

**New module**: `src/agentmemory/reason_pivot.py` (~100 lines)
- `check_coverage(store, query, seeds, threshold, min_seeds) -> CoverageCheck`
- `auto_pivot_to_wonder(store, query, coverage, max_agents, quality_floor) -> WonderResult | None`

**Insertion point**: After seed retrieval (Step 1) but before `store.find_consequence_paths()` in cli.py line ~2009.

**Loop tracking**: `reason_pivot_log` table (query_hash PK, last_pivot_ts, pivot_count).

**API surface**:
- CLI: `agentmemory reason --auto-pivot --coverage-threshold 0.4`
- MCP: Optional params on existing reason skill

**Settings keys**:
```
reason.auto_pivot:          false
reason.coverage_threshold:  0.4
reason.pivot_cooldown:      3600
reason.pivot_max_retries:   2
reason.quality_floor:       0.5
```

**No changes to wonder TTL/GC** -- speculative beliefs follow existing lifecycle.

## Gaps and Contradictions

1. **Embedding dependency**: Agent 3's quality gate uses cosine_similarity on embeddings. agentmemory has no embeddings. Substitute FTS5 term overlap or HRR similarity.
2. **Domain extraction**: The lateral thinking detector needs to extract the tool/domain from the query. L0 structural analysis (90.5% accuracy) is a starting point.
3. **Opt-in vs default**: Start opt-in, promote to default after data.

## Validation Results (2026-04-20)

Tested GCS assumptions against real belief database (65K beliefs, 4b0f8c37972f).

**Finding 1: FTS5 hit count is the strongest discriminator.**
- "bayesian confidence scoring": 2 FTS5 hits (well-covered)
- "correction detection accuracy": 3 FTS5 hits (well-covered)
- "deploy workflow publish script": 1 FTS5 hit (well-covered)
- "GitHub account migration": 0 FTS5 hits (poorly-covered)
- "kubernetes cluster autoscaling": 0 FTS5 hits (poorly-covered)

Zero FTS5 hits perfectly separated known-good from known-bad queries in this test.

**Finding 2: belief.confidence is NOT query relevance.**
The GCS formula used `top_relevance` from belief confidence scores, but confidence reflects Bayesian trust (how often the belief helped), not how relevant it is to the current query. All queries scored similar confidence (~0.375) regardless of domain match. The retrieval pipeline does not currently expose ranking scores on belief objects.

**Finding 3: Non-FTS retrieval layers fill the budget with noise when FTS5 finds nothing.**
The "GitHub account migration" query returned 9 seeds -- all from entity expansion and intention clustering pulling in unrelated beliefs about "migration" (paper trading migration, directory migration). The layers are doing their job (find related concepts) but without FTS5 anchoring, they produce cross-domain noise.

**Revised design (v1):**
- Fail-fast trigger: FTS5 returns 0 hits. No scoring function needed.
- Structured output: report FTS5 miss, list which non-FTS layers contributed (so user knows the graph has tangentially related content but nothing on-topic).
- v2: Expose retrieval ranking scores on belief objects, implement full GCS with weighted scores.

## Proposed Experiments

- **Exp 99**: Implement FTS5-based fail-fast. Run against 20 queries (10 well-covered, 10 poorly-covered). Measure false positive rate (well-covered queries with 0 FTS5 hits that shouldn't fail-fast).
- **Exp 100**: Implement auto-pivot with 1-cycle limit. Run against the poorly-covered queries. Measure: did wonder produce beliefs that would have changed the FTS5 hit count on retry?
- **Exp 101**: Measure quality gate effectiveness. For each wonder output, compare graph state before/after and score retrieval quality change.

## Validation Round 2: Lateral Thinking Detector (2026-04-20)

Tested 4 approaches for detecting "insufficient foundational knowledge":

### v1: Domain belief count (< 5 beliefs about the tool)
- Accuracy: 50% (6/12)
- Problem: Tool names don't match belief vocabulary. "bayesian" returns 0 hits despite hundreds of Bayesian-related beliefs. "sqlite" returns 1 hit.

### v2: FTS5 hit count as binary gate
- Accuracy: 57% (8/14)
- Problem: FTS5 matches individual words across 65K beliefs. "kubernetes cluster autoscaling" gets 4 hits from "cluster" and "policies" matching unrelated beliefs.

### v3: Term overlap ratio (fraction of query terms in best FTS5 match)
- Accuracy: 36% (5/14) at best threshold
- Problem: Beliefs are distilled statements, not query echoes. Same vocabulary gap problem agentmemory was built to solve.

### v4: Retrieval pipeline scores (top-3 mean from scored_beliefs dict)
- Accuracy: 43% (6/14) at best threshold
- Problem: Non-FTS layers (entity expansion, intention clustering) fill the budget with tangentially related beliefs. Scores don't penalize domain mismatch. "kubernetes cluster autoscaling" scores 0.401, nearly identical to well-covered queries (~0.34-0.39).

### Key insight

The retrieval pipeline is designed to maximize recall (find something even with zero vocabulary overlap). This is exactly what makes it bad at detecting "I have nothing relevant." The system that bridges vocabulary gaps is the same system that masks ignorance.

### Possible solutions (untested)

1. **FTS5-only pre-check**: Run ONLY the FTS5 layer (no expansion layers) and check hit count. If FTS5 finds 0 hits, the query has zero vocabulary overlap with the graph. The expansion layers would need to do ALL the work, which correlates with low relevance.
2. **Layer attribution**: Track which retrieval layer produced each belief. If >80% of results came from expansion layers (not FTS5), flag as potentially insufficient.
3. **Contrastive scoring**: Score the query against both the retrieved beliefs AND a random sample of beliefs. If the retrieved set scores only marginally better than random, the retrieval didn't find anything meaningfully relevant.
4. **Two-stage: FTS5 gate + contrastive check**: Combine approaches 1 and 3 for both vocabulary-overlap and relevance signals.

## Validation Round 3: Layer Attribution + Contrastive (2026-04-20)

### v5: Layer attribution (FTS5% of total results)
- Accuracy: 57% at best threshold
- Problem: FTS5 matches individual words even for off-topic queries. "GitHub account migration" gets 9 FTS5 hits (100% FTS5 ratio) because "migration", "account", "organization" each match unrelated beliefs.

### v6: Contrastive scoring (retrieved vs random sample lift)
- Accuracy: 57% at best threshold (1.5x lift)
- Problem: Random sample scores vary between runs (0.14-0.27), making lift ratio unstable. The scored retrieval pipeline treats off-topic queries nearly identically to on-topic ones.

### Conclusion across all 6 approaches

No single metric at the retrieval layer reliably discriminates well-covered from poorly-covered queries. The vocabulary gap bridging that makes agentmemory useful also prevents text-based signals from detecting ignorance.

**The detection must happen at the reasoning layer, not the retrieval layer.** The SOAR model was right: you don't detect insufficient knowledge before reasoning. You detect it when reasoning fails to make progress -- 0 paths found between seeds, all impasses with no resolution, no coherent evidence chain. That's a structural signal from the reason pipeline itself.

### Revised architecture

1. Remove the pre-check from the design. Don't try to detect insufficiency before reasoning.
2. Instead, instrument the reason pipeline to classify its own output:
   - 0 paths + majority impasses = INSUFFICIENT_KNOWLEDGE (not UNCERTAIN)
   - >0 paths + conflicting evidence = UNCERTAIN (genuine)
   - >0 paths + consistent evidence = CONFIRMED or REFUTED
3. The auto-wonder pivot triggers on INSUFFICIENT_KNOWLEDGE verdict, not on a pre-check.
4. This is cheaper to implement (no new scoring, just classify the existing reason output) and more accurate (uses the full reasoning result, not a proxy metric).

## Validation Round 4: Structural Signals (2026-04-20)

### Hypotheses tested

- H1: INSUFFICIENT cases have 0 consequence paths. UNCERTAIN cases have >= 1.
- H3: INSUFFICIENT cases have 0 edges between seeds.
- H4: Connectivity ratio separates the two classes.

### Results

**All hypotheses invalidated.** Both INSUFFICIENT and well-covered queries show identical structural signals: 0 edges between seeds, 0 paths, 0 shared neighbors. The graph doesn't discriminate at all.

### Root cause

The graph has 9,002 edges across 7,646 beliefs, but 99.7% are structural (SENTENCE_IN_FILE: 4,491, WITHIN_SECTION: 4,483). Only 28 semantic edges (CITES). Zero SUPERSEDES, CONTRADICTS, SUPPORTS, RELATES_TO edges.

The graph records document structure (which file a belief came from), not semantic relationships (how beliefs relate to each other). Consequence path finding returns 0 for ALL queries because no semantic paths exist.

### Implications

1. The INSUFFICIENT vs UNCERTAIN distinction cannot be made on graph structure because there IS no meaningful graph structure for semantic reasoning.
2. The reason pipeline's path-finding is essentially a no-op on this graph. It was designed for a denser semantic graph that doesn't exist in production.
3. Before implementing fail-fast detection, the graph needs semantic edges. The onboarding pipeline builds structural edges but doesn't infer semantic relationships (SUPPORTS, CONTRADICTS, SUPERSEDES) between beliefs.
4. This is a prerequisite blocker: build semantic graph edges first, then the INSUFFICIENT/UNCERTAIN distinction becomes testable.

### Updated status

The reason fail-fast feature is blocked on semantic edge generation. The correct sequence:
1. Build semantic edge inference (Exp 99 prerequisite)
2. Validate that semantic edges create distinguishable path patterns
3. Then implement INSUFFICIENT vs UNCERTAIN classification
4. Then implement auto-wonder pivot
