# Experiment 6: Temporal Coherence in Multi-Hop Chain Resolution

## Date: 2026-04-16

## Hypothesis

Exploring ALL historical entity values at each hop (not just the latest
serial) will recover the 34 GT-unreachable multi-hop questions identified
in the deep failure analysis, improving MH SEM from 58% toward the 96%
GT-reachable ceiling.

## Method

### Retrieval Change

Added `resolve_all()` to EntityIndex: returns every historical value per
(entity, property), not just the highest serial. Added
`multi_hop_retrieve_temporal()` that branches through all values at each
hop with a breadth cap of 30.

### Reader Protocol

10 parallel subagent readers (Opus), each handling 10 questions. System
prompt instructs counterfactual resolution with highest-serial-wins rule.

### Two Versions

- **v1**: No serial numbers in output lines. Reader had no way to identify
  which facts were newer. Result: 1% SEM (catastrophic regression).
- **v2**: Serial numbers prepended to each fact line (e.g.,
  "18321. Italy capital: Duluth"). Reader can now see which facts are
  newer. Result: 60% SEM (+2pp).

## Results

### GT-Reachable (Retrieval Quality)

| Condition | GT-Reachable | Delta |
|---|---|---|
| Baseline (latest-only) | 62/100 | - |
| Temporal (all values) | 96/100 | +34 |

### Context Size

| Metric | Baseline | Temporal | Ratio |
|---|---|---|---|
| Mean chars | 433 | 2,916 | 6.7x |
| Mean fact lines | 11 | 75 | 6.8x |
| Max chars | 1,688 | 7,422 | 4.4x |

### Reader SEM (Opus)

| Condition | SEM | Delta |
|---|---|---|
| Baseline (Exp 5 treatment_a) | 58% | - |
| Temporal v1 (no serials) | 1% | -57pp |
| Temporal v2 (with serials) | 60% | +2pp |

### Reader Accuracy Breakdown (v2)

| Question Set | N | Reader Correct | Accuracy |
|---|---|---|---|
| Previously reachable | 62 | 58 | 94% |
| Newly reachable | 34 | 2 | 6% |
| All | 100 | 60 | 60% |

## Analysis

### Why 6% on Newly-Reachable Questions

The 34 newly-reachable questions have GT chains that follow non-latest
serial values at intermediate hops. Example (Q2):

```
Question: "The continent where the country of citizenship of the spouse
of the author of American Pastoral is related to is what?"

Our chain (latest serials):
  American Pastoral -> F. Scott Fitzgerald (10793)
  F. Scott Fitzgerald married_to -> Carl Lumbly (16800, latest)
  Carl Lumbly citizen_of -> Colombia (9097)
  Colombia continent -> Antarctica (7238)

GT chain (non-latest intermediate):
  American Pastoral -> F. Scott Fitzgerald (10793)
  F. Scott Fitzgerald married_to -> Zelda Fitzgerald (8453, NOT latest)
  Zelda Fitzgerald citizen_of -> Australia (10172)
  Australia continent -> Oceania (2465)
```

Both chains are valid given the data. The reader correctly follows the
"highest serial wins" instruction, which always picks the latest-serial
chain. The GT chain uses a non-latest spouse value at hop 2.

### The Fundamental Issue

The MAB multi-hop GT was likely generated by resolving chains at a
specific point in time (when the question was created), not at the
globally-latest state. Our "highest serial wins" resolution is correct
for single-hop queries but produces different chains than GT for
multi-hop queries when intermediate entities have been updated since
the question was generated.

### Retrieval vs Reader

The retrieval problem is fully solved: 96% of GT answers appear in
context. The remaining gap is entirely reader chain resolution. The
reader independently resolves each hop to the latest serial, which is
the same chain the baseline (latest-only) retrieval produces. The
additional historical values in the temporal retrieval are unused by
the reader.

## Implications

### For the Benchmark

To close the 36pp gap (60% -> 96% ceiling), we would need either:
1. A mechanical chain resolver that enumerates all valid chains
2. A reader prompt that traces ALL possible chains, not just latest
3. Understanding how the GT was generated (temporal snapshot strategy)

Option 1 removes the reader variable entirely but risks gaming the
benchmark. Option 2 is more general but may not work with subagent
readers. Option 3 requires benchmark-specific knowledge.

### For Production (agentmemory)

The temporal branching mechanism (resolve_all + multi-hop) is useful
for the production system: when queried about a topic that has changed
over time, returning all historical values lets the consumer see the
full evolution. This maps to the real-world use case where a user wants
to understand how a decision evolved, not just its current state.

The serial-in-output finding is also production-relevant: when returning
beliefs with temporal metadata, always include timestamps or version
markers so the consumer can resolve conflicts.

### For Research

The reader accuracy split (94% on old, 6% on new) cleanly separates
retrieval quality from reader quality. Future work on the reader
(better prompting, chain-of-thought, or mechanical resolution) can be
tested against the 34 newly-reachable questions as an isolated test set.

## Files

| File | Change |
|---|---|
| benchmarks/mab_entity_index_adapter.py | +resolve_all(), +multi_hop_retrieve_temporal(), +serial output |
| benchmarks/mab_reader.py | New: LLM reader via Anthropic API |
| benchmarks/exp6_score.py | New: scoring script |

## Remaining Questions (4 GT-Unreachable)

Q6, Q32, Q37, Q86: entity extraction failures. The primary entity
cannot be parsed from the question. These are Priority 2 (entity
extraction improvement), not temporal coherence issues.
