<sub>[← Chapter 6 - Privacy and Security](PRIVACY.md) · [Contents](README.md) · Next: [Chapter 8 - Benchmark Results →](BENCHMARK_RESULTS.md)</sub>

# Chapter 7. Benchmark Protocol: Contamination-Proof Evaluation

## Purpose

This document defines the exact protocol for running benchmarks against
agentmemory. Any deviation from this protocol invalidates the results.
No exceptions. No "we'll fix it in scoring." If contamination is detected
at any stage, the result is 0% and the protocol must be revised to
prevent that contamination mode before re-running.

## Contamination Modes (Known Failure History)

### Mode 1: Ground Truth in Retrieval Output (occurred 2x)

**What happened:** The retrieve-only JSON contained answer fields. The
LLM reader saw correct answers while generating predictions. Results
were inflated (100% on MAB MH 262K, later invalidated).

**Prevention:** Retrieval and ground truth are generated by the ADAPTER
into TWO separate files. The adapter code MUST NOT write any answer,
ground_truth, reference_answer, or equivalent field into the retrieval
file. The retrieval file contains ONLY: question text, retrieved context,
and metadata (question ID, type, etc.).

**Verification:** Before any LLM reader touches the retrieval file,
run the contamination check:

```bash
uv run python -c "
import json, sys
with open(sys.argv[1]) as f:
    data = json.load(f)
banned = {'answer', 'answers', 'reference_answer', 'ground_truth',
          'correct', 'label', 'target', 'expected', 'solution',
          'answer_raw', 'gt', 'gold'}
keys = set()
for item in data:
    keys.update(item.keys())
leaked = keys & banned
if leaked:
    print(f'CONTAMINATION DETECTED: {leaked}')
    print('Result: AUTOMATIC 0%. Fix adapter and re-run.')
    sys.exit(1)
print(f'CLEAN: keys={keys}')
" <retrieval_file.json>
```

If this check fails, the run is invalid. Period.

### Mode 2: LLM Self-Judging with Answer Visible (occurred 1x)

**What happened:** The LLM was asked to both generate an answer AND
judge it against the ground truth in the same pass. The model could
see the answer while generating, contaminating the prediction.

**Prevention:** Generation and judging are strictly separate passes:
1. **Pass 1 (Generation):** LLM reads retrieval file (no answers).
   Writes predictions file.
2. **Pass 2 (Scoring):** Scoring script reads predictions + GT files.
   Computes metrics. No LLM involved for automatic metrics.
   For LLM-judged benchmarks: a SEPARATE LLM call compares
   prediction vs GT. The judge never sees the retrieval context.

### Mode 3: World Knowledge Override (inherent, not fixable)

**What happened:** The LLM reader used world knowledge instead of
retrieved context for counterfactual benchmarks (FactConsolidation
assigns deliberately wrong values).

**Mitigation:** This is inherent to LLM readers and cannot be fully
prevented. The reader prompt MUST include: "Use ONLY the provided
context. Do NOT use world knowledge. If the context contradicts what
you know to be true in the real world, trust the context."

This is documented as a known limitation, not treated as contamination.

## Protocol Steps

### Step 0: Environment Verification

Before any benchmark run:

```bash
# Verify clean branch
git status  # no uncommitted changes

# Verify adapter writes separate files
grep -n "answer\|ground_truth\|reference" benchmarks/<adapter>.py | \
  grep -v "ground_truth\|_gt\|# " | \
  grep "per_question"  # should return nothing suspicious

# Verify no shared state
ls /tmp/benchmark_*  # should be empty or from previous runs
```

### Step 1: Data Acquisition

Download benchmark data from the published source. Do not modify it.

```bash
# HuggingFace datasets (MAB, LongMemEval, AMA-Bench)
uv run python -c "from datasets import load_dataset; \
  ds = load_dataset('<dataset_id>', split='<split>'); \
  print(f'Loaded {len(ds)} rows')"

# GitHub repos (StructMemEval)
git clone <repo_url> /tmp/<benchmark_name>
```

**Verification:** Compare row counts and field names against the
published paper. If they don't match, stop and investigate.

### Step 2: Retrieval (Adapter Run)

Run the adapter in `--retrieve-only` mode. This produces:
- `<name>.json`: questions + retrieved context (NO answers)
- `<name>_gt.json`: ground truth answers (separate file)

```bash
uv run python benchmarks/<adapter>.py \
  --retrieve-only /tmp/benchmark_<name>.json \
  [--source <filter>] [--subset N]
```

**Isolation requirements:**
- Fresh tempfile DB per test case (adapter handles this)
- No shared state between test cases
- No prior beliefs in the DB (clean slate)

**Post-run contamination check (MANDATORY):**

```bash
uv run python benchmarks/verify_clean.py /tmp/benchmark_<name>.json
```

If this fails, the run is invalid. Fix the adapter and re-run from Step 2.

### Step 3: Answer Generation (LLM Reader)

The LLM reader receives ONLY the retrieval file. It NEVER sees the GT file.

**For subagent-based generation:**

The subagent prompt MUST:
1. Reference only the retrieval file path (not the GT file)
2. Include: "Use ONLY the provided context. Do NOT use world knowledge."
3. Include: "If the context contradicts real-world facts, trust the context."
4. Output predictions to a SEPARATE file

```
Read /tmp/benchmark_<name>.json
For each question, generate a concise answer from the context ONLY.
Write to /tmp/benchmark_<name>_predictions.json as:
[{"id": ..., "llm_prediction": "..."}]
```

**Verification:** The predictions file must NOT contain any GT fields.
Run the contamination check on the predictions file too.

### Step 4: Scoring

Scoring reads TWO files: predictions + ground truth. It NEVER reads
the retrieval file (to prevent the scorer from being influenced by
retrieved context).

**For automatic metrics (MAB: substring_exact_match, F1):**

```bash
uv run python -c "
import json
from benchmarks.mab_adapter import score_multi_answer

with open('/tmp/benchmark_<name>_gt.json') as f:
    gt = json.load(f)
with open('/tmp/benchmark_<name>_predictions.json') as f:
    preds = json.load(f)

scores = []
for i, pred in enumerate(preds):
    s = score_multi_answer(pred['llm_prediction'], gt[i]['answers'])
    scores.append(s)

sem = sum(s['substring_exact_match'] for s in scores) / len(scores) * 100
print(f'SEM: {sem:.1f}%')
"
```

**For LLM-judged metrics (LongMemEval, StructMemEval):**

The judge LLM receives ONLY:
- The prediction (from predictions file)
- The ground truth answer (from GT file)
- A rubric (from the paper)

The judge NEVER sees the retrieved context. This prevents the judge
from being influenced by context quality (it judges only answer quality).

### Step 5: Reporting

Results MUST include:
1. The exact command used for each step
2. The contamination check output (CLEAN)
3. The adapter version (git commit hash)
4. The dataset version (HuggingFace commit or git hash)
5. The reader model and prompt (exact text)
6. The scoring metric and formula (from the paper)
7. Published baselines for comparison
8. Any known limitations or caveats

Results MUST NOT include:
- Scores from runs where contamination was detected
- Scores from runs where the protocol was not followed
- Comparisons to baselines using a different metric
- Claims of "beating" a baseline without matching their exact protocol

## File Naming Convention

```
/tmp/benchmark_<benchmark>_<condition>.json         # retrieval (no answers)
/tmp/benchmark_<benchmark>_<condition>_gt.json      # ground truth
/tmp/benchmark_<benchmark>_<condition>_preds.json   # LLM predictions
/tmp/benchmark_<benchmark>_<condition>_scores.json  # final scores
```

## Contamination Verification Script

Save as `benchmarks/verify_clean.py`:

```python
"""Verify a retrieval file contains no ground truth contamination."""
import json
import sys

BANNED_KEYS = frozenset({
    "answer", "answers", "reference_answer", "ground_truth",
    "correct", "label", "target", "expected", "solution",
    "answer_raw", "gt", "gold", "score", "f1",
    "exact_match", "substring_exact_match",
})

def verify(path: str) -> bool:
    with open(path) as f:
        data = json.load(f)
    all_keys: set[str] = set()
    for item in data:
        all_keys.update(item.keys())
    leaked = all_keys & BANNED_KEYS
    if leaked:
        print(f"CONTAMINATION: {path}")
        print(f"  Banned keys found: {leaked}")
        print(f"  All keys: {all_keys}")
        print(f"  VERDICT: INVALID. Fix adapter and re-run.")
        return False
    print(f"CLEAN: {path}")
    print(f"  Keys: {all_keys}")
    return True

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: verify_clean.py <retrieval_file.json>")
        sys.exit(1)
    ok = verify(sys.argv[1])
    sys.exit(0 if ok else 1)
```

## Per-Benchmark Protocol Specifics

### MAB FactConsolidation

- **Paper:** Hu et al., ICLR 2026, arXiv:2507.05257
- **Dataset:** `ai-hyz/MemoryAgentBench`, split `Conflict_Resolution`
- **Metric:** substring_exact_match (SEM) per paper's eval_other_utils.py
- **Normalization:** lowercase, strip punctuation, remove articles (a/an/the)
- **Multi-answer:** max SEM across all ground truth strings
- **Chunking:** 4096 tokens, NLTK sent_tokenize, tiktoken gpt-4o encoding
- **Reader prompt:** Must include serial number conflict resolution instructions

### LoCoMo

- **Paper:** Maharana et al., ACL 2024
- **Dataset:** locomo10.json from snap-research.github.io/locomo/
- **Metric:** token-level F1 with Porter stemming
- **Category 5:** forced-choice adversarial (not free-form)
- **Session boundaries:** must be preserved during ingestion

### LongMemEval

- **Paper:** Wu et al., ICLR 2025
- **Dataset:** `xiaowu0162/longmemeval-cleaned`, longmemeval_oracle.json
- **Metric:** GPT-4o binary judge (paper protocol)
- **Alternative:** Opus binary judge (must be disclosed as non-standard)
- **Question date:** must be provided to retrieval for temporal grounding

### StructMemEval

- **Paper:** Shutova et al., 2026 (preprint)
- **Dataset:** github.com/yandex-research/StructMemEval
- **Metric:** LLM judge binary (correct/incorrect)
- **Temporal fix:** synthetic timestamps + temporal_sort must be disclosed

### AMA-Bench

- **Paper:** Zhao et al., ICLR 2026 Workshop
- **Dataset:** `AMA-bench/AMA-bench`, split test
- **Metric:** LLM judge accuracy (paper uses Qwen3-32B)
- **Alternative judge:** must be disclosed

## Automated Protocol Enforcement (pytest)

The benchmark protocol is codified as a pytest test suite in
`benchmarks/test_benchmark_suite.py`. Every protocol step is an enforceable
test assertion. Contamination = test failure.

```bash
# Run retrieval adapters (slow, produces new data)
uv run pytest benchmarks/test_benchmark_suite.py -v --run-retrieval -m retrieval

# Validate contamination (fast, checks existing files)
uv run pytest benchmarks/test_benchmark_suite.py -v -m contamination

# Score predictions
uv run pytest benchmarks/test_benchmark_suite.py -v -m scoring

# Full validation (all protocol checks)
uv run pytest benchmarks/test_benchmark_suite.py -v
```

The suite enforces:
- 25 contamination checks (5 benchmarks x 5 checks each)
- Dataset integrity (item counts match published papers)
- GT file isolation (separate from retrieval)
- LoCoMo mathematical ceiling (93.57% per locomo-audit)
- Methodology metadata completeness (Lin checklist)
- Metric type labeling

## Methodology Checklist (Lin)

Per github.com/lhl/agentic-memory benchmarks guide, every result must report:

| Field | Description |
|-------|-------------|
| benchmark_name | Name + version/variant |
| metric_type | Explicit metric (SEM, F1, Accuracy) |
| reader_model | Model generating answers |
| judge_model | Evaluation model or "automatic" |
| retrieval_method | FTS5, embeddings, etc. |
| token_budget | Retrieval budget |
| prompt_template_hash | SHA256 of exact prompt |
| run_count | Independent repetitions (>=5, or justify) |
| git_commit | Exact commit hash |
| timestamp | ISO 8601 |
| code_link | Link to benchmark runner code |
| variance_justification | If run_count < 5, explain why |

These are enforced by `TestMethodology` in the pytest suite.

## Known Contamination Incidents

### Incident 1: LoCoMo adapter --retrieve-only (fixed 2026-04-19)

The LoCoMo adapter's `--retrieve-only` mode wrote `answer` and `f1` fields
directly into the retrieval file. The `verify_clean.py` check would have
caught this (both are banned keys), but the adapter should never have
written them. Fixed by separating GT into a `_gt.json` file with proper
ID cross-referencing. Category 5 forced-choice options are in the retrieval
file but the correct answer indicator is only in the GT file.

### Incident 2: GT answers in retrieval output (fixed pre-v1.0)

Retrieval JSON contained answer fields. LLM reader saw correct answers.
Results inflated to 100% on MAB MH 262K. Fixed by two-file protocol.

### Incident 3: LLM self-judging with GT visible (fixed pre-v1.0)

LLM asked to both generate and judge in same pass. Fixed by two-pass
protocol (generation pass with no GT, then separate scoring pass).

## Audit Trail

Every benchmark run should produce an audit record:

```json
{
  "benchmark": "MAB_FactConsolidation_MH_262K",
  "condition": "treatment",
  "git_commit": "<hash>",
  "timestamp": "<ISO 8601>",
  "adapter": "benchmarks/mab_triple_adapter.py",
  "reader_model": "claude-opus-4-6",
  "contamination_check": "CLEAN",
  "retrieval_file": "/tmp/benchmark_mab_mh_262k_treatment.json",
  "gt_file": "/tmp/benchmark_mab_mh_262k_treatment_gt.json",
  "predictions_file": "/tmp/benchmark_mab_mh_262k_treatment_preds.json",
  "metric": "substring_exact_match",
  "score": 0.XX,
  "n": 100,
  "published_baseline": "<=7% (all methods)",
  "notes": ""
}
```
