# Classification Gap Analysis: Offline vs LLM

**Date:** 2026-04-11
**Method:** Ran both classifiers on all 4 validation repos (project-d, project-f, project-h, project-k). Sampled 100 sentences per repo, plus 50-sentence deep-dive on project-d. Hand-labeled 20 sentences as ground truth for test suite.

---

## Summary

LLM classification is strictly superior. The offline classifier never catches anything the LLM misses. The relationship is one-directional: offline produces a superset of LLM's persist decisions, and the extra items are all noise.

There is no complementary gap. Running both classifiers and merging results would only re-introduce the false positives that LLM correctly filters out.

---

## Question 1: What does offline catch that LLM misses?

Nothing. Across all 4 repos (5,595 + 6,905 + 1,374 + 220 = 14,094 total sentences), the offline classifier produced **zero** cases where it correctly persisted a sentence that LLM marked ephemeral. The offline_misses count is 0 in every repo.

This makes structural sense: the offline classifier defaults everything to persist=True (except questions starting with "what/how/why..." and ending with "?"). Since LLM already persists everything worth keeping, offline's "persist everything" strategy adds nothing.

## Question 2: What does LLM catch that offline misses?

LLM correctly identifies non-persist content that offline cannot:

| Content Type | Example | Offline | LLM |
|---|---|---|---|
| Section headings | "Files Created/Modified" | FACT, persist=True | META, persist=False |
| Date metadata | "**Date:** 2026-03-27" | FACT, persist=True | META, persist=False |
| YAML fragments | "patterns_established:" | FACT, persist=True | META, persist=False |
| Task IDs | "WRK-01 -- GSD slash commands ported" | FACT, persist=True | META, persist=False |
| Coordination steps | "Run Ansible playbook -- UCI config applied" | CORRECTION, persist=True | COORDINATION, persist=False |
| Checklist items | "[x] Homebridge log shows no errors" | CORRECTION, persist=True | COORDINATION, persist=False |

The offline classifier has no way to distinguish a section heading from a factual statement. LLM understands document structure.

## Question 3: Are there sentence types where offline is actually better?

No. The only place offline performs comparably is:

1. **Questions** -- both detect questions correctly (offline uses prefix + "?" heuristic, which works).
2. **Requirements with "must"** -- offline keyword match works for genuine requirements like "All code must use strict typing." But it also fires on headings like "Requirements Validated" and commit messages like "compile Ray requirements lockfiles."

For requirements specifically:
- project-d: offline found 247 REQUIREMENTs, LLM agreed on 199 of them. The 48 extras are false positives (headings, commit messages containing the word "requirements").
- Offline REQUIREMENT precision vs LLM: ~81%. Decent but not better than LLM.

For corrections, offline is much worse:
- project-d: offline found 805 CORRECTIONs, LLM agreed on 32. That is 4% precision.
- The correction detector fires on imperative verbs ("add", "use", "run", "fix"), negation ("not", "never"), and emphasis ("!"), which appear constantly in commit messages and documentation.

## Question 4: Should both classifiers run and merge? Or is LLM strictly superior?

**LLM is strictly superior. Do not merge.**

Merging would mean: if either classifier says persist, persist. Since offline says persist on ~95% of all text, merging is equivalent to running offline alone. The merge would undo all the gains from LLM classification:

| Repo | LLM beliefs | Offline beliefs | Merge (union) would be |
|---|---|---|---|
| project-d | 2,379 | 2,883 | ~2,883 (back to offline) |
| project-h | 311 | 1,374 | ~1,374 (back to offline) |
| project-k | 64 | 220 | ~220 (back to offline) |

The locked belief problem is even worse. Merging would restore 80-96% false lock rates because offline's correction detector fires on common words in non-correction text.

---

## Offline Classifier Failure Modes

### 1. Default-to-FACT (the main problem)

The offline classifier has no "not worth storing" bucket. If text does not match any keyword pattern and is not a question, it defaults to FACT with persist=True. This means every heading, metadata line, YAML fragment, checklist item, and structural text gets stored as a "fact."

### 2. Correction detector over-fires (the lock problem)

The correction detector (`detect_correction`) uses signal-counting with a threshold of 1. Any single signal triggers CORRECTION. Signals include:

- `imperative`: starts with "use", "add", "remove", "run", "keep", "stop" -- fires on commit messages ("add IoT checklist"), coordination ("run the playbook"), and documentation ("use the following config")
- `always_never`: contains "always", "never", "period" -- fires on factual statements ("the upload was never triggered because...")
- `negation`: contains "not ", "no " -- fires on almost any sentence with negation
- `directive`: contains "must", "require" -- fires on commit messages mentioning requirements

With threshold=1, a single pattern match produces a locked belief. The result: 805 "corrections" in project-d (4% actual precision), creating hundreds of permanent false locks.

### 3. Keyword ambiguity

The word "must" in "All code must use strict typing" is a real requirement. The word "must" in "compile Ray requirements lockfiles" is not. The word "never" in "Never skip the pre-flight checklist" is a real directive. The word "never" in "the upload was never triggered" is just a factual description. Offline cannot distinguish these.

---

## Quantified Impact

| Metric | Offline | LLM | LLM Advantage |
|---|---|---|---|
| Persist accuracy (project-d sample) | 76% | 99% (Exp 50) | +23pp |
| False persist rate | 24-76% by repo | ~1% | 24-75x fewer false persists |
| False lock rate | 79-96% by repo | <5% estimated | 16-19x fewer false locks |
| CORRECTION precision | 4-11% | ~95% (Exp 50) | 9-24x better |
| REQUIREMENT precision | ~81% | ~98% | modest improvement |
| QUESTION detection | ~100% | ~100% | tie |
| Type accuracy | 7-54% by repo | ~99% | massive improvement |

---

## Recommendation

1. **LLM classification should be the only production classifier.** The offline classifier adds no value that LLM does not already provide.
2. **Keep offline as a degraded-mode fallback** for when no API key is available or the API is unreachable. It is better than nothing (it persists everything, so recall is 100%), but it should never run alongside LLM.
3. **Do not attempt to merge results.** The merge is equivalent to offline-only, which defeats the purpose.
4. **The correction detector should not be used for classification.** Its 4-11% precision means it creates 10-25x more false locks than real ones. If offline mode must exist, the correction detector should require >= 2 signals instead of >= 1 to reduce false lock rate.

---

## Test Coverage

`tests/test_classification_comparison.py` -- 8 tests covering:
- Offline persist accuracy (confirms < 75%)
- Offline false correction rate (confirms >= 3 false corrections in 20 sentences)
- Offline never catches what LLM misses (confirms on ground truth)
- Offline question detection (confirms it works)
- LLM parse accuracy (confirms 100% with correct responses)
- LLM correction precision (confirms no false corrections)
- Head-to-head disagreement catalog (confirms LLM wins majority)
- Offline over-persist rate (confirms >= 50% of non-persist items are false-persisted)

---

## References

- ONBOARDING_VALIDATION_RESULTS.md -- 4-repo lock reduction numbers (79-96%)
- CORRECTION_BURDEN_REFRAME.md -- false positive cost analysis
- experiments/exp50_llm_classification.py -- 99% vs 36% accuracy finding
- src/agentmemory/classification.py -- both classifier implementations
- src/agentmemory/correction_detection.py -- correction detector signal logic
