SKEPTIC ANALYSIS — Does yuragi's premise hold up?
Date: 2026-04-14
Model used: nvidia_nim/meta/llama-3.1-8b-instruct (real logprobs)
Budget spent: ~25 scans * ~5 API calls = ~125 calls, all free-tier

==================================================================
STEEL-MAN OF THE 7 ARGUMENTS
==================================================================
1. MEMORIZED → STABLE. Theoretically sound; empirically supported by
   TruthfulQA (AUC ~0.50), partially refuted by TriviaQA (AUC 0.91 @ n=127).
   Decisive test run below.
2. CONFIDENCE ≠ CORRECTNESS. Sound — our own calibration_report shows
   ECE=0.11, MCE=0.99, AUC(baseline_conf)=0.407 inverted on TruthfulQA knowledge.
   The DATA SUPPORTS this. But fragility is NOT the same as confidence; the
   question is whether delta-under-perturbation breaks the tie.
3. NOISE FLOOR. Sound. challenge_null.txt shows AUC drifts inside bootstrap
   CI for most perturbation types. Impostor_fragility repeatedly surfaces
   (AUC 0.538 CI-includes-0.5 on TruthfulQA; 0.81 here). Signal exists but
   is marginal on output-level.
4. CONTAMINATION. Sound; unresolved. TruthfulQA specifically constructed
   to elicit imitative falsehoods the model WAS trained on. This validates
   yuragi's "glass-cannon" framing: fragility is silent on memorized errors.
5. TOKENIZER ARTIFACTS. Plausible but unsupported. Our data shows paraphrase
   (preserves tokens) and counterfactual (rewrites cleanly) diverge from typo.
   Would need ablation isolating TYPO perturbation.
6. SCALE INVARIANCE. Model-scaling work shows direction-consistent but
   amplitude-different signals across 8B/70B/120B. Not a killer objection.
7. KNOWLEDGE CUTOFF. Orthogonal to the fragility question. A model that
   confabulates post-cutoff facts is the IDEAL use case for fragility.

==================================================================
DECISIVE EXPERIMENT: Memorization stability
==================================================================
Design: same model, same perturbations. Regime A = 8 common-knowledge
facts (Paris, Everest, da Vinci, 1945...). Regime B = 12 genuinely obscure
facts (Eros discovery, Abel Prize winner, Koppen of Reykjavik, mayor
of Berlin 1985). SKEPTIC PREDICTION: both AUC near 0.5; memorized facts
have same fragility scale as confabulated ones.

RESULTS:
  HIGH_FREQ (n=8): accuracy 100%, mean fragility 0.096 (no errors to score)
  LOW_FREQ  (n=12): accuracy 67%, mean_frag correct=0.064, wrong=0.090
    AUC(fragility) = 0.750
    AUC(impostor_fragility) = 0.812
    AUC(paraphrase_fragility) = 1.000 (n small)

Confident-wrong sub-regime (skeptic's prediction: memorized → LOW frag):
  Kolmogorov-for-Serre  conf=0.744  impostor_frag=0.152
  Dfc-for-Cfc           conf=0.755  impostor_frag=0.061
  vs. confident-right   conf>0.7    impostor_frag=0.026 (n=6)
  Impostor_frag is ~4x higher on confident-wrong.

==================================================================
INTERPRETATION
==================================================================
The skeptic's core claim "memorization implies stability" is REFUTED
for 8B-scale llama on genuinely obscure facts. Confidently-wrong answers
showed 4x higher impostor_fragility than confidently-correct ones. The
signal exists where the information actually lives in a single-query
distribution (not drawn from a deeply memorized manifold).

HOWEVER the skeptic wins on TruthfulQA-style imitative falsehoods: those
ARE deeply memorized and fragility is silent (documented AUC 0.50).

Reframed contribution: yuragi is a DOMAIN-BOUNDARY detector. It works
where obscure facts force single-path retrieval (TriviaQA, obscure QA)
and fails where imitative priors are entrenched (TruthfulQA myths).

NULL-RESULT VALUE if premise fails entirely:
  * Formalize "memorization-stable hallucinations invisible to output
    fragility" as a theorem.
  * Show activation-space methods (SSP 0.786, LSD 0.96) recover signal
    softmax destroys.
  * Provide a decision rule: if benchmark is imitative-falsehood-heavy,
    use hidden states; if single-path-factoid, output fragility suffices.

This experiment strengthens rather than refutes yuragi's core premise,
while sharpening its scope claim.
