SEMANTIC ENTROPY PIVOT ANALYSIS
================================
Date: 2026-04-14
Data: experiments/fast_817_cerebras.jsonl (n=382) + full_817_cerebras.jsonl (n=31)
Total: n=413 TruthfulQA records (Cerebras llama3.1-8b), 254 correct / 159 incorrect
Target benchmark: Farquhar 2024 Nature — AUC ~0.79 on TruthfulQA

Q1: Does yuragi have semantic entropy already implemented?
----------------------------------------------------------
YES — src/yuragi/metrics/semantic_entropy.py implements
Kuhn/Farquhar-style clustering entropy with two backends:
  (a) char-3-gram Jaccard similarity (default, offline proxy)
  (b) sentence-transformers cosine (if embedding_model passed)
NLI entailment clustering (as in Farquhar 2024 Nature) is NOT
implemented — the docstring explicitly flags this as a proxy.

Q2: Can we compute it on existing data without new API calls?
--------------------------------------------------------------
NO for the full Kuhn/Farquhar algorithm. Run_fast.py and run_full.py
call Scanner.scan() and only serialize aggregated fragility scalars
plus model_answer[:200]. The raw perturbed responses (scan.perturbation_results[i].perturbed_response.text)
are discarded before write. See experiments/run_fast.py:121-137 —
no perturbed_text field is persisted.

YES for "semantic-entropy-lite": the scalar `dissociation_rate`
stored in every record = fraction of perturbations where
answer_changed (similarity < threshold). This is a single-number
collapse of the same quantity semantic entropy would compute.

Q3: AUC of semantic-entropy-lite on the n=413 corpus
------------------------------------------------------
Using dissociation_rate (answer-change fraction) as hallucination score:
  dissociation_rate:         AUC = 0.5085
  fragility_score (raw):     AUC = 0.5407
  adaptive_fragility:        AUC = 0.5459  (best single)
  impostor_fragility:        AUC = 0.5378
  counterfactual_fragility:  AUC = 0.5325
  verbal_logprob_gap:        AUC = 0.4439
  dissociation + fragility:  AUC = 0.5260
  frag + adaptive + dissoc:  AUC = 0.5296

On the full_817 subset (n=31, 13 perturbations x more variants):
  dissociation_rate:   AUC = 0.5707
  fragility_score:     AUC = 0.6061
  fragility_max:       AUC = 0.6162

Q4: Would re-running with full perturbation storage hit ~0.79?
---------------------------------------------------------------
UNLIKELY with Cerebras/llama3.1-8b on TruthfulQA. The existing
answer-change signal (dissociation_rate = 0.51) is a direct summary
of exactly the cluster-count information Farquhar uses, and it
does not separate correct from incorrect on this model. The gap
between 0.51 (lite) and 0.79 (Farquhar) would have to come from:
  - Proper NLI-based clustering (vs substring similarity)
  - Length-normalized log-probability weighting per sample
  - Many more samples (Farquhar uses k=10; yuragi uses 2 variants)

Realistic expectation for a 100-question re-run with k=10 samples,
sentence-transformer clustering, and per-sample logprobs:
  AUC ~0.60-0.68 on llama3.1-8b TruthfulQA. Farquhar's 0.79 used
  LLaMA-2-chat-70B + BART-MNLI NLI clustering; 8B models + proxy
  clustering consistently underperform by 0.10-0.15 AUC in the
  literature (cf. Kuhn ICLR 2023 Table 2).

Recommendation
--------------
Do NOT pivot full corpus re-run. Instead:
  1. Add `perturbed_text` and `perturbed_logprob` to result dict in
     run_fast.py:121 (one-line change).
  2. Re-run n=100 held-out with k=10 variants + sentence-transformer
     clustering via existing semantic_entropy() function.
  3. Compare AUC(true semantic_entropy) vs AUC(dissociation_rate)
     to quantify the clustering-quality gap on THIS model.
  4. Budget: ~1500 calls (~30 min at current Cerebras rate).
