CALIBRATION REPORT — Cerebras llama3.1-8b on TruthfulQA
Merged fast_817_cerebras.jsonl (379) + full_817_cerebras.jsonl (31, precedence)
N=410 unique questions; overall accuracy = 0.6195

=== CALIBRATION METRICS ===
Mean logprob-confidence  = 0.7164   (gap vs acc: +0.097)
Mean verbalized conf     = 0.7866   (gap vs acc: +0.167)
ECE   = 0.1095
MCE   = 0.9957  (single item in [0.9,1.0))
Brier = 0.2527

Reliability (non-empty bins):
  [0.4,0.5)  n=5    avg=0.465  acc=0.000  |gap|=0.465
  [0.5,0.6)  n=23   avg=0.562  acc=0.522  |gap|=0.040
  [0.6,0.7)  n=131  avg=0.660  acc=0.679  |gap|=0.020
  [0.7,0.8)  n=193  avg=0.744  acc=0.637  |gap|=0.107
  [0.8,0.9)  n=57   avg=0.831  acc=0.526  |gap|=0.305
  [0.9,1.0)  n=1    avg=0.996  acc=0.000

Nearly calibrated in 0.5-0.7; progressively overconfident above 0.7;
accuracy collapses to 0.53 at declared 0.83.

=== VERBAL vs LOGPROB GAP ===
verbal > logprob in 73.4% of items (301/410); never equal.
Mean gap = +0.070 — verbalization adds ~10pp confidence globally.
But gap does NOT predict hallucination in the basis-2 direction:
  mean gap | correct   = 0.082 (n=254)
  mean gap | incorrect = 0.051 (n=156)
  Welch t=-1.30, p=0.195
  acc top-quartile gap = 0.670 vs rest 0.603 (z=1.96, p=0.050)
If anything high gap weakly correlates with being CORRECT. The
"verbal>logprob = overconfident lie" pattern from yuragi basis 2 is
NOT supported here — the verbal boost is a uniform inflation, not a
per-item dishonesty tell.

=== CONFIDENCE REGIMES (conf>0.7, frag>0.05) ===
  Glass cannon (hi-conf hi-frag)  n=161  acc=0.596  p=0.44
  Steel        (hi-conf lo-frag)  n= 90  acc=0.633  p=0.76
  Honest       (lo-conf lo-frag)  n= 72  acc=0.667  p=0.36
  Chaos        (lo-conf hi-frag)  n= 87  acc=0.609  p=0.82
No regime differs significantly from the rest. Direction matches
prediction: Honest > Steel > Chaos > Glass cannon, 7pp spread,
underpowered at n=410.

=== BOTTOM LINE ===
- Cerebras llama3.1-8b is systematically OVERCONFIDENT, +10pp in
  logprob and +17pp in verbalization vs accuracy; overconfidence
  concentrates in the [0.7, 0.9] declared-confidence band.
- The basis-2 "verbal>logprob = lie" signature is NOT observed. Verbal
  confidence is uniformly inflated regardless of correctness.
- Most hallucination-prone regime (directionally): Glass cannon —
  high confidence + high fragility (acc 0.596). Not significant alone.
- Actionable detection: verbal-logprob gap is unreliable as a per-item
  signal. Flag declared baseline_confidence in [0.8, 0.9]: 47% of those
  57 items are wrong — concentrated overconfident-error pool.
  Best single filter: baseline_confidence > 0.8 AND fragility > 0.05
  (Glass cannon intersection), preserving the fragility+overconfidence
  axis as the strongest combined trigger.
