Calibration-breakdown hypothesis test

Hypothesis: fragility_score has solo AUC > 0.5 only on records where model
calibration is broken (|bc − empirical bin rate| is high).

Split each dataset by median |bc − empirical correct-rate in bc decile|.

variant                         stratum                   n    pos   fs_AUC               fs_CI   bc_AUC               bc_CI
TruthfulQA × is_correct         calibrated (low err)    206  0.359    0.507      [0.418, 0.592]     0.495      [0.410, 0.578] 
TruthfulQA × is_correct         broken (high err)       206  0.408    0.561      [0.480, 0.640]     0.460      [0.377, 0.543] 

TruthfulQA × llm_label          calibrated (low err)    206  0.607    0.531      [0.450, 0.616]     0.584      [0.505, 0.660]✓
TruthfulQA × llm_label          broken (high err)       206  0.621    0.514      [0.432, 0.595]     0.577      [0.499, 0.655] 

TriviaQA × is_correct           calibrated (low err)    100  0.250    0.609      [0.478, 0.727]     0.630      [0.516, 0.741]✓
TriviaQA × is_correct           broken (high err)       100  0.310    0.464      [0.339, 0.592]     0.811      [0.715, 0.892]✓

== Interpretation rule ==
  If fs_AUC CI > 0.5 ONLY in 'broken' stratum → hypothesis supported
  If fs_AUC CI > 0.5 in neither → hypothesis rejected, fragility is just noise
  If fs_AUC CI > 0.5 in both → fragility is a general detector (contradicts previous null)
