==========================================================================
ABLATION STUDY: Which perturbations are sufficient for hallucination
detection?
==========================================================================

Data sources:
  fast_817_cerebras.jsonl: n=382 (3 pert types: impostor,counterfactual,paraphrase)
  full_817_cerebras.jsonl: n=31 (13 pert types, per_type_fragility)
  relabel_results.jsonl:   n=412 (LLM-judge + flex labels)
  fast ∩ relabel: 381; full ∩ relabel: 31
  fast ∩ full: 0 (datasets are disjoint)

Perturbation coverage in full_817 (n=31):
  typo             present in 31/31 rows
  synonym          present in 15/31 rows
  omit             present in 31/31 rows
  reorder          present in 31/31 rows
  tone             present in 31/31 rows
  paraphrase       present in 31/31 rows
  conformity       present in 31/31 rows
  impostor         present in 31/31 rows
  anchoring        present in 31/31 rows
  semantic         present in 1/31 rows
  negation         present in 31/31 rows
  counterfactual   present in 31/31 rows
  code_switching   present in 31/31 rows

Usable perturbations (present in all 31): 11
  ['typo', 'omit', 'reorder', 'tone', 'paraphrase', 'conformity', 'impostor', 'anchoring', 'negation', 'counterfactual', 'code_switching']

==========================================================================
PART 1: SINGLE perturbation (full_817, n=31)
==========================================================================

--- llm_label (hallucination rate = 51.61%) ---
pert            AUC     CI95                Cohen_d   
reorder         0.625   [0.423,0.820]    0.663     
negation        0.608   [0.399,0.796]    0.537     
counterfactual  0.604   [0.395,0.803]    0.306     
typo            0.571   [0.342,0.761]    0.468     
conformity      0.525   [0.311,0.724]    0.218     
omit            0.479   [0.255,0.700]    0.149     
impostor        0.479   [0.273,0.709]    -0.006    
anchoring       0.450   [0.236,0.675]    0.091     
code_switching  0.446   [0.226,0.667]    -0.329    
tone            0.417   [0.205,0.632]    0.009     
paraphrase      0.392   [0.187,0.614]    -0.201    

--- flex_label (hallucination rate = 29.03%) ---
pert            AUC     CI95                Cohen_d   
paraphrase      0.636   [0.382,0.849]    0.475     
impostor        0.616   [0.286,0.880]    0.778     
anchoring       0.616   [0.363,0.820]    0.449     
counterfactual  0.591   [0.352,0.804]    0.508     
reorder         0.571   [0.300,0.828]    0.579     
code_switching  0.571   [0.325,0.788]    0.101     
tone            0.566   [0.283,0.849]    0.371     
omit            0.556   [0.331,0.761]    0.235     
typo            0.540   [0.271,0.788]    0.486     
negation        0.535   [0.273,0.778]    0.151     
conformity      0.525   [0.277,0.762]    0.109     

==========================================================================
PART 2: PAIR combinations (mean of 2 fragilities), full_817 n=31
==========================================================================

--- llm_label: Top 10 pairs ---
  reorder        + counterfactual  AUC=0.717
  typo           + counterfactual  AUC=0.708
  negation       + counterfactual  AUC=0.654
  omit           + counterfactual  AUC=0.613
  typo           + conformity      AUC=0.604
  reorder        + negation        AUC=0.604
  reorder        + conformity      AUC=0.600
  typo           + negation        AUC=0.596
  typo           + reorder         AUC=0.588
  conformity     + negation        AUC=0.588

--- flex_label: Top 10 pairs ---
  reorder        + paraphrase      AUC=0.677
  reorder        + impostor        AUC=0.662
  reorder        + counterfactual  AUC=0.662
  paraphrase     + anchoring       AUC=0.641
  typo           + paraphrase      AUC=0.631
  anchoring      + counterfactual  AUC=0.626
  omit           + paraphrase      AUC=0.621
  tone           + paraphrase      AUC=0.621
  typo           + impostor        AUC=0.616
  paraphrase     + impostor        AUC=0.611

==========================================================================
PART 3: K-perturbation comparison (best subset of size K, mean)
==========================================================================

--- llm_label ---
K   best AUC    best subset                                                 
1   0.625       reorder
2   0.717       reorder+counterfactual
3   0.721       typo+reorder+counterfactual
4   0.688       typo+reorder+negation+counterfactual
5   0.662       typo+reorder+conformity+negation+counterfactual
7   0.613       typo+omit+reorder+conformity+impostor+...(2 more)
11  0.496       typo+omit+reorder+tone+paraphrase+...(6 more)

--- flex_label ---
K   best AUC    best subset                                                 
1   0.636       paraphrase
2   0.677       reorder+paraphrase
3   0.692       reorder+paraphrase+impostor
4   0.672       reorder+tone+paraphrase+impostor
5   0.667       typo+reorder+paraphrase+conformity+impostor
7   0.672       typo+omit+reorder+tone+paraphrase+...(2 more)
11  0.591       typo+omit+reorder+tone+paraphrase+...(6 more)

==========================================================================
PART 4: Fast dataset (n=381) — 3 perts at scale with CI
==========================================================================

--- llm_label (hallucination rate = 62.20%, n=381) ---
pert                    AUC     CI95                Cohen_d   
paraphrase              0.571   [0.513,0.626]    0.243     
adaptive_fragility      0.547   [0.486,0.603]    0.184     
impostor                0.537   [0.477,0.595]    0.171     
fragility_score (5-avg) 0.520   [0.460,0.578]    0.089     
counterfactual          0.486   [0.424,0.549]    -0.124    
adversarial_fragility   0.485   [0.426,0.544]    -0.074    

--- llm_label: PAIR combos in fast (mean of 2) ---
  impostor       + counterfactual  AUC=0.500
  impostor       + paraphrase      AUC=0.567
  counterfactual + paraphrase      AUC=0.511

--- flex_label (hallucination rate = 39.11%, n=381) ---
pert                    AUC     CI95                Cohen_d   
adaptive_fragility      0.540   [0.482,0.598]    0.175     
fragility_score (5-avg) 0.531   [0.474,0.590]    0.148     
impostor                0.531   [0.475,0.587]    0.130     
counterfactual          0.525   [0.469,0.585]    0.065     
adversarial_fragility   0.515   [0.457,0.577]    0.064     
paraphrase              0.492   [0.432,0.553]    0.019     

--- flex_label: PAIR combos in fast (mean of 2) ---
  impostor       + counterfactual  AUC=0.537
  impostor       + paraphrase      AUC=0.514
  counterfactual + paraphrase      AUC=0.519

==========================================================================
SUMMARY / TAKEAWAYS
==========================================================================

1. DATA CONSTRAINTS. Only full_817 exposes per-type fragility for all 13
   perturbations, but only n=31 rows are available — confidence intervals
   are wide. fast_817 (n=381) exposes only 3 types.

2. SINGLE-PERTURBATION AUC (full_817, LLM-judge labels):
   BEST:  reorder (0.625), negation (0.608), counterfactual (0.604)
   WORST: paraphrase (0.392), tone (0.417), code_switching (0.446)
   => No single perturbation reaches AUC > 0.70.

3. SINGLE-PERTURBATION AUC (full_817, FLEX labels):
   BEST:  paraphrase (0.636), impostor (0.616), anchoring (0.616)
   WORST: conformity (0.525), negation (0.535), typo (0.540)
   => Ordering DISAGREES with LLM-judge labels — label choice dominates!

4. PAIR COMBOS (LLM-judge): reorder+counterfactual = 0.717 AUC
   This IS the practical sweet spot: just 2 perturbations, AUC > 0.70.
   Cost-saving regime: 3 extra calls vs 5+ in current top-5 setup.

5. K-PERTURBATION SCALING (LLM-judge, best subset at each K):
   K=1: 0.625  K=2: 0.717  K=3: 0.721  K=5: 0.662  K=all(11): 0.496
   => Plateau around K=2-3. Adding more perturbations HURTS because
      uninformative ones (paraphrase, tone, code_switching) dilute signal.

6. FAST DATASET (n=381) CONFIRMS limited single-pert signal:
   - All single-perturbation AUCs are in 0.49-0.57 range (near chance).
   - adaptive_fragility aggregate ≈ 0.547 — no individual type beats it.
   - Confidence intervals span 0.5 for every single perturbation.

7. MOST INFORMATIVE PERTURBATIONS (across label regimes):
   - reorder, counterfactual, impostor, negation: appear in top tiers
     under multiple labeling schemes.
   LEAST INFORMATIVE:
   - tone, code_switching: bottom under both schemes — low cost/benefit.
   - paraphrase: high under flex, low under LLM-judge (label artifact).

8. RECOMMENDATION for deployment:
   - 2-perturbation combo (reorder + counterfactual) delivers AUC≈0.72
     on LLM-judge labels at n=31 (wide CIs; needs validation at scale).
   - This is COMPARABLE or BETTER than current 5-perturbation top-K,
     with 60% fewer extra API calls.
   - The "yuragi top-5" default is not Pareto-optimal: drop tone,
     code_switching, paraphrase; keep reorder, counterfactual, impostor,
     negation, anchoring.

9. CAVEAT: n=31 is very small; the K=2 winner could be optimistically
   biased by the best-subset search. Pre-register and validate on a held-
   out slice (ideally re-run full-13-pert scan on the fast-set subset).