Loaded 379 records from /home/runza/oss/yuragi/experiments/fast_817_cerebras.jsonl
Label distribution (1=hallucination): 147 / 379 (38.8% hallucinations)
Base feature matrix: (379, 11)

=== Single-feature AUC (base features) ===
  verbal_logprob_gap            AUC = 0.5500
  answer_length                 AUC = 0.5394
  adaptive_fragility            AUC = 0.5360
  baseline_confidence           AUC = 0.5323
  fragility_score               AUC = 0.5295
  impostor_fragility            AUC = 0.5281
  counterfactual_fragility      AUC = 0.5275
  vulnerability                 AUC = 0.5202
  adversarial_fragility         AUC = 0.5164
  paraphrase_fragility          AUC = 0.5114
  dissociation_rate             AUC = 0.5026

=== Models on BASE features ===
LogisticRegression  5-fold CV AUC = 0.5011 ± 0.0306
RandomForest(100)   5-fold CV AUC = 0.5445 ± 0.0705
XGBoost             5-fold CV AUC = 0.5609 ± 0.0470

=== Models on BASE + INTERACTION features (27 feats) ===
LogReg+inter        5-fold CV AUC = 0.5336 ± 0.0234
LogReg-L1+inter     5-fold CV AUC = 0.5357 ± 0.0145
RandomForest+inter  5-fold CV AUC = 0.5719 ± 0.0499
XGBoost+inter       5-fold CV AUC = 0.5801 ± 0.0349

=== Single-feature AUC (interaction terms) ===
  conf_x_frag                   AUC = 0.5282
  gap_x_conf                    AUC = 0.5585
  conf_x_diss                   AUC = 0.5050
  conf_x_adv                    AUC = 0.5152
  conf_x_para                   AUC = 0.5079
  frag_x_diss                   AUC = 0.5074
  gap_x_frag                    AUC = 0.5506
  overconf_lie_frag             AUC = 0.5282
  overconf_lie_adv              AUC = 0.5152
  overconf_lie_vuln             AUC = 0.5174
  low_conf_hi_frag              AUC = 0.5025
  conf_minus_frag               AUC = 0.5291
  conf_minus_gap                AUC = 0.5353
  hi_frag_lo_conf               AUC = 0.5014
  hi_conf_hi_frag               AUC = 0.5219
  ratio_gap_conf                AUC = 0.5439

=== Feature importance ===
-- RandomForest importance --
  baseline_confidence           0.0626
  impostor_fragility            0.0504
  counterfactual_fragility      0.0495
  conf_minus_frag               0.0489
  gap_x_frag                    0.0486
  adaptive_fragility            0.0485
  conf_minus_gap                0.0474
  adversarial_fragility         0.0471
  conf_x_para                   0.0444
  low_conf_hi_frag              0.0439
  paraphrase_fragility          0.0436
  gap_x_conf                    0.0436
  overconf_lie_adv              0.0434
  conf_x_adv                    0.0426
  vulnerability                 0.0425
  ratio_gap_conf                0.0413
  verbal_logprob_gap            0.0411
  conf_x_frag                   0.0407
  overconf_lie_vuln             0.0406
  overconf_lie_frag             0.0393
  fragility_score               0.0374
  frag_x_diss                   0.0150
  conf_x_diss                   0.0135
  answer_length                 0.0135
  dissociation_rate             0.0075
  hi_conf_hi_frag               0.0017
  hi_frag_lo_conf               0.0015
-- LogReg |coef| (standardized) --
  gap_x_conf                    coef = -1.0321
  ratio_gap_conf                coef = +1.0083
  answer_length                 coef = -0.2970
  conf_minus_gap                coef = +0.2725
  dissociation_rate             coef = -0.2524
  vulnerability                 coef = -0.2147
  conf_x_para                   coef = -0.2105
  conf_x_frag                   coef = +0.2007
  overconf_lie_frag             coef = +0.2007
  adaptive_fragility            coef = +0.1515
  low_conf_hi_frag              coef = -0.1422
  conf_x_adv                    coef = -0.1159
  overconf_lie_adv              coef = -0.1159
  frag_x_diss                   coef = +0.1119
  fragility_score               coef = +0.1111
  counterfactual_fragility      coef = -0.1005
  gap_x_frag                    coef = -0.0973
  verbal_logprob_gap            coef = -0.0899
  impostor_fragility            coef = -0.0687
  conf_x_diss                   coef = +0.0652
  conf_minus_frag               coef = -0.0568
  hi_frag_lo_conf               coef = +0.0377
  hi_conf_hi_frag               coef = +0.0367
  adversarial_fragility         coef = +0.0202
  paraphrase_fragility          coef = +0.0199
  overconf_lie_vuln             coef = -0.0181
  baseline_confidence           coef = -0.0116
-- XGBoost importance (gain) --
  dissociation_rate             0.0834
  hi_frag_lo_conf               0.0562
  conf_minus_gap                0.0549
  baseline_confidence           0.0491
  conf_x_para                   0.0481
  conf_x_adv                    0.0469
  gap_x_frag                    0.0462
  counterfactual_fragility      0.0460
  conf_x_frag                   0.0447
  verbal_logprob_gap            0.0435
  adversarial_fragility         0.0407
  ratio_gap_conf                0.0400
  overconf_lie_vuln             0.0388
  answer_length                 0.0383
  frag_x_diss                   0.0375
  conf_minus_frag               0.0365
  vulnerability                 0.0353
  gap_x_conf                    0.0342
  adaptive_fragility            0.0329
  low_conf_hi_frag              0.0317
  impostor_fragility            0.0316
  conf_x_diss                   0.0309
  paraphrase_fragility          0.0275
  fragility_score               0.0254
  overconf_lie_frag             0.0000
  overconf_lie_adv              0.0000
  hi_conf_hi_frag               0.0000

=== Knowledge vs Fiction split ===
Fiction rows:   106 (hall rate 0.434)
Knowledge rows: 273 (hall rate 0.370)
-- FICTION (n=106, hall=0.434, cv=5) --
   LogReg   AUC = 0.5194 ± 0.0687
   RF       AUC = 0.5457 ± 0.0980
   XGB      AUC = 0.5424 ± 0.1386
-- KNOWLEDGE (n=273, hall=0.370, cv=5) --
   LogReg   AUC = 0.5722 ± 0.0893
   RF       AUC = 0.6287 ± 0.0534
   XGB      AUC = 0.6363 ± 0.0399

=== SUMMARY ===
Best model: XGB (KNOWLEDGE)  CV AUC = 0.6363 ± 0.0399
Does composite cross 0.70? NO
All results (sorted):
  XGB (KNOWLEDGE)               0.6363 ± 0.0399
  RF (KNOWLEDGE)                0.6287 ± 0.0534
  XGB (base+int)                0.5801 ± 0.0349
  LogReg (KNOWLEDGE)            0.5722 ± 0.0893
  RF (base+int)                 0.5719 ± 0.0499
  XGBoost (base)                0.5609 ± 0.0470
  RF (FICTION)                  0.5457 ± 0.0980
  RF (base)                     0.5445 ± 0.0705
  XGB (FICTION)                 0.5424 ± 0.1386
  LogReg-L1 (base+int)          0.5357 ± 0.0145
  LogReg (base+int)             0.5336 ± 0.0234
  LogReg (FICTION)              0.5194 ± 0.0687
  LogReg (base)                 0.5011 ± 0.0306
