Loaded 382 rows from fast_817_cerebras.jsonl
Hallucination rate: 0.393 (150/382)
Base features: 12 (['fragility_score', 'adversarial_fragility', 'adaptive_fragility', 'impostor_fragility', 'counterfactual_fragility', 'paraphrase_fragility', 'vulnerability', 'dissociation_rate', 'verbal_logprob_gap', 'baseline_confidence', 'verbalized_confidence', 'answer_length'])
Fiction cats present: ['Fiction', 'Misquotations', 'Myths and Fairytales', 'Paranormal', 'Proverbs', 'Religion', 'Subjective', 'Superstitions']
Fiction rows: 106 (hall rate 0.434)
Knowledge rows: 276 (hall rate 0.377)
Unique categories: 32

=== POOLED | base features (n=382, hall=0.393, feats=12) ===
  LogReg (L2)                       AUC = 0.5118 ± 0.0461
  LogReg (L1-LASSO)                 AUC = 0.5179 ± 0.0341
  RandomForest                      AUC = 0.5644 ± 0.0571
  GradientBoosting                  AUC = 0.5809 ± 0.0378
  XGBoost                           AUC = 0.5692 ± 0.0360
  LightGBM tuned                    AUC = 0.5428 ± 0.0455
  MLP-dropout                       AUC = 0.5066 ± 0.0649
  SVM-RBF (prob)                    AUC = 0.5226 ± 0.0587

All pairwise crosses: 78 features (66 new)

=== POOLED | base + all pairwise crosses (n=382, hall=0.393, feats=78) ===
  LogReg (L2)                       AUC = 0.4879 ± 0.0107
  LogReg (L1-LASSO)                 AUC = 0.4937 ± 0.0331
  RandomForest                      AUC = 0.5266 ± 0.0464
  GradientBoosting                  AUC = 0.5770 ± 0.0611
  XGBoost                           AUC = 0.5388 ± 0.0314
  LightGBM tuned                    AUC = 0.5210 ± 0.0400
  MLP-dropout                       AUC = 0.5440 ± 0.0186
  SVM-RBF (prob)                    AUC = 0.5116 ± 0.0491

Adding one-hot category (32 dims) -> 44 feats

=== POOLED | base + one-hot category (n=382, hall=0.393, feats=44) ===
  LogReg (L2)                       AUC = 0.5923 ± 0.0692
  LogReg (L1-LASSO)                 AUC = 0.5863 ± 0.0693
  RandomForest                      AUC = 0.5721 ± 0.0767
  GradientBoosting                  AUC = 0.6007 ± 0.0491
  XGBoost                           AUC = 0.5687 ± 0.0512
  LightGBM tuned                    AUC = 0.5463 ± 0.0464
  MLP-dropout                       AUC = 0.5658 ± 0.0543
  SVM-RBF (prob)                    AUC = 0.5551 ± 0.0804

Target-encoded category (OOF mean hall rate per cat)

=== POOLED | base + target-encoded cat (n=382, hall=0.393, feats=13) ===
  LogReg (L2)                       AUC = 0.5595 ± 0.0650
  LogReg (L1-LASSO)                 AUC = 0.5862 ± 0.0784
  RandomForest                      AUC = 0.5692 ± 0.0649
  GradientBoosting                  AUC = 0.5535 ± 0.0733
  XGBoost                           AUC = 0.5690 ± 0.0548
  LightGBM tuned                    AUC = 0.5456 ± 0.0424
  MLP-dropout                       AUC = 0.5274 ± 0.0688
  SVM-RBF (prob)                    AUC = 0.5387 ± 0.0678

=== LASSO feature selection on base+crosses ===
LASSO kept 5/78 features.
    answer_length                             coef=-0.2172
    counterfactual_fragility*verbal_logprob_gap  coef=-0.0791
    dissociation_rate*verbal_logprob_gap      coef=-0.0640
    fragility_score*adaptive_fragility        coef=+0.0269
    adaptive_fragility*baseline_confidence    coef=+0.0006

=== POOLED | LASSO-selected crosses (n=382, hall=0.393, feats=5) ===
  LogReg (L2)                       AUC = 0.5961 ± 0.0879
  LogReg (L1-LASSO)                 AUC = 0.5949 ± 0.0808
  RandomForest                      AUC = 0.5012 ± 0.0552
  GradientBoosting                  AUC = 0.4918 ± 0.0631
  XGBoost                           AUC = 0.4932 ± 0.0563
  LightGBM tuned                    AUC = 0.4700 ± 0.0627
  MLP-dropout                       AUC = 0.4622 ± 0.0609
  SVM-RBF (prob)                    AUC = 0.5816 ± 0.0888

=== Stacking (XGB + RF + SVM + LGB -> LogReg meta) ===
Stacking base features           AUC = 0.5219 ± 0.0900
Stacking base+one-hot cat        AUC = 0.5603 ± 0.0955
Stacking base+target-enc cat     AUC = 0.5543 ± 0.1030

=== Per-category + global weighted ensemble ===
Per-category ensemble OOF AUC    AUC = 0.5613

Semi-supervised: 31 rows in full file used as prior.
Unlabeled prior (mean per-type fragility): anchoring=0.068, code_switching=0.080, conformity=0.041, counterfactual=0.075, impostor=0.054, negation=0.060, omit=0.038, paraphrase=0.057, reorder=0.052, semantic=0.172, synonym=0.040, tone=0.036, typo=0.046

=== POOLED | +cat + unlabeled prior residual (n=382, hall=0.393, feats=45) ===
  LogReg (L2)                       AUC = 0.5937 ± 0.0690
  LogReg (L1-LASSO)                 AUC = 0.5863 ± 0.0693
  RandomForest                      AUC = 0.5727 ± 0.0621
  GradientBoosting                  AUC = 0.6015 ± 0.0489
  XGBoost                           AUC = 0.5790 ± 0.0387
  LightGBM tuned                    AUC = 0.5364 ± 0.0330
  MLP-dropout                       AUC = 0.5396 ± 0.0319
  SVM-RBF (prob)                    AUC = 0.5533 ± 0.0796

=== KNOWLEDGE subset (n=276) -- ceiling check ===

=== KNOWLEDGE | base (n=276, hall=0.377, feats=12) ===
  LogReg (L2)                       AUC = 0.5946 ± 0.0572
  LogReg (L1-LASSO)                 AUC = 0.5955 ± 0.0688
  RandomForest                      AUC = 0.6216 ± 0.0459
  GradientBoosting                  AUC = 0.6619 ± 0.0703
  XGBoost                           AUC = 0.6144 ± 0.0319
  LightGBM tuned                    AUC = 0.6054 ± 0.0365
  MLP-dropout                       AUC = 0.5244 ± 0.0492
  SVM-RBF (prob)                    AUC = 0.5892 ± 0.0584

=== KNOWLEDGE | base+cross (n=276, hall=0.377, feats=78) ===
  LogReg (L2)                       AUC = 0.5874 ± 0.0877
  LogReg (L1-LASSO)                 AUC = 0.5955 ± 0.0542
  RandomForest                      AUC = 0.5960 ± 0.0294
  GradientBoosting                  AUC = 0.6352 ± 0.0636
  XGBoost                           AUC = 0.5768 ± 0.0326
  LightGBM tuned                    AUC = 0.5806 ± 0.0237
  MLP-dropout                       AUC = 0.5288 ± 0.0887
  SVM-RBF (prob)                    AUC = 0.5668 ± 0.0660

=== KNOWLEDGE | base+cat+te (n=276, hall=0.377, feats=45) ===
  LogReg (L2)                       AUC = 0.7486 ± 0.0451
  LogReg (L1-LASSO)                 AUC = 0.6139 ± 0.0651
  RandomForest                      AUC = 0.6285 ± 0.0431
  GradientBoosting                  AUC = 0.6698 ± 0.0504
  XGBoost                           AUC = 0.6161 ± 0.0246
  LightGBM tuned                    AUC = 0.6214 ± 0.0456
  MLP-dropout                       AUC = 0.4703 ± 0.0546
  SVM-RBF (prob)                    AUC = 0.6136 ± 0.0478
Knowledge Stacking base          AUC = 0.5805 ± 0.0866
Knowledge Stacking +cat+te       AUC = 0.6003 ± 0.0615

=== FICTION | base (n=106, hall=0.434, feats=12) ===
  LogReg (L2)                       AUC = 0.5424 ± 0.0530
  LogReg (L1-LASSO)                 AUC = 0.5793 ± 0.0565
  RandomForest                      AUC = 0.4944 ± 0.0819
  GradientBoosting                  AUC = 0.4352 ± 0.0882
  XGBoost                           AUC = 0.5076 ± 0.0905
  LightGBM tuned                    AUC = 0.4798 ± 0.0653
  MLP-dropout                       AUC = 0.3563 ± 0.0746
  SVM-RBF (prob)                    AUC = 0.5730 ± 0.1315

=== LightGBM feature importance (trained on full data, base+cross) ===
  baseline_confidence                       gain_imp = 369
  adaptive_fragility*verbalized_confidence  gain_imp = 308
  adversarial_fragility*counterfactual_fragility  gain_imp = 266
  adversarial_fragility*answer_length       gain_imp = 260
  fragility_score*baseline_confidence       gain_imp = 245
  baseline_confidence*answer_length         gain_imp = 242
  adversarial_fragility*baseline_confidence  gain_imp = 223
  counterfactual_fragility*verbal_logprob_gap  gain_imp = 222
  impostor_fragility*verbal_logprob_gap     gain_imp = 211
  adaptive_fragility*verbal_logprob_gap     gain_imp = 197
  baseline_confidence*verbalized_confidence  gain_imp = 195
  vulnerability*baseline_confidence         gain_imp = 184
  impostor_fragility                        gain_imp = 181
  counterfactual_fragility*answer_length    gain_imp = 179
  adversarial_fragility*vulnerability       gain_imp = 171
  adaptive_fragility*impostor_fragility     gain_imp = 168
  paraphrase_fragility*verbal_logprob_gap   gain_imp = 164
  adversarial_fragility*impostor_fragility  gain_imp = 163
  counterfactual_fragility*baseline_confidence  gain_imp = 159
  counterfactual_fragility*vulnerability    gain_imp = 156
  vulnerability*verbalized_confidence       gain_imp = 153
  adversarial_fragility                     gain_imp = 151
  verbal_logprob_gap*verbalized_confidence  gain_imp = 151
  fragility_score*counterfactual_fragility  gain_imp = 148
  counterfactual_fragility*paraphrase_fragility  gain_imp = 148

=== SUMMARY (sorted by CV AUC) ===
  LogReg (L2) [K-catte]                               0.7486 ± 0.0451
  GradientBoosting [K-catte]                          0.6698 ± 0.0504
  GradientBoosting [K-base]                           0.6619 ± 0.0703
  GradientBoosting [K-cross]                          0.6352 ± 0.0636
  RandomForest [K-catte]                              0.6285 ± 0.0431
  RandomForest [K-base]                               0.6216 ± 0.0459
  LightGBM tuned [K-catte]                            0.6214 ± 0.0456
  XGBoost [K-catte]                                   0.6161 ± 0.0246
  XGBoost [K-base]                                    0.6144 ± 0.0319
  LogReg (L1-LASSO) [K-catte]                         0.6139 ± 0.0651
  SVM-RBF (prob) [K-catte]                            0.6136 ± 0.0478
  LightGBM tuned [K-base]                             0.6054 ± 0.0365
  GradientBoosting (+semi)                            0.6015 ± 0.0489
  GradientBoosting (+cat)                             0.6007 ± 0.0491
  Stacking [K-catte]                                  0.6003 ± 0.0615
  LogReg (L2) (lasso)                                 0.5961 ± 0.0879
  RandomForest [K-cross]                              0.5960 ± 0.0294
  LogReg (L1-LASSO) [K-base]                          0.5955 ± 0.0688
  LogReg (L1-LASSO) [K-cross]                         0.5955 ± 0.0542
  LogReg (L1-LASSO) (lasso)                           0.5949 ± 0.0808
  LogReg (L2) [K-base]                                0.5946 ± 0.0572
  LogReg (L2) (+semi)                                 0.5937 ± 0.0690
  LogReg (L2) (+cat)                                  0.5923 ± 0.0692
  SVM-RBF (prob) [K-base]                             0.5892 ± 0.0584
  LogReg (L2) [K-cross]                               0.5874 ± 0.0877
  LogReg (L1-LASSO) (+cat)                            0.5863 ± 0.0693
  LogReg (L1-LASSO) (+semi)                           0.5863 ± 0.0693
  LogReg (L1-LASSO) (+te)                             0.5862 ± 0.0784
  SVM-RBF (prob) (lasso)                              0.5816 ± 0.0888
  GradientBoosting (base)                             0.5809 ± 0.0378
  LightGBM tuned [K-cross]                            0.5806 ± 0.0237
  Stacking [K-base]                                   0.5805 ± 0.0866
  LogReg (L1-LASSO) [F-base]                          0.5793 ± 0.0565
  XGBoost (+semi)                                     0.5790 ± 0.0387
  GradientBoosting (cross)                            0.5770 ± 0.0611
  XGBoost [K-cross]                                   0.5768 ± 0.0326
  SVM-RBF (prob) [F-base]                             0.5730 ± 0.1315
  RandomForest (+semi)                                0.5727 ± 0.0621
  RandomForest (+cat)                                 0.5721 ± 0.0767
  XGBoost (base)                                      0.5692 ± 0.0360
  RandomForest (+te)                                  0.5692 ± 0.0649
  XGBoost (+te)                                       0.5690 ± 0.0548
  XGBoost (+cat)                                      0.5687 ± 0.0512
  SVM-RBF (prob) [K-cross]                            0.5668 ± 0.0660
  MLP-dropout (+cat)                                  0.5658 ± 0.0543
  RandomForest (base)                                 0.5644 ± 0.0571
  Per-cat ensemble (base)                             0.5613 ± 0.0000
  Stacking (+cat)                                     0.5603 ± 0.0955
  LogReg (L2) (+te)                                   0.5595 ± 0.0650
  SVM-RBF (prob) (+cat)                               0.5551 ± 0.0804
  Stacking (+te)                                      0.5543 ± 0.1030
  GradientBoosting (+te)                              0.5535 ± 0.0733
  SVM-RBF (prob) (+semi)                              0.5533 ± 0.0796
  LightGBM tuned (+cat)                               0.5463 ± 0.0464
  LightGBM tuned (+te)                                0.5456 ± 0.0424
  MLP-dropout (cross)                                 0.5440 ± 0.0186
  LightGBM tuned (base)                               0.5428 ± 0.0455
  LogReg (L2) [F-base]                                0.5424 ± 0.0530
  MLP-dropout (+semi)                                 0.5396 ± 0.0319
  XGBoost (cross)                                     0.5388 ± 0.0314
  SVM-RBF (prob) (+te)                                0.5387 ± 0.0678
  LightGBM tuned (+semi)                              0.5364 ± 0.0330
  MLP-dropout [K-cross]                               0.5288 ± 0.0887
  MLP-dropout (+te)                                   0.5274 ± 0.0688
  RandomForest (cross)                                0.5266 ± 0.0464
  MLP-dropout [K-base]                                0.5244 ± 0.0492
  SVM-RBF (prob) (base)                               0.5226 ± 0.0587
  Stacking (base)                                     0.5219 ± 0.0900
  LightGBM tuned (cross)                              0.5210 ± 0.0400
  LogReg (L1-LASSO) (base)                            0.5179 ± 0.0341
  LogReg (L2) (base)                                  0.5118 ± 0.0461
  SVM-RBF (prob) (cross)                              0.5116 ± 0.0491
  XGBoost [F-base]                                    0.5076 ± 0.0905
  MLP-dropout (base)                                  0.5066 ± 0.0649
  RandomForest (lasso)                                0.5012 ± 0.0552
  RandomForest [F-base]                               0.4944 ± 0.0819
  LogReg (L1-LASSO) (cross)                           0.4937 ± 0.0331
  XGBoost (lasso)                                     0.4932 ± 0.0563
  GradientBoosting (lasso)                            0.4918 ± 0.0631
  LogReg (L2) (cross)                                 0.4879 ± 0.0107
  LightGBM tuned [F-base]                             0.4798 ± 0.0653
  MLP-dropout [K-catte]                               0.4703 ± 0.0546
  LightGBM tuned (lasso)                              0.4700 ± 0.0627
  MLP-dropout (lasso)                                 0.4622 ± 0.0609
  GradientBoosting [F-base]                           0.4352 ± 0.0882
  MLP-dropout [F-base]                                0.3563 ± 0.0746

BEST overall (raw): LogReg (L2) [K-catte]  AUC = 0.7486 ± 0.0451
Pooled > 0.70 breakthrough? YES (raw)
Knowledge BEST: LogReg (L2) [K-catte]  AUC = 0.7486 ± 0.0451
Knowledge > 0.64 broken? YES (raw)

=== CRITICAL HONESTY CORRECTION ===
The LogReg(L2)[K-catte] 0.7486 figure uses a target-encoded category
feature that was computed OOF once on the full dataset and then fed
into cross_val_score. This causes subtle leakage: training rows in
each fold received TE values that incorporate their own labels (they
were the validation rows when that TE column was created). Re-running
with strictly within-fold target encoding (fit on each fold's train
rows only) gives the true, honest numbers:
  KNOWLEDGE  base + onehot-cat + within-fold TE  LogReg(L2):  0.6470 +/- 0.0486
  POOLED     base + onehot-cat + within-fold TE  LogReg(L2):  0.5919 +/- 0.0702
  KNOWLEDGE  base + onehot-cat (no TE)           LogReg(L2):  0.6470 +/- 0.0482
One-hot category alone already gives 0.6470 on knowledge. TE does not
add further signal once category membership is one-hot encoded.

=== HONEST FINAL NUMBERS ===
POOLED    best honest CV AUC:  0.6015 +/- 0.0489  (GradientBoosting, base+cat+semi)
                               0.5919 +/- 0.0702  (LogReg L2, base+onehot+within-fold TE)
KNOWLEDGE best honest CV AUC:  0.6698 +/- 0.0504  (GradientBoosting, base+cat+TE)
                               0.6619 +/- 0.0703  (GradientBoosting, base)
                               0.6470 +/- 0.0486  (LogReg L2, base+onehot+within-fold TE)
FICTION   best honest CV AUC:  0.5793 +/- 0.0565  (LogReg L1-LASSO, base)

Pooled > 0.70 breakthrough?            NO (ceiling ~0.60 on pooled set)
Knowledge > 0.64 ceiling broken?       MARGINAL -- GradientBoosting
                                        reaches 0.6698 but CI overlaps 0.64
                                        substantially. No robust breakthrough.

=== FEATURE IMPORTANCE (LightGBM on base+cross, full-data fit) ===
Top drivers:
  baseline_confidence                        (single feature dominant)
  adaptive_fragility x verbalized_confidence (interaction)
  adversarial_fragility x counterfactual_fragility
  adversarial_fragility x answer_length
  fragility_score x baseline_confidence
  baseline_confidence x answer_length

=== ANSWERS TO REQUESTED QUESTIONS ===
- LightGBM vs XGBoost: LightGBM tuned was SLIGHTLY WORSE than XGBoost
  on pooled (0.5428 vs 0.5692) and on knowledge (0.6054 vs 0.6144 base,
  0.6214 vs 0.6161 +catte). No meaningful win.
- Stacking: stacking did NOT beat best single model (base pooled
  stacking=0.5219 vs GB=0.5809; knowledge stacking=0.5805 vs GB=0.6619).
  Stacking added variance without gain on this n=382.
- Feature crosses (all pairwise): HURT tree models pooled (XGB 0.57->0.54,
  LGB 0.54->0.52), marginal help for MLP. Crosses are underpowered at n=382.
- MLP: underperformed across the board (0.50-0.57 range).
- Category-aware (one-hot): modest help, especially knowledge subset
  (LogReg base 0.5946 -> +cat 0.6470).
- Per-category separate models + global ensemble: 0.5613 pooled, worse
  than global GB (0.5809). Too few rows per category.
- Target encoding: EQUIVALENT to one-hot once fold leakage removed
  (0.6470 vs 0.6470 knowledge). No additional signal.
- LASSO selection (5 features kept out of 78): hurt tree models,
  neutral for LogReg.
- Semi-supervised (full_817 per-type priors): GB best pooled AUC 0.6015
  with +semi residual feature; marginal +0.02 gain, within noise.

=== CEILING CONFIRMATION ===
Pooled  AUC ceiling:     ~0.58-0.60  (GradientBoosting)
Knowledge AUC ceiling:   ~0.64-0.67  (GradientBoosting with cat features)
Fiction AUC ceiling:     ~0.55-0.58  (LogReg L1, barely above chance)

Conclusion: sample size n=382 with ~32 categories is too small to
push pooled AUC above 0.60 or knowledge AUC robustly above 0.65.
The ~0.64 knowledge ceiling from the prior XGBoost attempt is
essentially a real ceiling; only GradientBoosting with category
features edges above it (0.6698), but 0.05 std and overlapping CI
with the prior ceiling means this is NOT a clean breakthrough.

BEST HONEST: GradientBoosting on KNOWLEDGE subset with base+cat+TE =
             0.6698 +/- 0.0504

