Power analysis + raw data-quality audit

== Part A: perturbation-signal health across all experiments/*.jsonl ==
Broken = >80% records with all perturbation keys = 0, or fragility_score std = 0.

Total files scanned = 42
Broken files        = 9
Usable (pert signal variable) = 20

Broken files (do NOT replicate conclusions on these):
  batch_groq_meta-llama_llama-4-scout-17b-16e-instruct_300.jsonl  n=    1  frag_zero=  0.0%  frag_mean=0.0594  (frag_std=0.0e+00)
  halueval_cerebras.jsonl                                  n=   46  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)
  missing_axis.jsonl                                       n=   31  frag_zero=  0.0%  frag_mean=0.0000  (frag_std=0.0e+00)
  multi_groq_llama-3.3-70b-versatile.jsonl                 n=    4  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)
  multi_groq_meta-llama_llama-4-scout-17b-16e-instruct.jsonl  n=    5  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)
  multi_groq_qwen_qwen3-32b.jsonl                          n=    2  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)
  multi_sambanova_DeepSeek-V3-0324.jsonl                   n=    2  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)
  nli_se_cache_n50.jsonl                                   n=   26  frag_zero=  0.0%  frag_mean=0.0000  (frag_std=0.0e+00)
  qwen35_122b.jsonl                                        n=    4  frag_zero=100.0%  frag_mean=0.0000  (all-zero pert 100%, frag_std=0.0e+00)

Top usable files (largest n with healthy perturbation signal):
  fast_817_cerebras.jsonl                                  n=  382  frag_mean=0.0659  frag_std=0.0338  frag_zero=0.0%
  triviaqa_scale_n200.jsonl                                n=  200  frag_mean=0.0813  frag_std=0.0388  frag_zero=0.0%
  slow_cohere_command-a-03-2025.jsonl                      n=  100  frag_mean=0.0989  frag_std=0.0603  frag_zero=0.0%
  slow_mistral_mistral-small-latest.jsonl                  n=  100  frag_mean=0.0717  frag_std=0.0418  frag_zero=0.0%
  slow_nvidia_nim_meta_llama-3.1-8b-instruct.jsonl         n=  100  frag_mean=0.1498  frag_std=0.1140  frag_zero=0.0%
  multi_model_voting.jsonl                                 n=   91  frag_mean=0.1448  frag_std=0.1218  frag_zero=11.0%
  nq_open_pilot.jsonl                                      n=   75  frag_mean=0.0743  frag_std=0.0366  frag_zero=0.0%
  cross_model_probe.jsonl                                  n=   40  frag_mean=0.0584  frag_std=0.0546  frag_zero=15.0%
  model_scaling.jsonl                                      n=   33  frag_mean=0.0800  frag_std=0.0444  frag_zero=0.0%
  full_817_cerebras.jsonl                                  n=   31  frag_mean=0.0550  frag_std=0.0267  frag_zero=0.0%
  pilot_all13_n30.jsonl                                    n=   30  frag_mean=0.0538  frag_std=0.0271  frag_zero=0.0%
  selfcheckgpt.jsonl                                       n=   30  frag_mean=0.0691  frag_std=0.0365  frag_zero=0.0%

== Part B: power analysis (TruthfulQA n=412, resampled) ==
Δ = A4 − A1 (full − no-pert) under Logistic Regression OOF.
min detectable Δ at 80% power ≈ 2.8 × std(Δ_bootstrap).

 target n    Δ_mean    Δ_std                   95%CI   CI width   min_Δ_80%
       50   -0.0383   0.0562  [-0.1505,+0.0690]     0.2195      0.1574
      100   -0.0351   0.0374  [-0.1067,+0.0404]     0.1470      0.1046
      200   -0.0360   0.0217  [-0.0798,+0.0052]     0.0850      0.0608
      400   -0.0369   0.0036  [-0.0443,-0.0296]     0.0147      0.0101
      800   -0.0359   0.0150  [-0.0644,-0.0068]     0.0577      0.0419
     1600   -0.0367   0.0109  [-0.0581,-0.0159]     0.0422      0.0304
     3200   -0.0366   0.0074  [-0.0506,-0.0217]     0.0290      0.0206

== Implications ==
  Observed |Δ| across experiments ≤ 0.07. At n=412 min detectable Δ ≈ 0.010.
  If the real effect is |Δ| ≤ 0.03, need n ≈ 400 to detect at 80% power.

Caveats:
  * Resampling from one dataset understates between-dataset variance.
  * Real n=800 or n=1600 experiment would also shift |Δ| if the true
    signal scales with dataset size.
  * Broken data files must be excluded before any replication claim.
