Metadata-Version: 2.4
Name: yuragi
Version: 0.4.3
Summary: LLM Confidence Fragility Analyzer — Measure how fragile your AI's confidence really is
Project-URL: Homepage, https://github.com/hinanohart/yuragi
Project-URL: Documentation, https://hinanohart.github.io/yuragi
Project-URL: Repository, https://github.com/hinanohart/yuragi
Project-URL: Issues, https://github.com/hinanohart/yuragi/issues
Project-URL: Changelog, https://github.com/hinanohart/yuragi/blob/main/CHANGELOG.md
Author: hinanohart
License-Expression: MIT
License-File: LICENSE
Keywords: ai-safety,confidence,confidence-calibration,evaluation,explainability,fragility,hallucination-detection,llm,llm-evaluation,model-testing,neural-network,nlp,perturbation-testing,prompt-engineering,robustness,stress-testing,uncertainty-quantification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: click>=8.0.0
Requires-Dist: litellm>=1.40.0
Requires-Dist: rich>=13.0.0
Provides-Extra: all
Requires-Dist: datasets>=2.14; extra == 'all'
Requires-Dist: matplotlib>=3.7.0; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: plotly>=5.15.0; extra == 'all'
Requires-Dist: scipy>=1.10.0; extra == 'all'
Requires-Dist: sentence-transformers>=2.2; extra == 'all'
Provides-Extra: benchmarks
Requires-Dist: datasets>=2.14; extra == 'benchmarks'
Provides-Extra: dev
Requires-Dist: bandit>=1.7.0; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0; extra == 'docs'
Requires-Dist: mkdocs>=1.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.24; extra == 'docs'
Provides-Extra: semantic
Requires-Dist: numpy>=1.24.0; extra == 'semantic'
Requires-Dist: scipy>=1.10.0; extra == 'semantic'
Requires-Dist: sentence-transformers>=2.2; extra == 'semantic'
Provides-Extra: stats
Requires-Dist: numpy>=1.24.0; extra == 'stats'
Requires-Dist: scipy>=1.10.0; extra == 'stats'
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7.0; extra == 'viz'
Requires-Dist: numpy>=1.24.0; extra == 'viz'
Requires-Dist: plotly>=5.15.0; extra == 'viz'
Description-Content-Type: text/markdown

# yuragi — Measure how unstable your LLM's confidence really is

<p align="center"><a href="README.ja.md">日本語</a></p>

[![PyPI version](https://img.shields.io/pypi/v/yuragi)](https://pypi.org/project/yuragi/)
[![PyPI downloads](https://img.shields.io/pypi/dm/yuragi)](https://pypi.org/project/yuragi/)
[![CI](https://github.com/hinanohart/yuragi/actions/workflows/ci.yml/badge.svg)](https://github.com/hinanohart/yuragi/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)

## Instant Demo

No API key needed:

```bash
pip install yuragi
yuragi demo
```

<p align="center"><img src="docs/demo_en.svg" alt="yuragi demo output" width="700"></p>

---

## What It Does

yuragi measures **confidence fragility**: how much a model's certainty shifts when you rephrase the same question. It generates 13 perturbation variants of your prompt (typos, tone changes, paraphrases, authority framing), calls your model, and compares the confidence across responses. When the answer text stays the same but confidence moves, that's fragility — a property of the prompt wording, not the model's knowledge.

---

## Install

```bash
pip install yuragi
```

Optional extras:

```bash
pip install yuragi[viz]        # heatmap / reliability diagram output
pip install yuragi[semantic]   # sentence-transformers for semantic entropy
pip install yuragi[stats]      # numpy/scipy for statistical tests
pip install yuragi[all]        # everything
```

Supports any [litellm](https://docs.litellm.ai/docs/providers)-compatible provider — OpenAI, Anthropic, Google, local Ollama, and 100+ others.

---

## Python API

```python
from yuragi import Scanner

result = Scanner(model="cerebras/llama-3.1-8b-instruct").scan("Is quantum computing practical?")
print(result.fragility_score)    # 0.056
print(result.dissociation_rate)  # 0.07 — answer same, confidence shifted
```

<details>
<summary>Psychology experiments / Trilayer / Semantic Entropy API</summary>

```python
from yuragi.experiments.registry import get_experiment
from yuragi.experiments.runner import run_experiment

result = run_experiment(get_experiment("asch"), model="ollama/llama3.2", num_samples=5)
print(result.avg_delta)        # average confidence change
print(result.effect_confirmed) # True if max_delta >= 0.15
```

```python
from yuragi.analysis.trilayer import measure_trilayer

result = measure_trilayer("Is AI dangerous?", model="ollama/llama3.2")
print(result.logprob_confidence)     # Layer 1: token probability
print(result.sampling_confidence)    # Layer 2: behavioral consistency
print(result.verbalized_confidence)  # Layer 3: self-reported
print(result.internal_conflict)      # True if discrepancy > 0.2
```

```python
from yuragi.metrics.semantic_entropy import semantic_entropy
h_sem = semantic_entropy(samples=["Paris", "It's Paris.", "The capital is Paris"])
```

</details>

---

## CLI Quickstart

```bash
# Scan a prompt for fragility
yuragi scan "Is quantum computing practical?" --model cerebras/llama-3.1-8b-instruct

# Find the single weakest word
yuragi find-weakness "Explain the theory of relativity" --model ollama/llama3.2

# Run a psychology stress test
yuragi experiment asch --model ollama/llama3.2
```

---

## Use Cases

**CI/CD regression detection** — catch fragility regressions before they reach production:

```bash
yuragi check prompts.txt --baseline baseline.json --model gpt-4o-mini
```

**Fragility-aware routing** — route each prompt to the model that answers most stably:

```bash
yuragi route "What causes inflation?" --models gpt-4o-mini,ollama/llama3.2,cerebras/llama-3.1-8b
```

**Abstention guard** — refuse to answer when fragility exceeds safety thresholds (medical: < 0.03, safety: < 0.02):

```bash
yuragi guard "What medication should I take?" --domain medical --model gpt-4o-mini
```

**Model selection** — find the best model for your use case by fragility profile:

```bash
yuragi recommend --use-case factual --models gpt-4o-mini,ollama/llama3.2 --budget medium
```

**Automated red teaming** — discover model weaknesses across all 13 perturbation types:

```bash
yuragi red-team prompts.txt --model gpt-4o-mini --output report.json
```

---

## Research Results

Real-data empirical results on llama-3.1-8B-Instruct (Cerebras + NVIDIA NIM endpoints, April 2026):

### 🎯 Primary finding: Ensemble hallucination detection on TruthfulQA

| Metric | Value |
|:------|:------|
| Dataset | TruthfulQA, n=412 LLM-judge-labeled questions |
| Method | LogReg over 105 engineered features (13 fragility + interactions + inversions) |
| AUC-ROC | **0.7304** |
| 95% CI (5-fold CV) | **[0.6776, 0.7792]** |
| Brier score | 0.219 (calibrated) |

Single-signal fragility saturates at AUC ≈ 0.62 due to decoder-stochasticity noise floor; the ensemble crosses AUC 0.70 because LLM-judge labels remove label noise AND 105 interacting features explicitly exploit the sign-inversion (see below). Source: [`experiments/ensemble_final.txt`](https://github.com/hinanohart/yuragi/blob/main/experiments/ensemble_final.txt)

### 🔄 Secondary finding: Confidence sign-inversion on 8B

| Dataset | Raw baseline_confidence AUC | Inverted AUC |
|:-------|:----------------------------|:-------------|
| TruthfulQA (n=412) | 0.407 | 0.593 |
| **TriviaQA (n=200)** | **0.252** | **0.748** |
| Multi-judge majority (n=200) | 0.365 | 0.635 |

On llama-3.1-8B, **higher self-reported confidence correlates with _higher_ hallucination probability** — the opposite sign that temperature scaling, abstention thresholds, and RLHF calibration objectives assume.

Scope: single-model (llama-3.1-8B), two-provider (Cerebras + NVIDIA NIM). **Cross-family replication on Mistral-7B / Qwen2-7B is the load-bearing next experiment**, not yet completed. Treat as a hypothesis with convergent evidence rather than a validated claim. See [`paper/revolutionary_reframe.md`](https://github.com/hinanohart/yuragi/blob/main/paper/revolutionary_reframe.md).

### 🗺️ Domain boundary

Fragility is not a universal hallucination detector. On 413 TruthfulQA questions categorised by axis:

| Axis | Example | yuragi AUC |
|:-----|:--------|:-----------|
| **Single-path factoids** (obscure trivia) | "Who discovered argon?" | **~0.75** (works) |
| **Imitative falsehoods** (well-known misconceptions) | "What happens if you break a mirror?" | **~0.50** (fails) |

Fragility measures _uncertainty_, not _incorrectness_. When a model is confidently-wrong from training-data imitation (TruthfulQA's fiction/myth axis, 40% of the benchmark), perturbations do not shake that confidence. See [`paper/domain_boundary_section.md`](https://github.com/hinanohart/yuragi/blob/main/paper/domain_boundary_section.md).

### ⚠️ Reliability audit

Test–retest Pearson correlation on paired scans (same prompt, different seed):

| Signal | r | Recommendation |
|:-------|:--|:---------------|
| baseline_confidence | 0.88 | ✓ Primary |
| paraphrase_fragility | 0.80 | ✓ Primary |
| adaptive_fragility | 0.78 | ✓ Primary |
| impostor_fragility | 0.70 | ○ Supporting |
| fragility_score (aggregate) | 0.64 | ○ Supporting |
| **counterfactual_fragility** | **0.18** | **✗ Noise-dominated, do not use** |

### 📝 Supporting findings

- **Confidence tracks text, not knowledge** — When answer text is identical (Jaccard=1.0), max confidence shift is 0.021 (below noise floor). When text differs, confidence shifts up to 0.528. See [`RESEARCH.md`](https://github.com/hinanohart/yuragi/blob/main/RESEARCH.md).
- **Fragility scaling trend** — Across 5 models (1.2B to 22B active parameters), mean fragility follows F(N) = a/√N + b with R²=0.987. Nonzero asymptote suggests irreducible fragility at scale. See [`RESEARCH.md`](https://github.com/hinanohart/yuragi/blob/main/RESEARCH.md).

### 📉 Honest limitations

- **Statistical**: Single robust claim survives Bonferroni correction. Observed AUC of 0.50–0.55 on single perturbation types is noise; only the ensemble and the inverted-confidence signal are well-powered.
- **Generalisation**: One model, one hardware pair, one language. Cross-family, cross-domain, cross-language replication pending.
- **Theoretical ceiling**: SSP (AUC 0.786) measures the same perturbation at hidden states and outperforms us by ~0.05. LSD (AUC 0.96) uses full activation geometry. Output-level methods (ours) are bounded by `I(correct; h_internal)`.
- 4 limitations audits + 7 meta-audits committed to [`experiments/`](https://github.com/hinanohart/yuragi/tree/main/experiments/) for honest scope disclosure.

---

## Integration

**pandas — score a DataFrame of prompts:**

```python
import pandas as pd
from yuragi import Scanner

scanner = Scanner(model="gpt-4o-mini")
df["fragility"] = df["prompt"].apply(lambda p: scanner.scan(p).fragility_score)
```

**pytest — assert stability in tests:**

```python
from yuragi import Scanner

def test_prompt_stability():
    result = Scanner(model="gpt-4o-mini").scan("What is the capital of France?")
    assert result.fragility_score < 0.05, f"Fragility too high: {result.fragility_score}"
```

**GitHub Actions — CI/CD fragility gate:**

```yaml
- name: Check fragility regression
  run: yuragi check prompts.txt --baseline baseline.json --model gpt-4o-mini
```

A [reusable GitHub Actions workflow](.github/workflows/yuragi-check.yml) is included.

---

## Full CLI Reference

<details>
<summary>All 18 commands</summary>

| Command | Description |
|---------|-------------|
| `demo` | Run pre-computed demo (no API key needed) |
| `scan` | Full fragility scan (13 perturbation types) |
| `find-weakness` | Find the single word that most collapses confidence |
| `experiment` | Run a psychology template (11 types) |
| `compare-models` | Multi-model fragility comparison with heatmap |
| `check` | CI/CD fragility regression detection |
| `route` | Fragility-aware multi-model routing |
| `guard` | Abstention system for high-stakes domains |
| `recommend` | Model selection based on fragility profiles |
| `red-team` | Automated vulnerability discovery |
| `trajectory` | Track confidence across a prompt sequence |
| `stats` | Statistical analysis (Cohen's d, Wilcoxon, bootstrap CI) |
| `trilayer` | Measure confidence via 3 simultaneous methods |
| `profile` | Fragility profile: CCI / RE / NLS |
| `linguistic` | Analyze linguistic confidence markers (hedges, assertiveness) |
| `volatility` | Financial-engineering metrics (VIX, Sharpe ratio) for confidence |
| `phase-map` | Map confidence phase transitions across parameter space |
| `compare` | Compare two scan results (A/B test) |
| `export` | Export scan results to CSV/JSON |

</details>

---

## Research

Key discoveries, empirical data, and scaling trends: [RESEARCH.md](https://github.com/hinanohart/yuragi/blob/main/RESEARCH.md)

White-box layer entropy experiments:

```bash
python experiments/whitebox_design.py --exp entropy_trajectory
python experiments/whitebox_design.py --exp critical_layer_heatmap
python experiments/whitebox_design.py --exp cpu  # lightweight demo
```

See also [docs/related_work.md](https://github.com/hinanohart/yuragi/blob/main/docs/related_work.md) for comparison with lm-polygraph, SelfCheckGPT, PromptBench, CCPS, SYCON-Bench, TRUTH DECAY, SycEval, and FRS.

---

## Paper

ICML 2026 MI Workshop submission:

> "From Black-Box Fragility to White-Box Dynamics: Layer-Resolved Entropy Signatures of Confidence Perturbation in LLMs"

Source: [`paper/icml2026_mi/`](https://github.com/hinanohart/yuragi/tree/main/paper/icml2026_mi/). Three contributions: (1) conditional two-phase entropy signature, (2) perturbation-type-specific layer pathways, (3) confidence stability scaling trend F(N) = a/√N + b.

## Citation

```bibtex
@software{yuragi2025,
  title  = {yuragi: Confidence Fragility in Neural Networks},
  author = {hinanohart},
  year   = {2026},
  url    = {https://github.com/hinanohart/yuragi}
}
```

## Contributing / License

Issues and PRs welcome. See [CONTRIBUTING.md](https://github.com/hinanohart/yuragi/blob/main/CONTRIBUTING.md).

Known limitations: [KNOWN_LIMITATIONS.md](https://github.com/hinanohart/yuragi/blob/main/KNOWN_LIMITATIONS.md). Raw benchmark data: [`docs/bench/real/`](https://github.com/hinanohart/yuragi/tree/main/docs/bench/real/). Theory and metric definitions: [`docs/theory.md`](https://github.com/hinanohart/yuragi/blob/main/docs/theory.md).

MIT
