Metadata-Version: 2.4
Name: autoverify-llm
Version: 0.1.1
Summary: A modular framework for evaluating and verifying agentic LLM outputs.
Author: Marten Panchev
Author-email: marten@aquithm.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: litellm>=1.0.0
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: scikit-learn>=1.2.0
Requires-Dist: openai>=1.0.0
Requires-Dist: pyngrok>=7.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: questionary>=2.0.0
Requires-Dist: PyYAML>=6.0.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Asymptotic Agent Evaluation Framework

The **Asymptotic Agent Evaluation Framework** is a Python-based system designed to evaluate AI agents across a global multi-dimensional input space using a linear pipeline architecture. It strategically explores the boundaries of agent reliability to map the "Safe Polytope" of operation.

## 🚀 Overview

The framework evaluates how AI agents perform as they move through different "dimensions" of complexity (e.g., urgency, ambiguity, complexity). By sampling these dimensions and generating synthetic test cases, the system maps the **Safe Polytope** — the operational volume where an agent's performance is mathematically reliable ($P_{sat} \ge 0.95$).

### Key Features
- **Adaptive Navigation:** Uses Sobol sequences for uniform expansion from stable regions and binary search to pinpoint failure boundaries.
- **Chain-of-Thought (CoT) Evaluation:** A reasoning-first judge LLM analyzes compliance before scoring, ensuring high-fidelity boundary detection.
- **Geometric Analysis:** Calculates the hyper-volume of the agent's safe operational space using N-dimensional Convex Hulls.
- **Persistence:** Automatically saves the evaluation state to JSON after each iteration for crash-resilience.
- **Linear Pipeline:** Strictly typed state progression from raw coordinates to final performance metrics.
- **Local Execution:** Evaluates code directly through AST parsing and local binding execution without WebSocket overhead.
- **Provider Agnostic:** Uses LiteLLM to dynamically connect to any LLM provider (OpenAI, Gemini, vLLM, etc.) via simple API keys.

## 🔄 The Linear Pipeline & Iterative Mutation

The evaluation framework operates as a streaming loop that systematically explores the boundary of an agent's failure. 

> [!NOTE]
> **Algorithm Update:** The framework has transitioned from Zero-Shot Synthesis to the **Iterative Mutation Pipeline** to prevent semantic drift, reduce API overhead, and achieve perfect variable isolation.

```mermaid
graph LR
    A[InputSpace] -->|Origin Prompt| B[Sequential Mutation Engine]
    B -->|Mutated Prompt| C[AgentRunner]
    C -->|Executed Vector| D[AgentOutputEvaluator]
    D -->|Evaluated Vector| E[InputSpace History]
    E -->|Evaluated Vector| F[Adaptive Navigator]
    F -->|Next Coord| B
```

### State Transitions (Core Vector Hierarchy)
The framework uses a strictly typed progression to ensure data integrity:
1. **ScenarioVector (Stage 1):** Raw N-dimensional mutation coordinates (Urgency, Flooding, Bijection).
2. **SynthesizedVector (Stage 2):** Origin prompt mutated by the coordinates.
3. **ExecutedVector (Stage 3):** Mutated prompt run through the target agent (N runs).
4. **EvaluatedVector (Stage 4):** Performance score ($P_{sat}$) evaluated against the target rubric rule.

## 🛠 Technology Stack
- **Language:** Python 3.11+
- **Data Modeling:** `pydantic`
- **Math/Geometry:** `numpy`, `scipy` (Sobol sequences, ConvexHull analysis)
- **Orchestration:** `asyncio` for parallel scenario generation and execution.
- **LLM Integration:** `litellm` for universal API connectivity.

## 📂 Project Structure

```text
├── engine/                      # Core Evaluation Logic
│   ├── domain/                  # Vector state hierarchy and Prompt features
│   ├── stages/                  # Pipeline stages (input_space, generator, evaluator, navigator, etc.)
│   ├── pipeline.py              # Central evaluation pipeline orchestrator
│   └── prompt_loader.py         # Utility for loading external LLM prompts
├── adapters/                    # External Interfaces
│   ├── llms/                    # Universal LLM Client (LiteLLM Wrapper)
│   ├── runners/                 # Target agent execution abstractions
│   └── local_bindings/          # AST scanning and local function execution
├── cli/                         # Interactive CLI Interface
│   └── app.py                   # CLI Application and real-time dashboard logic
├── prompts/                     # Externalized LLM prompt templates (.txt)
├── reporting/                   # Results Analysis
│   └── analyser.py              # Report compiler & vulnerability analyser
├── results/                     # Evaluated state & reports
└── README.md                    # Project documentation
```

## 🚦 Getting Started

### Prerequisites
- Python 3.11+
- Access to an LLM inference server or API Key (e.g., OpenAI, Gemini, or a local vLLM instance)

### Setup & Installation
1. **Environment Setup:**
   ```bash
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   ```

2. **Configuration:**
   The framework has an interactive configuration step built-in. Launch the CLI to pick a target python file, select/enter rules, configure scenarios, and supply your chosen LLM model/API Key.

### Running the Evaluation
Execute the interactive CLI:
```bash
PYTHONPATH=. python3 cli/app.py
```

The system will:
1. Prompt you for setup configurations in a CLI Wizard.
2. Initialize the search space and extract safety dimensions.
3. Spawn workers to systematically stress-test the agent rules locally.
4. Render a real-time CLI dashboard with a progress bar and events log.
5. Save evolving states to `results/` and automatically launch the final HTML results dashboard in your browser.

> [!NOTE]
> The `results/` directory is tracked by Git, but its contents (JSON states, HTML reports, logs) are ignored to keep the repository clean.

## 📈 Analysis Results
The framework outputs a hyper-volume metric (Reliability Coverage) which represents the percentage of the input space where the agent meets the reliability threshold. This allows for direct mathematical comparison between different model versions or system prompts.

## 📄 License

This project is licensed under the Apache License, Version 2.0.
See the [LICENSE](LICENSE) file for more information.
