Metadata-Version: 2.4
Name: voicetest
Version: 1
Summary: A generic test harness for voice agent workflows
Project-URL: Homepage, https://github.com/pld/voicetest
Project-URL: Repository, https://github.com/pld/voicetest
Project-URL: Issues, https://github.com/pld/voicetest/issues
Author-email: pld <50706+pld@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agent,livekit,retell,testing,vapi,voice
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: click>=8.0
Requires-Dist: dspy>=2.6
Requires-Dist: duckdb>=1.4.3
Requires-Dist: livekit-agents<2.0,>=1.0
Requires-Dist: livekit-plugins-openai<2.0,>=1.0
Requires-Dist: livekit-plugins-silero<2.0,>=1.0
Requires-Dist: pyarrow>=22.0.0
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: textual>=7.2.0
Provides-Extra: api
Requires-Dist: fastapi>=0.115; extra == 'api'
Requires-Dist: uvicorn>=0.32; extra == 'api'
Provides-Extra: dev
Requires-Dist: httpx>=0.28; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Description-Content-Type: text/markdown

# voicetest

A generic test harness for voice agent workflows. Test agents from Retell, Vapi, LiveKit, and custom sources using a unified execution and evaluation model.

## Installation

```bash
pip install voicetest
```

Or with uv:

```bash
uv add voicetest
```

## Quick Start

### Interactive Shell

```bash
# Launch interactive shell (default)
uv run voicetest

# In the shell:
> agent tests/fixtures/retell/sample_agent.json
> tests tests/fixtures/retell/sample_tests.json
> set agent_model ollama_chat/qwen2.5:0.5b
> run
```

### CLI Commands

```bash
# List available importers
voicetest importers

# Run tests against an agent definition
voicetest run --agent agent.json --tests tests.json

# Export agent as Mermaid diagram
voicetest export --agent agent.json --format mermaid

# Launch full TUI
voicetest tui --agent agent.json --tests tests.json
```

## Test Case Format

```json
[
  {
    "id": "test-1",
    "name": "Customer billing inquiry",
    "user_prompt": "## Identity\nYour name is Jane.\n\n## Goal\nGet help with a charge on your bill.\n\n## Personality\nPolite but confused.",
    "metrics": ["Agent greeted the customer", "Agent addressed the billing concern"],
    "required_nodes": ["greeting", "billing"],
    "max_turns": 10
  }
]
```

## Features

- **Multi-source import**: Retell JSON, custom Python functions (Vapi in M2)
- **Unified IR**: AgentGraph representation for any voice agent
- **Configurable LLMs**: Separate models for agent, simulator, and judge
- **DSPy-based evaluation**: LLM judges with reasoning
- **Flow validation**: Required/forbidden node constraints
- **CLI + TUI + Shell**: Multiple interface options

## LLM Configuration

Configure different models for each role using [LiteLLM format](https://docs.litellm.ai/docs/providers):

```python
from voicetest.models.test_case import RunOptions

options = RunOptions(
    agent_model="openai/gpt-4o-mini",
    simulator_model="gemini/gemini-1.5-flash",
    judge_model="anthropic/claude-3-haiku-20240307",
    max_turns=20,
)
```

Or use Ollama for local execution:

```python
options = RunOptions(
    agent_model="ollama_chat/qwen2.5:0.5b",
    simulator_model="ollama_chat/qwen2.5:0.5b",
    judge_model="ollama_chat/qwen2.5:0.5b",
)
```

In the shell:
```
> set agent_model gemini/gemini-1.5-flash
> set simulator_model ollama_chat/qwen2.5:0.5b
```

## Development

```bash
# Clone and install
git clone https://github.com/pld/voicetest
cd voicetest
uv sync --all-extras

# Run unit tests
uv run pytest tests/unit

# Run integration tests (requires Ollama with qwen2.5:0.5b)
uv run pytest tests/integration

# Lint
uv run ruff check voicetest/ tests/
```

## Project Structure

```
voicetest/
├── voicetest/           # Package
│   ├── api.py           # Core API
│   ├── cli.py           # CLI
│   ├── rest.py          # REST API server
│   ├── models/          # Pydantic models
│   ├── importers/       # Source importers (retell, custom)
│   ├── exporters/       # Graph export (mermaid, livekit)
│   ├── engine/          # Execution engine
│   ├── simulator/       # User simulation
│   ├── judges/          # Evaluation judges
│   └── tui/             # TUI and shell
├── tests/
│   ├── unit/            # Unit tests
│   └── integration/     # Integration tests (Ollama)
└── docs/
    └── M2_SPEC.md       # Future roadmap
```

## License

MIT
