Metadata-Version: 2.5
Name: sdvm
Version: 0.1.3
Summary: Python SDK for the Synthetic Data Vending Machine (SDVM) API
Project-URL: Homepage, https://sdvm.ai
Project-URL: Documentation, https://docs.sdvm.ai
Author-email: SDVM <info@sdvm.ai>
License: MIT
Keywords: ai,llm,nlp,synthetic-data,training-data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27.0
Provides-Extra: dev
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Provides-Extra: examples
Requires-Dist: datasets>=2.0; extra == 'examples'
Requires-Dist: python-dotenv>=1.0; extra == 'examples'
Description-Content-Type: text/markdown

# sdvm

Official Python SDK for [SDVM](https://sdvm.ai) — the Synthetic Data Vending Machine.

SDVM **audits** datasets for LLM training and **fixes** what it finds — grammar, coherence,
mislabeled answers, weak distractors, length shortcuts — while **never making your data worse**.
Feed it noisy text or messy multiple-choice items; get back the defects it found, the fixes it
made, and an honest flag on anything it couldn't fully resolve.

## Installation

```bash
uv pip install sdvm        # or: uv add sdvm  (in a uv-managed project)
```

<sub>Prefer pip? `pip install sdvm` works too. To run the `examples/` scripts, add the extra:
`uv pip install "sdvm[examples]"`.</sub>

## Quick Start

```python
from sdvm import Fixer
from sdvm.types import TextSample

fixer = Fixer(api_key="sdvm_...")

data = [
    TextSample(text="omg i just got the job i cant believe it im literally shaking rn"),
    TextSample(text="i wanna no what da weather is gonna b like tmrw in nyc"),
    TextSample(text="whats my bank account balance rn"),
]

fixed = fixer.run(data)
for item in fixed:
    print(item.text)

# omg i just got the job i can't believe it i'm literally shaking rn
# i wanna know what the weather is gonna be like tmrw in nyc
# what's my bank account balance rn
```

Note what it did **not** do: no capitals added, no "omg" expanded, no sentences rewritten. It
repairs defects and leaves your author's voice alone.

Get an API key at [sdvm.ai](https://sdvm.ai).

## The three clients

| Client       | What it does                                                                 |
| ------------ | ---------------------------------------------------------------------------- |
| `Auditor`    | Measures. Each sample comes back carrying its verdicts on `.audit`. `confirm=N` majority-votes the verdicts (denoise). |
| `Fixer`      | Repairs, in a single pass (audit-routed). Each sample carries `.repair`; never deterministically worse. Does not re-audit. |
| `Refinery`   | The whole loop in one call: `audit → fix → re-audit`, with `loops` and `confirm`.      |

Each has an async twin (`AsyncAuditor`, `AsyncFixer`, `AsyncRefinery`) with the same `run` method.

## Before / After (text)

`Fixer` corrects spelling, grammar and punctuation, and preserves everything else — meaning,
wording, tone, register and dialect. Informal text stays informal; a sample it judges to have no
defect comes back unchanged, character for character. Real responses:

| Before                                                                    | After                                                                       |
| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `i didnt feel humiliated`                                                 | `i didn't feel humiliated`                                                  |
| `im feeling really angry about this situation its so unfair and annoying` | `i'm feeling really angry about this situation it's so unfair and annoying` |
| `i wanna no what da weather is gonna b like tmrw in nyc`                  | `i wanna know what the weather is gonna be like tmrw in nyc`                |
| `she sels sea shells by the sea shor`                                     | `she sells sea shells by the sea shore`                                     |
| `plz help me set up a direct debit 4 my rent`                             | *unchanged — nothing here is a mistake*                                     |
| `can u transfer 50 bucks to my friend john`                               | *unchanged — `u` and `bucks` are the author's voice, not errors*            |

`no` became `know` and `b` became `be` because they are misspellings; `wanna`, `gonna`, `tmrw` and
`nyc` stayed because they are not.

### Real-world impact

We fixed 90 emotion-classification training samples (from the [dair-ai/emotion](https://huggingface.co/datasets/dair-ai/emotion) dataset) and retrained a TF-IDF + Logistic Regression baseline:

| Metric    | Original data | Fixed data |       Change       |
| --------- | :-----------: | :--------: | :----------------: |
| Accuracy  |     40.0%     |   43.3%    | **+8.3% relative** |
| Macro F1  |     0.388     |   0.395    |       +1.8%        |
| `joy` F1  |     0.40      |    0.57    |      **+43%**      |
| `fear` F1 |     0.62      |    0.80    |      **+30%**      |

Fixed datasets and trained models are published on HuggingFace:

- [SDVM/nlp-transformers-refined](https://huggingface.co/datasets/SDVM/nlp-transformers-refined) — 35 before/after pairs across 11 NLP chapters
- [SDVM/dair-ai-emotion](https://huggingface.co/datasets/SDVM/dair-ai-emotion) — 20,000 samples with a fixed-text column
- [SDVM/emotion-clf-original](https://huggingface.co/SDVM/emotion-clf-original) — baseline model (40.0% accuracy)
- [SDVM/emotion-clf-refined](https://huggingface.co/SDVM/emotion-clf-refined) — fixed model (43.3% accuracy)

## Audit

`Auditor.run` attaches per-sample findings to `.audit`; `Auditor.aggregate` gives a free, local,
deterministic dataset-level view (choice-count distribution, answer-position bias).

```python
from sdvm import Auditor
from sdvm.types import MultipleChoiceSample

samples = [MultipleChoiceSample(context="What is the capital of France?",
                                choices=["Berlin", "Paris", "Madrid", "Rome"],
                                answer_index=1, style="qa")]

audited = Auditor(api_key="sdvm_...").run(samples)
print(audited[0].audit)
# {'no_length_shortcut': True, 'ctx_grammatical': True, 'label_correct': True,
#  'single_valid_answer': True, 'distractors_discriminating': True, ...}

agg = Auditor.aggregate(audited)          # local, free, no model
print(agg["answer_position"]["biased"])   # is the correct answer stuck in one slot?
```

## Question and answer

`QuestionAnswerSample` is the shape with no choices to pick from: a `question` and the `answer`
the dataset publishes — a bare value, a sentence, or a worked solution. Any other column (a
reasoning trace, a source id) rides along in `extra`. Auditing one is free today: the
deterministic checks cost no tokens, and the LLM dimensions for this shape stay off until they
have been measured the way the multiple-choice ones were. `Fixer` declines this type rather than
rewriting an answer it has no measured repair path for.

```python
from sdvm.types import QuestionAnswerSample

audited = Auditor(api_key="sdvm_...").run([
    QuestionAnswerSample(
        question="Natalia sold 48 clips in April and half as many in May. How many altogether?",
        answer="She sold 48 / 2 = 24 clips in May, so 48 + 24 = 72 altogether.",
    ),
])
print(audited[0].audit["answer_non_empty"])
```

## Multiple choice

`MultipleChoiceSample` covers both task shapes with one `style` flag — `"continuation"` (a stem the
correct choice continues, e.g. HellaSwag) or `"qa"` (a question the correct choice answers, e.g.
MMLU). `Fixer.run` repairs an audited item and records what it changed:

```python
fixed = Fixer(api_key="sdvm_...").run(audited)
print(fixed[0].repair)                    # {"changes": [...], "flagged": bool}

flagged = [s for s in fixed if s.repair and s.repair["flagged"]]   # the fix-or-flag contract
```

`flagged` marks a sample the fix couldn't fully resolve (surfaced for review) — the fix **never**
makes a sample deterministically worse than its input.

### Dataset conventions

If a dataset has formatting that is a *convention*, not a defect — uniform lowercasing, markup
tokens, intentional truncation — declare it with `conventions=...` so the quality audit doesn't
flag it. (On HellaSwag's wikiHow text this takes the grammatical false-alarm from ~40% to ~12%.)

```python
WIKIHOW = ("Bracketed markers like [header] [title] [step] are section labels, not errors; the "
           "text is uniformly lowercased — do not flag lowercasing; the context may end "
           "mid-sentence by design — do not flag it as incomplete.")

audited = Auditor(api_key="sdvm_...").run(samples, conventions=WIKIHOW)
fixed   = Fixer(api_key="sdvm_...").run(audited, conventions=WIKIHOW)
```

## The Refinery pipeline

`Refinery` runs the whole loop server-side in one call: `audit(confirm) → [fix → audit(confirm)] ×
loops`. `loops=1` is audit-fix-audit, so each returned sample's `.audit` reflects the **post-fix**
state (did the fix help?) and `.repair` records what the fix changed.

```python
from sdvm import Refinery

refinery = Refinery(api_key="sdvm_...", loops=1, confirm=3,
                    conventions=WIKIHOW)          # forwarded to every stage
result = refinery.run(samples)                    # each carries .audit (final) and .repair
```

`loops=2` iterates fix→re-audit twice (the second audit sees the first fix's result); `confirm` is
the audit denoising described below.

## Controlling output length (text fixes)

```python
fixed = fixer.run(data)                          # model decides (default)
fixed = fixer.run(data, output_length="preserve")  # keep same length
condensed = fixer.run(data, output_length=0.5)     # ~50% of original
expanded  = fixer.run(data, output_length=1.5)     # ~150% of original
```

## Denoising the audit (`confirm`)

The judged dimensions are readings of the item, and on a borderline one the verdict can differ
between requests. `confirm=N` audits each
multiple-choice sample N times and returns the **majority** verdict per dimension — so a flaky flip
doesn't sway the result. It lives on `Auditor` (and `Refinery`), not `Fixer`:

```python
audited = Auditor(api_key="sdvm_...", confirm=3).run(mc_data)   # majority of 3 audits
audited = auditor.run(mc_data, confirm=5)                       # per-call override
refined = Refinery(api_key="sdvm_...", confirm=3).run(mc_data)  # every audit stage votes
```

`confirm=1` (default) is a single audit. Higher denoises at the cost of extra audit calls;
deterministic dimensions are computed once regardless.

## Async support

```python
import asyncio
from sdvm import AsyncFixer
from sdvm.types import TextSample

async def main():
    fixer = AsyncFixer(api_key="sdvm_...")
    result = await fixer.run([TextSample(text="she sels sea shells by the sea shor")])
    for item in result:
        print(item.text)
    # She sells sea shells by the sea shore.

asyncio.run(main())
```

`AsyncAuditor` and `AsyncRefinery` work the same way — `await client.run(data)`.

## Error handling

```python
from sdvm import Fixer, AuthenticationError, InsufficientCreditsError, RateLimitError, APIError
from sdvm.types import TextSample

try:
    result = Fixer(api_key="sdvm_...").run([TextSample(text="hello")])
except AuthenticationError:
    print("Invalid or revoked API key.")
except InsufficientCreditsError:
    print("Not enough credits — add more at https://sdvm.ai/profile.")
except RateLimitError:
    print("Rate limit hit — retry in a few seconds.")
except APIError as e:
    print(f"Unexpected error {e.status_code}: {e}")
```

## API Reference

### `Auditor(api_key, *, confirm=1, base_url, timeout)`

Synchronous audit client. `AsyncAuditor` is the async twin.

| Method                                        | Description                                                            |
| --------------------------------------------- | --------------------------------------------------------------------- |
| `run(data, *, conventions=None, confirm=None)` | Audit up to 100 samples; each returned sample carries `.audit`. `confirm` majority-votes the verdicts. |
| `aggregate(samples)` *(static)*               | Local, free dataset-level view (field distributions, answer-position bias). |

### `Fixer(api_key, *, output_length="preserve", base_url, timeout)`

Synchronous single-pass fix client. `AsyncFixer` is the async twin.

| Method                                                | Description                                            |
| ----------------------------------------------------- | ----------------------------------------------------- |
| `run(data, *, output_length=None, conventions=None)`  | Fix up to 100 samples; each carries `.repair` `{changes, flagged}`. |

### `Refinery(api_key, *, loops=1, confirm=1, conventions=None, output_length="preserve", base_url, timeout)`

The full pipeline in one call. `AsyncRefinery` is the async twin. `run(data, *, loops=None,
confirm=None, conventions=None)` runs `audit(confirm) → [fix → audit(confirm)] × loops`; each
returned sample carries the final `.audit` and last `.repair`.

### Samples

```python
from sdvm.types import TextSample, MultipleChoiceSample, QuestionAnswerSample

TextSample(text="...")

MultipleChoiceSample(
    context="...", choices=["...", "..."], answer_index=0,
    style="continuation",   # or "qa"
    extra={...},            # passthrough metadata
)

QuestionAnswerSample(
    question="...", answer="...",   # a question and its written answer, no choices to pick from
    extra={...},                    # passthrough metadata
)
# after a run, samples carry .audit (dict | None) and .repair ({"changes": [...], "flagged": bool} | None)
```

### `FixerConfig`

```python
@dataclass
class FixerConfig:
    output_length: float | str | None = None  # None, "preserve", or 0.1–3.0
```

### Exceptions

| Exception                  | HTTP | Description                       |
| -------------------------- | ---- | --------------------------------- |
| `AuthenticationError`      | 401  | Invalid or revoked API key        |
| `InsufficientCreditsError` | 402  | Insufficient credits              |
| `RateLimitError`           | 429  | Too many requests (100/min limit) |
| `APIError`                 | any  | Unexpected server error           |

All inherit from `SDVMError`.

## Pricing

Per token, at the rate of the model that did the work — the **auditor** judges, the **fixer**
generates. Snapshot from 2026-09-15, USD per 1M tokens (input / output), rounded up to the cent:

- **Audit**: $0.12 / $0.45
- **Fix**: $0.27 / $0.54
- Minimum **$0.01** per request

Deterministic audit checks are free (no model call). Rates can change;
`GET https://api.sdvm.ai/pricing` returns the live rates and limits, and
[docs.sdvm.ai/pricing](https://docs.sdvm.ai/pricing) explains them.
