Metadata-Version: 2.4
Name: rubric-sdk
Version: 0.1.0
Summary: Lightweight tracing SDK for the Rubric LLM eval + observability platform
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"

# rubric-sdk

A tiny, dependency-light tracing SDK for [Rubric](./SPEC.md), an LLM eval +
observability platform. Wrap your LLM calls with `@trace` / `span(...)`, and
the SDK ships the resulting trace tree to your Rubric ingestion endpoint —
asynchronously, in the background, and **fire-and-forget**: if the endpoint
is down or unset, your app keeps working and spans are just dropped (logged
at debug, never raised).

This repo owns the frozen v1 trace/span schema (`schema_version = "1"`, see
[SPEC.md section 4](./SPEC.md#4-frozen-tracespan-schema-v1)) and does
nothing else — no storage, no eval logic, no dashboard. Those live in the
platform repo.

## Install

```bash
pip install rubric-sdk
```

For local development against a clone of this repo:

```bash
pip install -e ".[dev]"
```

**Version pinning:** the wire schema is frozen per `schema_version`, but pin
a minor version anyway (`rubric-sdk~=0.1.0`) so a future release can't
change field defaults or add required fields out from under you.

## Quickstart

```python
from rubric import trace, span

@trace(app="myapp")
def handle(prompt: str) -> str:
    with span("llm.call", input=prompt) as sp:
        sp.output = f"echo: {prompt}"  # replace with your real LLM call
        return sp.output

print(handle("hello rubric"))
```

That's it — no client to construct, no manual shipping call. `@trace` opens
a trace, `span(...)` records a timed unit of work inside it (with
`parent_span_id` linked automatically via `contextvars`, safe across both
`asyncio` tasks and threads), and the completed trace ships in the
background as soon as `handle(...)` returns.

## Instrumenting a Bedrock `converse` call

```python
import boto3
from rubric import trace, span

bedrock = boto3.client("bedrock-runtime")
MODEL_ID = "anthropic.claude-3-haiku-20240307-v1:0"

@trace(app="myapp")
def ask(prompt: str) -> str:
    with span("bedrock.converse", span_type="llm", input=prompt) as sp:
        response = bedrock.converse(
            modelId=MODEL_ID,
            messages=[{"role": "user", "content": [{"text": prompt}]}],
        )
        usage = response["usage"]
        sp.output = response["output"]["message"]["content"][0]["text"]
        sp.model = MODEL_ID
        sp.input_tokens = usage["inputTokens"]
        sp.output_tokens = usage["outputTokens"]
        return sp.output
```

`cost_usd` is computed automatically from `rubric/pricing.py`'s table when
`model` + token counts are set (override/extend it via `RUBRIC_PRICING_JSON`,
see below). If you've already made the call and just want to log it in one
line, use `record_llm(...)` instead of the `with span(...)` block:

```python
from rubric import record_llm

record_llm(
    "bedrock.converse",
    input=prompt,
    output=text,
    model=MODEL_ID,
    input_tokens=usage["inputTokens"],
    output_tokens=usage["outputTokens"],
)
```

## Configuration

All tunables are environment variables — nothing is hardcoded. See
[`.env.example`](./.env.example) for the full list with defaults:

| Env var                  | Purpose                                                        |
|---------------------------|----------------------------------------------------------------|
| `RUBRIC_ENDPOINT`         | Ingestion URL. Unset → spans are captured but never shipped.   |
| `RUBRIC_API_KEY`          | Sent as `Authorization: Bearer <key>` when shipping. Optional. |
| `RUBRIC_DISABLED`         | `1`/`true` → the whole SDK becomes a no-op. See below.         |
| `RUBRIC_BATCH_SIZE`       | Max items per shipped HTTP POST. Default `20`.                |
| `RUBRIC_FLUSH_INTERVAL`   | Seconds between background flush attempts. Default `2.0`.     |
| `RUBRIC_MAX_BUFFER_SIZE`  | In-memory buffer cap; oldest item dropped once full. Default `1000`. |
| `RUBRIC_TIMEOUT`          | HTTP timeout in seconds for shipping requests. Default `5.0`.  |
| `RUBRIC_PRICING_JSON`     | JSON override/extension of the built-in cost-per-model table.  |

### Kill switch

Set `RUBRIC_DISABLED=1` to make the SDK a complete no-op: `@trace`, `span`,
and `record_llm` skip all bookkeeping (no objects built, no contextvars
touched) and nothing is ever buffered or shipped. This makes instrumenting
a real app risk-free — you can ship the `@trace`/`span` calls and flip them
off instantly if anything looks wrong.

## Schema

The frozen v1 `Trace`/`Span` pydantic models live in `rubric/schema.py`.
Regenerate the equivalent JSON Schema (so it can never drift from the
models) with:

```bash
python -m rubric.export_schema
```

This writes `rubric/schema.json`. `schema_version` is `"1"` — see
[SPEC.md](./SPEC.md) for the frozen field list and the retrieval-span
`output` convention.

## Running the tests / demo

```bash
pip install -e ".[dev]"
pytest -q

# Prints the captured Trace + Span tree as JSON — no network needed.
python scripts/demo_local.py
```

To see real shipping, point `RUBRIC_ENDPOINT` at any HTTP server (e.g.
`python -m http.server`) and run the demo again; then unset it and confirm
the demo still runs cleanly with spans silently dropped.

## Releasing

Publishing to PyPI is automated via
[`.github/workflows/publish.yml`](./.github/workflows/publish.yml):

1. Bump `version` in `pyproject.toml`.
2. Tag the commit and push the tag: `git tag v0.1.1 && git push origin v0.1.1`.
3. The workflow runs the test suite, verifies the tag matches
   `pyproject.toml`'s version, builds the wheel/sdist, and publishes via
   [PyPI trusted publishing](https://docs.pypi.org/trusted-publishers/)
   (OIDC — no API token stored in this repo).
4. It can also be re-run manually from the Actions tab (`workflow_dispatch`),
   e.g. to retry a failed publish for an already-tagged version.

**One-time setup required on PyPI** before the first release: on the
project's PyPI page, add a trusted publisher pointing at this GitHub repo,
workflow file `publish.yml`, and environment `pypi`. The publish job also
targets a `pypi` GitHub Environment — configure protection rules there
(e.g. required reviewers) if you want a manual approval gate before publish.

## Out of scope

No storage, no eval logic, no dashboard — this repo's only job is: define
the schema, capture traces, ship them safely. See [SPEC.md](./SPEC.md) for
the full platform design.
