# Recipe — Async HTTP client with `mgf.common.http`

> **Audience:** consumers integrating with REST APIs (third-party
> providers OR your own services). Fits both "make 3 HTTP calls in a
> CLI tool" and "5,000 calls/sec from a FastAPI service."
> **Optional extra:** `pip install 'mgf-common[http]'`.

## At a glance

```python
from mgf.common.http import AsyncHttpClient, RetryPolicy

async with AsyncHttpClient(
    base_url="https://api.example.com",
    timeout=10.0,
    retry=RetryPolicy(max_attempts=3),
) as http:
    response = await http.get("/users", params={"limit": 50})
    response.raise_for_status()
    users = response.json()
```

That's a complete client with:
- User-Agent auto-set (`<your-app>/<your-ver> mgf-common/<lib-ver>`)
- Retry on transient failures (timeouts, 503s, network errors) with
  exponential backoff + jitter
- `Authorization` / `Cookie` / `X-Api-Key` headers redacted in logs
  (the value, not the name — the name in logs helps debugging)
- TLS verification on by default; loud `UserWarning` if you ever
  pass `verify_tls=False`
- `httpx.TimeoutException` etc. wrapped into `HttpClientError`
  (subclass of `mgf.common.exceptions.OperationError`) so the
  standard CLI exit-code translator routes them correctly
- Best-effort OTel client span when `[observability]` extra is
  installed (no-op otherwise)

## What's in the box

### `AsyncHttpClient`

```python
AsyncHttpClient(
    *,
    base_url: str = "",
    timeout: float = 30.0,
    verify_tls: bool = True,
    user_agent: str | None = None,
    default_headers: Mapping[str, str] | None = None,
    retry: RetryPolicy | None = None,
)
```

- `base_url` — relative URLs in `request(url=...)` calls join here.
- `timeout` — per-request timeout in seconds. Per-call override via
  `request(timeout=...)`.
- `verify_tls` — TLS cert verification. **Always leave at default
  in production.**
- `user_agent` — override the auto-derived UA.
- `default_headers` — sent on every request; per-call headers
  merge over them.
- `retry` — see :class:`RetryPolicy`. Default: zero retries.

Convenience methods:
`get`, `post`, `put`, `patch`, `delete`, `head`. Each forwards to
`request(METHOD, url, ...)`.

Lifecycle: use as `async with AsyncHttpClient() as http:` OR call
`await http.aclose()` explicitly. Idempotent; safe to call twice.

Escape hatch: `http.client` returns the wrapped
`httpx.AsyncClient` for streaming downloads, multipart uploads, or
custom transports. Reaching through skips the wrapper's retry +
redaction + error translation.

### `RetryPolicy`

```python
RetryPolicy(
    max_attempts: int = 0,
    base_backoff_seconds: float = 0.5,
    max_backoff_seconds: float = 30.0,
    retryable_methods: frozenset[str] = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE"},
    retryable_statuses: frozenset[int] = {429, 503, 504},
)
```

- `max_attempts` — number of RETRIES after the first attempt. Total =
  `max_attempts + 1`. Default 0 = single attempt = deterministic.
- Backoff is **exponential with full jitter**: attempt N waits a
  uniform random in `[0, base * 2^N]`, capped at `max_backoff`.
  Avoids the thundering-herd shape pure-exponential produces.
- `retryable_methods` defaults to RFC 7231 idempotent + DELETE
  (which is idempotent in practice). **POST is excluded** — repeating
  POST may double-submit. Override explicitly if your endpoint is
  safely retryable.
- `retryable_statuses` covers the canonical "service unavailable,
  try again" set.

Frozen dataclass; safe to share between requests.

### `HttpClientError`

```python
class HttpClientError(OperationError):
    url: str
    cause_summary: str
    attempts: int
```

Raised AFTER retries exhaust on transport-level failures. Original
`httpx` exception preserved via `__cause__`. 4xx / 5xx responses
**don't raise** here — they return as `httpx.Response`; use
`response.raise_for_status()` if you want a typed exception.

## Common patterns

### Provider auth via per-request header

```python
async with AsyncHttpClient(base_url="https://api.example.com") as http:
    response = await http.get(
        "/private",
        headers={"Authorization": f"Bearer {token}"},
    )
```

Logs the header NAME (so you can see auth was attempted) but
redact the VALUE (`<redacted>` in the log line).

### Vault-resolved auth

```python
from mgf.common.vault import EnvVault

vault = EnvVault()

async with AsyncHttpClient() as http:
    token = vault.get("STRIPE_SECRET_KEY")  # raises VaultError if absent
    response = await http.get(
        "https://api.stripe.com/v1/charges",
        headers={"Authorization": f"Bearer {token}"},
    )
```

### Per-request retry override

```python
# Use stricter retry for a critical idempotent call.
response = await http.get(
    "/health",
    retry=RetryPolicy(max_attempts=5, base_backoff_seconds=0.1),
)
```

### Reusing a client across the lifetime of a service

In FastAPI / a long-running service, construct once at lifespan
startup:

```python
@app.on_event("startup")
async def _startup() -> None:
    app.state.http = AsyncHttpClient(
        base_url=settings.upstream_api,
        retry=RetryPolicy(max_attempts=3),
    )

@app.on_event("shutdown")
async def _shutdown() -> None:
    await app.state.http.aclose()

# Inside handlers:
async def my_handler(request: Request) -> dict:
    response = await request.app.state.http.get("/users")
    ...
```

The connection pool persists across requests — far cheaper than a
fresh client per request.

(See `mgf.common.fastapi` for a Depends-based helper that wraps
this same pattern.)

### Testing with respx

```python
import httpx
import respx
import pytest
from mgf.common.http import AsyncHttpClient

@pytest.mark.asyncio
@respx.mock
async def test_my_call() -> None:
    respx.get("https://api.example.com/users").mock(
        return_value=httpx.Response(200, json=[{"id": 1}])
    )
    async with AsyncHttpClient(base_url="https://api.example.com") as http:
        response = await http.get("/users")
    assert response.json() == [{"id": 1}]
```

`respx` mocks at the transport layer; the wrapper is transparent.
Install via the `[http-test]` extra: `pip install 'mgf-common[http-test]'`.

## Migrating an existing HTTP transport

If your project already has a custom httpx wrapper (apiprobe's
`HttpxTransport` is the canonical example), this is the migration
shape that preserves your public surface while delegating the
mechanics to `AsyncHttpClient`.

### Step 1 — keep your transport class as the public adapter

Don't replace your transport with `AsyncHttpClient` directly —
keep your existing class so callers' imports don't change. Make
it a thin adapter:

```python
# myapp/transport/httpx_transport.py — was a hand-rolled httpx wrapper
from mgf.common.http import AsyncHttpClient, HttpClientError

from myapp.transport.errors import TransportError # your own typed error

class HttpxTransport:
    """Project-internal transport. Adapts AsyncHttpClient to the
    project's own error type + auth conventions."""

    def __init__(self, *, base_url: str, api_token: str) -> None:
        self._client = AsyncHttpClient(
            base_url=base_url,
            default_headers={"Authorization": f"Bearer {api_token}"},
        )

    async def get(self, path: str, **kw) -> dict:
        try:
            response = await self._client.get(path, **kw)
            response.raise_for_status()
            return response.json()
        except HttpClientError as e:
            raise TransportError(
                f"upstream call failed: {e}"
            ) from e

    async def aclose(self) -> None:
        await self._client.aclose()
```

The adapter pattern is the migration win: callers keep their
existing typed exception (`TransportError`); the wrapper-of-a-wrapper
gives them retry policy + redaction + identity headers + OTel for
free, AND consumers don't see the change.

### Step 2 — translate httpx errors to your errors at the seam

Your domain code shouldn't catch `HttpClientError` or `httpx.HTTPError`
directly — that leaks the transport into the domain. The adapter
class is the seam where translation happens (the second `raise`
above). Your domain code catches `TransportError` only.

The original httpx exception is preserved via `__cause__` chain
(httpx → HttpClientError → TransportError); for debugging, walk the
chain.

### Step 3 — let mgf-common own the retry policy

If your hand-rolled transport had its own retry loop, delete it.
Pass a `RetryPolicy` to `AsyncHttpClient` instead:

```python
from mgf.common.http import AsyncHttpClient, RetryPolicy

self._client = AsyncHttpClient(
    base_url=base_url,
    retry=RetryPolicy(max_attempts=3),  # default excludes POST/PATCH
)
```

The library's policy uses full-jitter exponential backoff, the
shape that survives every retry-storm postmortem in the literature.
Don't roll your own.

### Step 4 — drop your User-Agent override

`AsyncHttpClient` derives the User-Agent from `mgf.common`'s
identity automatically (`<app_name>/<app_version>` after `bootstrap()`).
Hand-rolled UA strings drift. Delete yours unless you have a
contractual reason to override.

### Step 5 — tighten the dep pin

```toml
# pyproject.toml — was
"mgf-common>=0.4,<0.6"
"httpx>=0.27"

# becomes
"mgf-common[http]>=0.6.0,<0.8"
# httpx falls out — pulled by the [http] extra.
```

The `[http]` extra is mgf-common's promise to keep httpx in scope.
Don't dual-pin; let the extra own it.

### Step 6 — verify behavior in CI

The adapter shape is observably equivalent to the original wrapper.
Two checks:

1. Your existing transport tests (mocked at the httpx level via
   respx) MUST still pass — that's the regression gate.
2. Your TransportError chain is intact: assert `isinstance(e.__cause__,
   HttpClientError)` and `isinstance(e.__cause__.__cause__,
   httpx.HTTPError)` for transport-level failures.

### Worked migration: `mgf-apiprobe`

`mgf-apiprobe`'s `HttpxTransport` migrated to this shape in
(apiprobe commit `ae03466`, against mgf-common ).
Net deletion: ~200 LOC of retry / redaction / UA / span boilerplate.
The public `Transport` Protocol stayed unchanged; consumer code
in apiprobe didn't move.

## Anti-patterns

- **Don't construct a fresh `AsyncHttpClient` per request.** The
  connection pool is the value. Hold one client per process or per
  upstream API.
- **Don't enable retries on every method.** POST + PATCH on
  non-idempotent endpoints can double-submit on retry. Default policy
  excludes them; override explicitly only when you know it's safe.
- **Don't catch `httpx.HTTPError` directly.** The wrapper translates
  them to `HttpClientError` before they reach you. Catch the typed
  one + use `__cause__` if you need the original.
- **Don't pass `verify_tls=False` in production.** The runtime
  warning fires on every construction — silencing it is a code
  smell.
- **Don't mutate `client.headers` directly.** Use the `default_headers`
  constructor arg or per-call `headers=` kwarg. The wrapper assumes
  the underlying httpx client's headers are stable.

## See also

- `docs/public/recipes/cli.md` — building CLIs that call HTTP APIs.
- `docs/public/recipes/testing.md` — pytest fixtures for HTTP-using code.
- `docs/public/06_vault.md` — secrets resolution patterns.
- `docs/public/01_lifecycle.md` — `bootstrap()` + AppContext.
