# 06 — Crash reporting

`report_now()` writes an atomic crash file + best-effort ships
to remote sinks. `install_excepthook()` /
`install_threading_excepthook()` chain to `report_now` so any
unhandled exception lands a crash report before the process
exits.

> **Read first:** [`library_common.md`](library_common.md),
> [`02_exceptions.md`](02_exceptions.md),
> [`05_observability.md`](05_observability.md).

---

## Motivation

Production processes crash. Without a reliable last-ditch
crash recorder, the diagnostic information vanishes when the
process exits. `report_now()` writes the crash to disk before
any remote ship attempt — even if the network is down, the
operator finds the report in `<state>/<app>/crashes/`.

Sinks are pluggable: `local` is always-on; `sentry`, `otlp`,
`http`, and any consumer-registered name fan out from the same
report. `report_now()` itself NEVER raises — last line of
defense, no chance to make things worse.

---

## Goal

```python
from mgf.common.observability import (
    install_excepthook,
    install_threading_excepthook,
    report_now,
)

# Wired automatically by bootstrap(); shown here for clarity.
install_excepthook()
install_threading_excepthook()

# Manual report (e.g., a caught-but-unrecoverable exception):
try:
    do_risky()
except UnrecoverableError as e:
    report_path = report_now(e, source="my_critical_path")
    raise SystemExit(1)
```

---

## Quick start

`bootstrap()` installs both excepthooks automatically. Anything
escaping the main loop OR a non-main thread lands a crash
report.

For caught-but-noteworthy exceptions, call `report_now()`
manually:

```python
from mgf.common.observability import report_now

try:
    expensive_third_party_call()
except SomeSDKError as e:
    # Translate at the boundary (see 02_exceptions.md), AND
    # capture the original for postmortem:
    report_now(e, source="sdk.expensive_call")
    raise CommandError(argv=[...], returncode=1, stderr=str(e)) from e
```

---

## Public API

| Name | Purpose |
|---|---|
| [`report_now(exc, *, source)`](../../PUBLIC_API.md#mgfcommonobservability) | Build, persist locally, and best-effort ship a crash report. Returns the local file path. **NEVER raises.** |
| [`install_excepthook()`](../../PUBLIC_API.md#mgfcommonobservability) | Process-wide unhandled-exception hook. Idempotent. |
| [`install_threading_excepthook()`](../../PUBLIC_API.md#mgfcommonobservability) | Non-main-thread excepthook. Idempotent. |
| [`register_crash_sink(name)`](../../PUBLIC_API.md#mgfcommonobservability) | Decorator: register a custom crash-sink handler. |
| [`unregister_crash_sink(name)`](../../PUBLIC_API.md#mgfcommonobservability) | No-op if unknown. |
| [`CrashSink`](../../PUBLIC_API.md#mgfcommonobservability) | Type alias: `Callable[[Path, dict[str, Any]], None]`. |

Code:
[`src/mgf/common/observability/crash_reporter.py`](../../src/mgf/common/observability/crash_reporter.py),
[`excepthook.py`](../../src/mgf/common/observability/excepthook.py),
[`threading_excepthook.py`](../../src/mgf/common/observability/threading_excepthook.py).

---

## What lands in a crash report

Atomic JSON file at
`<state>/<app>/crashes/crash-<UTC-timestamp>-<source>.json`:

```json
{
  "timestamp": "2026-04-28T18:42:00.123456Z",
  "app": "myapp",
  "version": "1.0.0",
  "source": "my_critical_path",
  "exception": {
    "type": "MyAppError",
    "args": ["..."],
    "module": "myapp.errors",
    "machine_fields": {"name": "vm-42", "..."}
  },
  "traceback": "Traceback (most recent call last):\n  ...",
  "host": {"hostname": "...", "platform": "..."},
  "redacted": true
}
```

Machine fields (e.g., `ResourceNotFoundError.name`,
`CommandError.argv`) are extracted from typed exceptions and
included as structured data alongside the message.

The file is written atomically (temp + rename + chmod 0o600
per CF-04 / SC-04). Caller never observes a partially-written
file. Redaction is applied to every string in the payload via
the active `Redactor` (see
[`05_observability.md`](05_observability.md)).

---

## Pluggable sinks

`MgfSettings.crash.sinks` is a list of registered sink names.
Built-in: `local` (always-on file write), `sentry`, `otlp`,
`http`. Consumer-registered names also accepted.

```python
from pathlib import Path
from typing import Any

from mgf.common.observability import register_crash_sink

@register_crash_sink("slack")
def ship_to_slack(report_path: Path, payload: dict[str, Any]) -> None:
    import requests
    requests.post(
        "https://hooks.slack.com/services/...",
        json={"text": f"crash in {payload['app']}: {payload['exception']['type']}"},
        timeout=5,
    )
```

Activate via settings:

```python
from mgf.common.settings import MgfSettings, CrashSettings

settings = MgfSettings(
    crash=CrashSettings(sinks=["local", "slack"]),
)
```

Multi-sink fan-out: each sink fires independently with
try/except wrapping (rule EH-10 best-effort); failure of one
doesn't affect others.

---

## Patterns

### Crash on first call vs aggregated

`report_now()` is appropriate when the exception is being
re-raised or the process is about to exit. Don't call it for
expected exceptions in a retry loop — every report is a
disk-write + remote-ship cycle. Throttle / dedup at the call
site if needed.

### Custom sink for compliance

```python
from pathlib import Path
import boto3

@register_crash_sink("s3-bucket")
def ship_to_s3(report_path: Path, payload: dict) -> None:
    s3 = boto3.client("s3")
    s3.upload_file(str(report_path), "myapp-crashes", report_path.name)
```

Useful when on-disk crash files need to flow into a compliance
bucket on a different host.

### Excepthook idempotency

`install_excepthook()` carries a sentinel attribute so chained
installs don't double-fire crash reports per crash. Safe to
call from multiple entry points if needed.

---

## Links

- **Public API:** [`PUBLIC_API.md` `mgf.common.observability` section](../../PUBLIC_API.md#mgfcommonobservability)
- **Engineering standards:**
  [`ERROR_HANDLING.md`](../standards/ERROR_HANDLING.md) EH-04 / EH-05 / EH-10
- **Examples:** [`examples/05_crash_reporting/`](../../examples/05_crash_reporting/)
- **Code:**
  [`crash_reporter.py`](../../src/mgf/common/observability/crash_reporter.py),
  [`excepthook.py`](../../src/mgf/common/observability/excepthook.py),
  [`threading_excepthook.py`](../../src/mgf/common/observability/threading_excepthook.py)
- **Related docs:**
  - [`02_exceptions.md`](02_exceptions.md) — typed exceptions
    contribute machine fields to the report
  - [`05_observability.md`](05_observability.md) — the redactor
    runs over the report before any sink fires
  - [`09_cli.md`](09_cli.md) — the CLI translator's
    catch-all branch ships a crash report before exiting
