# Recipe — Observability shim layering

> **Audience:** consumers who want to add their own redactor
> patterns, log handlers, formatters, or other observability
> extensions without forking `mgf.common.observability` —
> through a project-local `<myapp>/observability/__init__.py`
> re-export shim.
> **No optional extra** — works against the bare
> `pip install mgf-common`. (Add `mgf-common[observability]` if
> you need OTel-side extensions too.)

## Why a shim — even without legacy call sites

The naïve answer is "for back-compat" — a shim that re-exports
`mgf.common.observability` lets pre-existing
`from myapp.observability import setup_logging` calls keep
working when the implementation moves into `mgf-common`. That's
real, but it's not the main reason.

**The main reason is extensibility.** A typed cornerstone library
can't anticipate every consumer's domain-specific observability
needs:

- **Domain-specific redaction patterns.** `mgf.common.observability`'s
  default redactor scrubs the obvious set (passwords, tokens,
  API keys). Your domain has its own secrets to scrub —
  account numbers, customer ids in a specific format, vendor
  webhook signatures — that don't generalise to a cross-project
  default.
- **Custom log handlers.** A handler that fans logs out to a
  proprietary SaaS, an internal Kafka cluster, or a database
  for compliance retention.
- **Project-shaped formatter wrappers.** A formatter that
  pre-formats records with project-specific top-level keys
  (e.g., `tenant_id`, `cluster`, `region`) sourced from a
  contextvar.
- **Project-specific span attributes.** A wrapper around
  `operation_span` that auto-attaches your project's
  identifiers to every span.

The shim is where these extensions live. Consumer code keeps
importing from a single, stable path —
`from myapp.observability import get_logger` — and the shim
either re-exports the upstream symbol unchanged or returns a
project-extended version. The upstream library never knows;
the consumer code never branches.

## At a glance

```python
# myapp/observability/__init__.py

"""Project-local observability shim.

Re-exports mgf.common.observability with project-specific
extensions layered on top. Consumer code SHOULD import from
this module rather than mgf.common.observability directly.
"""

from __future__ import annotations

# Re-export the public surface of mgf.common.observability.
# Add or remove names here as your project's needs evolve;
# the canonical list lives in mgf.common.observability.__all__.
from mgf.common.observability import (
    DEFAULT_REDACTION_KEYS,
    AsyncJsonHandler,
    CrashSink,
    JsonFormatter,
    LogFormat,
    RedactionGroup,
    Redactor,
    Span,
    SpanKind,
    add_event,
    get_current_span,
    get_logger,
    install_excepthook,
    install_threading_excepthook,
    operation_span,
    register_crash_sink,
    register_log_handler,
    register_redaction_pattern_group,
    report_now,
    set_attribute,
    set_status,
    setup_logging,
    setup_otel,
)

# Project extensions go here. Examples:
from myapp.observability._myapp_handler import MyAppKafkaHandler
from myapp.observability._myapp_redactor import MyAppRedactor
from myapp.observability._myapp_span import myapp_operation_span

__all__ = [
    # Re-exports (subset most commonly used by consumer code)
    "DEFAULT_REDACTION_KEYS",
    "AsyncJsonHandler",
    "CrashSink",
    "JsonFormatter",
    "LogFormat",
    "RedactionGroup",
    "Redactor",
    "Span",
    "SpanKind",
    "add_event",
    "get_current_span",
    "get_logger",
    "install_excepthook",
    "install_threading_excepthook",
    "operation_span",
    "register_crash_sink",
    "register_log_handler",
    "register_redaction_pattern_group",
    "report_now",
    "set_attribute",
    "set_status",
    "setup_logging",
    "setup_otel",
    # Project extensions
    "MyAppKafkaHandler",
    "MyAppRedactor",
    "myapp_operation_span",
]
```

Consumer code anywhere else in the project:

```python
# Anywhere in myapp/...
from myapp.observability import get_logger, operation_span, MyAppRedactor

LOG = get_logger(__name__)

with operation_span("fetch", user_id="..."):
    LOG.info("starting", extra={"tenant_id": "..."})
```

## Layering custom extensions

### Custom redactor

Subclass or compose `mgf.common.observability.Redactor` — never
fork it:

```python
# myapp/observability/_myapp_redactor.py
from __future__ import annotations

from mgf.common.observability import Redactor


class MyAppRedactor(Redactor):
    """Redactor that adds account-number + customer-id patterns
    on top of the cross-project defaults."""

    @classmethod
    def with_myapp_defaults(cls) -> "MyAppRedactor":
        base = Redactor.default()
        return cls(
            keys=base.keys | frozenset({"account_number", "customer_id"}),
            value_patterns=[
                *base.value_patterns,
                # MyApp account numbers: ACC-NNNNNNNN
                r"\bACC-\d{8}\b",
                # MyApp customer ids: CUST-<UUID>
                r"\bCUST-[0-9a-fA-F-]{36}\b",
            ],
        )
```

Wire it at bootstrap by passing the project redactor to
`MgfSettings`:

```python
import mgf.common
from myapp.observability import MyAppRedactor
from mgf.common.settings import MgfSettings, RedactionSettings

with mgf.common.bootstrap(
    settings=MgfSettings(
        redaction=RedactionSettings(
            extra_keys=frozenset({"account_number", "customer_id"}),
            extra_value_patterns=[
                r"\bACC-\d{8}\b",
                r"\bCUST-[0-9a-fA-F-]{36}\b",
            ],
        ),
    ),
):
    ...
```

(For most cases, `RedactionSettings(extra_keys=..., extra_value_patterns=...)`
is enough and a custom subclass is overkill — reach for
subclassing only when you need behavior changes beyond
configuration.)

### Custom log handler

```python
# myapp/observability/_myapp_handler.py
from __future__ import annotations

import logging

from mgf.common.observability import JsonFormatter


class MyAppKafkaHandler(logging.Handler):
    """Send JSON-formatted records to an internal Kafka topic.

    Attaches alongside the standard console + rotating-file
    handlers — does NOT replace them.
    """

    def __init__(self, *, topic: str, redactor) -> None:
        super().__init__()
        self.setFormatter(JsonFormatter(redactor=redactor))
        self._producer = _make_kafka_producer(topic)

    def emit(self, record: logging.LogRecord) -> None:
        try:
            payload = self.format(record).encode("utf-8")
            self._producer.send(payload)
        except Exception:  # noqa: BLE001 — logging handlers swallow
            self.handleError(record)
```

Attach it to the root logger after `bootstrap()`:

```python
import logging
from myapp.observability import MyAppKafkaHandler, Redactor

with mgf.common.bootstrap(...):
    handler = MyAppKafkaHandler(topic="myapp.logs", redactor=Redactor.default())
    logging.getLogger().addHandler(handler)
    # ... your app ...
```

### Wrapper around `operation_span`

```python
# myapp/observability/_myapp_span.py
from __future__ import annotations

from contextlib import contextmanager
from typing import Any

from mgf.common.observability import operation_span


@contextmanager
def myapp_operation_span(name: str, **attrs: Any):
    """operation_span that auto-attaches MyApp identifiers."""
    from myapp.context import current_tenant_id, current_cluster

    with operation_span(
        name,
        tenant_id=current_tenant_id() or "(none)",
        cluster=current_cluster() or "(none)",
        **attrs,
    ) as span:
        yield span
```

## The closed-box rule applies INSIDE the shim

The shim is consumer-side code, but it's still bound by the
closed-box rule on `mgf.common`:

**Never import from `mgf.common.observability._*`** (any
underscore-prefixed module). Only use the public surface
documented in [`PUBLIC_API.md`](../../../PUBLIC_API.md).

Concretely:

```python
# ✅ OK
from mgf.common.observability import Redactor, JsonFormatter, operation_span

# ❌ NOT OK — _helpers is private; subject to change in any release
from mgf.common.observability._helpers import _redact_dict_inplace
```

If you find yourself wanting a private symbol, that's a signal
to file a `FEEDBACK.md` entry — the symbol either needs
promoting to the public surface or `mgf-common` is missing the
right shape for your use case.

## Enforce the closed-box with `import-linter`

Add a contract to your project's `.importlinter` so a casual
import doesn't slip the rule by:

```ini
[importlinter:contract:no-private-mgf-common-imports]
name = myapp must not reach into mgf.common._* private modules
type = forbidden
source_modules =
    myapp
forbidden_modules =
    mgf.common._lifecycle
    mgf.common._identity
    mgf.common._errors
    mgf.common._env_source
    mgf.common._test_helpers
    mgf.common._warnings
    mgf.common._request_id
ignore_imports =
    # Allow internal mgf-common imports IF you maintain mgf-common too;
    # otherwise leave this empty.
```

`mgf-common`'s closed-box rule is enforced via documentation
plus the `__all__` discipline; consumers add the import-linter
contract for AST-level enforcement on their side.

## When NOT to use the shim

The shim adds an indirection. Don't reach for it when:

- **Your project has no observability extensions yet.** Just
  `from mgf.common.observability import ...` directly. Add
  the shim later if and when extensions appear — the migration
  is a one-line search-and-replace per import site.
- **You're writing a one-off script or a notebook.** The
  observability stack is configured by `bootstrap()`; the
  shim only matters when the project has its own modules.
- **You're a library, not an app.** Libraries should not
  bootstrap and should not own observability extensions —
  consumers do that. (See
  [`library_common.md`](../library_common.md) §"I'm a
  library" for the right shape.)

## Migrating onto the shim

If you have existing imports scattered across the project, the
migration is mechanical:

1. Create `myapp/observability/__init__.py` with the re-export
   block above.
2. Search-and-replace `from mgf.common.observability` →
   `from myapp.observability` across `myapp/`. (NOT in
   `tests/` — tests should keep importing from
   `mgf.common.observability` to verify the upstream behavior
   unchanged.)
3. Add the import-linter contract.
4. Run your gates (`ruff`, `mypy`, `pytest`, `lint-imports`)
   to confirm nothing broke.
5. Layer in your project extensions one at a time, each
   covered by its own tests.

## See also

- [`05_observability.md`](../05_observability.md) — the
  upstream observability stack (logging, OTel, redaction,
  crash reporter).
- [`docs/standards/LOGGING.md`](../../standards/LOGGING.md) —
  the cross-project logging standard, including the redaction
  + structured-logging MUSTs.
- [`docs/standards/SECURITY_STANDARD.md`](../../standards/SECURITY_STANDARD.md) —
  the SC-* matrix, including the secrets-in-logs rules that
  redaction enforces.
- [`PUBLIC_API.md`](../../../PUBLIC_API.md) — the public
  surface your shim is allowed to re-export.
