---
status: "proposed"
date: 2026-05-25
decision-makers: Adam, John E.
---

# Backward-Compatible Migrations for Fides Deployments

## Context and Problem Statement

Fides has no standard for migration backward-compatibility. Today, an upgrade can change the database schema in ways that the prior webserver version cannot tolerate — drop a column it still reads, add a NOT NULL without default, rename a table, restructure a relationship. When an upgrade goes wrong, rolling the webserver image back to the prior version is therefore not a reliable recovery path: the older code may crash or misbehave against the upgraded schema. The only first-class recovery option we have is `/admin/db/{action}`, which requires a healthy webserver to invoke — exactly what we don't have after a failed upgrade.

A prior draft of this ADR proposed solving the recovery problem by building first-class infrastructure for downgrading the database (an out-of-band migration runner that knows how to reliably undo schema changes). After review, we've changed direction. The reasons:

- **DB downgrades are intrinsically fragile.** The load-bearing objection is that many real migrations involve data transformations (column splits, normalization changes, type coercions, data backfills) that cannot be losslessly reversed — no amount of `downgrade()` discipline fixes that. A secondary objection is that `downgrade()` is only ever run at incident time, so it rots between authoring and use; on its own this "you can't trust what you don't exercise" point is weak (it could be leveled at any safety property), but here it is *asymmetric*: the backward-compat property is exercised continuously — on every rolling deploy, where prior and new webservers run against the same post-migration schema, and on every release via the gating integration test (see Decision Outcome) — whereas a `downgrade()` path is exercised only when something has already gone wrong. The thing we depend on for recovery is therefore the thing we run constantly, not the thing we run never.
- **The catch-22 only exists because the prior schema and the new schema are mutually incompatible.** If the upgraded schema were still operable by the prior webserver, app rollback alone would be sufficient — no DB rollback required. That property is achievable as a *property of the migration*, enforceable at PR time, and far easier to test than schema-rollback machinery.
- **The same property unlocks zero-downtime rolling upgrades**, which is independently a high-priority goal. During a rolling deploy, prior and new webserver versions run simultaneously against the same (post-migration) schema. Backward-compatible migrations make rolling deploys safe; the absence of that guarantee is precisely why we can't do rolling deploys today.
- **This is the industry-mature pattern.** Forward-only schemas with backward-compatible migrations via expand/contract — rather than bidirectional schema rollback — is well-documented practice. The expand/contract (a.k.a. *parallel change*) pattern is cataloged by Fowler and Sato ([Parallel Change](https://martinfowler.com/bliki/ParallelChange.html)) and in *Refactoring Databases* (Ambler & Sadalage); [Stripe](https://stripe.com/blog/online-migrations) documents the four-phase dual-write form of it, and [Shopify](https://kirshatrov.com/posts/async-migrations) decouples backward-compatible migrations from deploys. The hard-won lesson is that the policy is cheaper to maintain than the machinery.

So the *primary* recovery mechanism we want is **app rollback against an unchanged DB**, made safe by a policy that every migration leaves the database in a state the prior webserver version can operate against. The infrastructure we still need (a decoupled migration runner) is in service of that policy, not in service of routinely rolling the schema back.

Scope assumptions: the database itself is available; pod-level unavailability of webserver/worker is the failure mode we design for. Supported deployment shape is Kubernetes (EKS), driven by Terraform `helm_release` in production, the Pulumi Automation API in tenant-manager, and direct `helm` for self-hosted customers.

Related ticket: [ENG-3710 — Support Fides downgrades (+ harden upgrades)](https://ethyca.atlassian.net/browse/ENG-3710)

## Considered Options

* **Backward-compatible migrations as policy, with the migration runner as supporting infrastructure** — the new chosen direction.
* **First-class DB downgrade infrastructure** — the previous direction of this ADR: build a robust, tested runner whose primary job is to reliably reverse schema changes within an N-2 window.
* **Status quo** — no policy, no enforcement, no standard recovery path; rely on runbooks and snapshots in the moment.

## Decision Outcome

We will adopt **backward-compatible migrations as policy**, with a small amount of supporting infrastructure (a migration runner with a separate entrypoint) that decouples migration execution from webserver startup. The policy is the centerpiece; the infrastructure is auxiliary.

The policy, stated plainly: **every migration must leave the database in a state that the prior minor release's webserver can operate against correctly.** Specifically — for a release N, the schema after N's migrations are applied must still be a valid operating state for N-1 webservers. Here "N-1" means one minor version: concretely, the database after the `2.89.x` migrations have run must remain a valid operating state for a `2.88.0` webserver. So if a `2.89.x` upgrade goes wrong, an operator can roll the webserver image back to `2.88.x` against the already-migrated `2.89.x` database and expect it to work, with the DB schema untouched. (The window is measured in minor versions; patch releases within a minor are not separately bounded.) This makes app rollback a safe and supported recovery path, and as a co-benefit makes rolling deploys safe. The compatibility window is N-1 minor versions for now — we may extend to N-2 once the practice is mature and we have signal that the additional invariant is sustainable.

Destructive schema changes are not abandoned — they happen across multiple releases using the expand/contract pattern (release N adds new structure with dual-writes; N+1 migrates readers; N+2 stops writing the old structure; N+3 drops it). Authoring guidance will codify this.

We are *not* committing to a routinely-exercised DB downgrade path. The runner does support a break-glass schema rollback as an escape hatch for cases backward-compat cannot cover (e.g., a migration that contains a destructive data-transformation bug discovered after deploy), but it is explicitly not a routine operation — it is documented, gated by operator acknowledgement, and assumes a recent snapshot exists.

This option is chosen because the policy resolves the original recovery problem (app rollback becomes a real option) *and* enables zero-downtime upgrades *and* avoids the fragility of bidirectional schema machinery — using a single architectural commitment that's enforceable at PR time. The prior direction (first-class DB downgrade infrastructure) is rejected because the testing burden is high, the path rots when unused, and many legitimate schema changes cannot be made losslessly reversible regardless of how robust the runner is. Status quo is rejected because today's deployment posture treats "rollback" as a hope rather than a feature.

The implementation plan centers on the policy and its enforcement. A migration runner is useful supporting infrastructure but is explicitly **not** central to this decision — the backward-compat guarantee stands on its own, and the runner can be deferred or descoped without invalidating it. The plan is therefore split into the core (the policy and its enforcement) and optional supporting infrastructure (the runner).

#### Core — the policy and its enforcement

- **The compatibility policy, written down.** A short, normative document stating the N-1 backward-compat requirement, the rationale, the expand/contract pattern for destructive changes, and the boundary of acceptable migration authoring. Lives alongside the migration authoring guide.
- **CI lint for statically detectable violations.** Lint rules on the migrations directories of both `fides` and `fidesplus` flag:
  - Column drops within the compatibility window
  - `NOT NULL` additions without server defaults
  - Column or table renames (which are not expand/contract-safe without the multi-release pattern)
  - Type changes that are not transparently widening
  - Foreign-key constraint changes that would invalidate prior writes
  Violations block merge. Intentional exceptions require a PR-time annotation that surfaces in release notes and indicates the multi-release plan for completing the destructive change.
- **Per-release integration test for backward-compat.** A release-gating automated test (in our deployment test suite, e.g. Cardea) that deploys N-1 webservers, applies the N migrations against the same database, and asserts the N-1 webservers continue to serve representative traffic correctly against the new schema *without being rolled forward*. This is the empirical proof that the policy is upheld, and catches violations the lint misses.
- **Migration authoring guide.** Documented expand/contract patterns, worked examples of multi-release destructive changes, common pitfalls (ORM-side caching, default-value semantics, enum extensions), and review expectations. Distributed via the engineering handbook and linked from PR templates. Initial draft: [`src/fides/api/alembic/migrations/README.md`](/src/fides/api/alembic/migrations/README.md), collocated with the migrations.
- **Server tolerates schema/runtime version skew at startup.** App rollback only works if the rolled-back N-1 webserver actually *boots* against an N schema. The server must start cleanly — and pass readiness — when the database's applied migration revision is *ahead of* (or otherwise not equal to) the Alembic revision the running code was built against. It must not hard-fail, refuse to start, or block readiness solely because the DB migration state is misaligned with what the app runtime expects. This is the bootstrap-level corollary of the backward-compat policy and a prerequisite for both app rollback and rolling deploys (where mixed versions run against one schema simultaneously). Note this is independent of the runner: today's `automigrate`-at-startup behavior must be audited for any "schema is newer than I expect" failure paths, and those must degrade to a clean start rather than a crash.
- **Release/SLA documentation and customer communication.** The guarantee is only useful if operators know about it and stay within the window. We document the supported-rollback policy publicly (app rollback is supported within N-1) in customer-facing release/upgrade docs, and treat it as an SLA-level commitment for Fides Cloud. Crucially, the guarantee is *conditional* on customers upgrading minor-by-minor (landing on every minor release, or upgrading release by release): a deployment that jumps multiple minors at once steps outside the N-1 window, and for those, app rollback is **not** guaranteed — recovery falls back to heavier mechanisms (e.g., DB snapshot restore). We therefore actively encourage release-by-release upgrades and communicate this clearly to both self-hosted customers and Fides Cloud operators.
- **Documentation surface (recovery).** Runbook for the standard upgrade flow and runbook for app-rollback as the recovery procedure. (The break-glass schema-rollback runbook is tied to the optional runner, below.)

#### Optional / supporting — the migration runner

Not central to this decision; the runner strengthens the rollout (rolling deploys, clean migration observability, a break-glass escape hatch) but the policy does not depend on it, and it can be sequenced later. If built, it comprises:

- **A separate, minimal entrypoint** into the existing fides image — runs Alembic against the DB without exercising the webserver bootstrap. It earns its place by: letting us turn off `automigrate` at webserver startup (removing the hidden ordering dependency between schema rollout and webserver rollout, and enabling rolling deploys where migrations run cleanly *before* webservers roll); providing the venue for the break-glass schema rollback when backward-compat genuinely cannot cover a recovery scenario; and giving clean observability for migration runs (Job exit codes, K8s events, logs) versus migrations buried in webserver startup. One artifact, parameterized by direction (`upgrade` is the routine path; `downgrade` is the escape hatch) and target release version.
- **Operator-friendly versioning.** Operators parameterize the runner by release version (e.g., `--target-version=2.85.0`), not by Alembic revision hash. A release manifest committed in the repo at release time and bundled in every image resolves the version → Alembic head mapping locally. This makes both routine upgrades and the rare escape-hatch rollback expressible in customer-facing terms.
- **Per-orchestrator integration of the runner.** Thin invocation glue per orchestrator (Terraform `kubernetes_job_v1` in `modules/hosting_kubernetes`, a Job step in the Pulumi program for tenant-manager, a Job template in the `fides-helm` chart). The Job artifact is shared; only the invocation layer differs.
- **Break-glass runbook and templates.** The break-glass runbook for the escape-hatch schema rollback, plus the updated Terraform / Pulumi / chart templates that wire in the Job.

### Consequences

* Good — App rollback becomes a real, supported, tested recovery path. The catch-22 of needing a healthy webserver to recover from a broken webserver is gone.
* Good — Zero-downtime rolling deploys become possible by extension, since the same property — prior webserver tolerates new schema — is the prerequisite for both.
* Good — A single architectural commitment (a policy) covers two problems (rollback + zero-downtime) that would otherwise need separate infrastructure.
* Good — Aligns with how mature SaaS shops operate; the patterns and pitfalls are well-trodden and we can borrow heavily from existing literature.
* Good — Avoids investing in bidirectional schema machinery that would rot in the gap between authoring and use.
* Bad — Significant cultural and process shift for migration authoring. Every migration now needs PR-time scrutiny against the compatibility policy; destructive changes that used to be single-PR are now multi-release sequences.
* Bad — Lint rules catch the static cases but not the semantic ones. Real backward-compatibility bugs (e.g., a NOT NULL column that's added with a default but where the prior webserver writes `NULL` explicitly) will still slip through; the integration test is the safety net but adds release-cycle time and a new class of release-blocking failure.
* Bad — The escape-hatch schema rollback path is real but de-emphasized. When a true rollback is needed (e.g., a destructive migration bug), the runbook is heavier than "just roll the webserver back," and customers may not have practiced it.
* Bad — The rollback guarantee is conditional on upgrade cadence. It only holds within the N-1 window, so customers who batch several minor releases into a single upgrade fall outside it and must fall back to heavier recovery (e.g., DB snapshot restore). This requires sustained customer communication and a cultural push toward frequent, incremental, release-by-release upgrades.
* Bad — Transitioning existing deployments from `automigrate=true` to runner-driven migrations is a behavior change with ordering implications and needs its own sequenced rollout plan. (Mitigated by the runner being optional — the policy and app-rollback guarantee land independently of this transition.)
* Neutral — Production tenants share an Aurora cluster (40+ tenants today). Backward-compat is per-schema, so the policy applies uniformly; but if app rollback is ever needed, the operational decision still affects all tenants on that cluster simultaneously. This is a property of the deployment topology, not of this design, but worth naming.

## Pros and Cons of the Options

### Backward-compatible migrations as policy, with the migration runner as supporting infrastructure

Pros:
* Resolves the recovery problem by making app rollback work, rather than by making schema rollback reliable — a far easier property to enforce.
* The property is enforceable at PR time (static lint + integration test) rather than only at incident time.
* Co-enables zero-downtime rolling deploys with no additional architectural commitment.
* Industry-mature; we can lean on existing patterns and tooling for expand/contract.
* The runner is small in scope — it executes migrations cleanly out of band, but is not on the hook for guaranteeing bidirectional correctness across an N-2 window.
* Operators recover by changing the application version, which is a familiar deployment-level operation, not a schema-level one.

Cons:
* Requires a sustained culture shift in how migrations are authored and reviewed. Will take real investment to land and to maintain.
* Destructive schema changes become multi-release sequences, slowing the pace of those specific changes (usually acceptable, occasionally painful).
* Some semantic backward-compat violations are not lint-detectable; we depend on the integration test as a safety net, and on engineer judgment.
* The break-glass schema rollback path remains, but is exercised rarely enough that it will need explicit drilling to stay credible.

### First-class DB downgrade infrastructure

Pros:
* Symmetric — every migration can be run forward and backward, conceptually clean.
* Doesn't require migration authors to reason about prior webserver behavior at PR time; the policy is "ship a working `downgrade()`," which is mechanically clear.
* Doesn't constrain destructive schema changes — they can land in a single release as long as the downgrade is written.

Cons:
* Reversibility you don't routinely use is reversibility you can't trust. The `downgrade()` paths rot between authoring and incident.
* Many real migrations cannot be losslessly reversed (column splits, normalization, data backfills). The policy forces either fakes-it-anyway downgrades, irreversibility annotations, or snapshot dependencies — none of which compose well.
* Testing the full N → N-2 round-trip is expensive and fragile; the matrix grows quickly.
* Does not, by itself, enable zero-downtime upgrades; that would still require a separate backward-compat commitment, meaning we'd end up needing both.
* Larger infrastructure surface area to build, maintain, and operate.

### Status quo

Pros:
* No new investment required.

Cons:
* The catch-22 remains: webserver-hosted `/admin/db/{action}` is the only first-class recovery path, and it requires the broken thing to be healthy enough to invoke.
* App rollback may or may not work depending on whether the migration happened to be backward-compatible. Recovery becomes a hope, not a feature.
* Zero-downtime upgrades remain impossible.
* The path forward only gets harder as the schema accumulates more uncoordinated changes.

## Out of Scope / Follow-ups

* **The full set of expand/contract conventions.** The authoring guide will spell these out; this ADR does not.
* **Lint tooling selection.** Open-source candidates exist (Squawk, Atlas, custom Alembic checks); the choice is implementation work that follows the policy decision.
* **Multi-tenant blast radius for app rollback.** Production tenants share an Aurora cluster; app rollback for one tenant doesn't affect schema, but it does mean a heterogeneous fleet temporarily — worth a separate operational treatment.
* **DB unavailability scenarios.** Out of scope; this ADR assumes DB availability and webserver/worker unavailability.
* **Break-glass schema rollback runbook.** Documented separately from the standard recovery runbook; not central to this decision.
* **Tenant-manager `refresh` evolution.** Today's refresh is destructive (drop DB + recreate release). A non-destructive variant that uses the runner is implementation work that follows this ADR.
