# Authoring Backward-Compatible Migrations

Migrations in this directory must be **backward compatible**: after a release's
migrations run, the _previous_ minor release's webserver must still operate
against the schema correctly. This is what makes application rollback a safe
recovery path — roll the app image back, leave the schema untouched — and it is
the prerequisite for zero-downtime rolling deploys. See the
[ADR](/docs/ADRs/2026-05-19-fides-migration-backward-compatibility.md) for the
full rationale.

## The policy

For a release N, the schema after N's migrations must remain a valid operating
state for an **N-1** (one minor version back) webserver.

Concretely: after the `2.89.x` migrations run, a `2.88.0` webserver must still
read and write the schema correctly — so a bad `2.89.x` upgrade can be recovered
by rolling the app image back to `2.88.x` against the already-migrated database.
The window is measured in minor versions; patch releases within a minor are not
separately bounded.

## Rule of thumb

> Never remove, rename, or narrow something in place. Add the new shape, move
> readers and writers across releases, and drop the old shape only once nothing
> within the compatibility window depends on it.

Keep this and the rest mostly takes care of itself.

## Safe vs. unsafe in one release

A change is **safe** if a webserver built _before_ the migration ran can still
operate against the resulting schema.

| Change                                                    | Safe in one release? | Notes |
| --------------------------------------------------------- | :------------------: | ----- |
| Add a **nullable** column                                 | ✅ | Old code ignores it. |
| Add a column **with a server-side default**               | ✅ | Old code's inserts get the default. |
| Add a new table                                           | ✅ | Old code ignores it. |
| Add an index (`CONCURRENTLY`)                             | ✅ | See [recipes](#recipes). |
| Widen a type transparently (`varchar(50)` → `text`)       | ✅ | Old reads/writes stay valid. |
| Add a **`NOT NULL`** column **without** a server default  | ❌ | Old inserts omit it → constraint violation. |
| Drop a column / table                                     | ❌ | Old code may still read or write it. |
| Rename a column / table                                   | ❌ | Equivalent to drop + add. |
| Narrow a type (`text` → `varchar(50)`, `bigint` → `int`)  | ❌ | Old code may write values that no longer fit. |
| Add a `CHECK` / `NOT NULL` / `UNIQUE` constraint or FK    | ❌* | Old write patterns may violate it. |

`*` = safe only once you've confirmed no code path within the N-1 window produces
a value the new constraint would reject. When in doubt, treat it as unsafe.

Anything in the ❌ rows must be split across releases using expand/contract.

## Expand/contract for destructive changes

Destructive changes aren't abandoned — they're spread across releases so every
intermediate schema stays operable by the prior webserver:

1. **Expand** — add the new structure alongside the old, backward-compatibly.
2. **Migrate** — dual-write, backfill, then move readers to the new structure.
3. **Contract** — once nothing within the window uses the old structure, remove it.

Each step is its own release and its own backward-compatible migration.

### Worked example: renaming `user.name` → `user.full_name`

| Release | Migration (schema)                                    | Application code |
| ------- | ----------------------------------------------------- | ---------------- |
| **N**   | Add nullable `full_name`; backfill `full_name = name` | Write **both**; read `name`. |
| **N+1** | —                                                     | Read `full_name`; still write both. |
| **N+2** | —                                                     | Write/read only `full_name`. |
| **N+3** | Drop `name`.                                          | Only references `full_name`. |

At every step the previous release's webserver still works against the schema.
`name` is dropped only in N+3, by which point no webserver within the window
touches it. You can compress the read/write-switch releases, but **never**
collapse "add new" and "drop old" into the same release.

Other destructive changes follow the same shape: making a column `NOT NULL` (add
nullable + backfill, then enforce next release), changing a column type (add new
column, dual-write + backfill, switch readers, drop old), or splitting a table.

## Recipes

**Adding a column** — always nullable, or with a **server-side** default. A
`server_default` (in the DB) is backward-compatible; a Python-side ORM `default=`
is not, because the prior webserver doesn't have it.

```python
op.add_column("user", sa.Column("full_name", sa.String(), nullable=True))
```

**Adding a required column** — two releases: add nullable + backfill (and make
N's code always write it), then enforce `NOT NULL` in N+1.

**Indexes** — create them without locking the table. Alembic runs migrations in a
transaction, which `CREATE INDEX CONCURRENTLY` forbids, so open an autocommit block:

```python
with op.get_context().autocommit_block():
    op.create_index(
        "ix_user_full_name", "user", ["full_name"], postgresql_concurrently=True
    )
```

**Enums** — adding a value is fine, but don't *write* the new value until the
following release so the prior webserver never reads a value it can't handle.
Never remove an enum value within the window. Modeling the field as a plain
string sidesteps enum migration and compatibility entirely.

## Application-side pitfalls

The schema is only half of compatibility — the prior webserver has to keep working:

- **ORM column references.** Code that selects explicit columns will error on a
  column that's been dropped. This is why "stop reading" must precede "drop" by a
  release.
- **Default-value semantics.** A server default only applies to inserts that omit
  the column. If the prior webserver writes the column explicitly as `NULL`, a
  `NOT NULL`+default column still fails — confirm it never writes `NULL`.
- **Constraint-violating writes.** Before adding a `CHECK`, `UNIQUE`, or FK, audit
  the prior release's writes; it may legitimately produce values the constraint
  rejects.

## Review checklist

- [ ] No column/table is dropped or renamed in the same release it stops being used.
- [ ] New columns are nullable **or** have a **server-side** default.
- [ ] New `NOT NULL` / `CHECK` / `UNIQUE` / FK constraints are split across
      releases, with data and write paths already conforming.
- [ ] Type changes are transparently widening, or done via expand/contract.
- [ ] Index creation uses `CONCURRENTLY` (outside the migration transaction).
- [ ] The prior release's webserver can read and write the resulting schema.
