# Postmortem: ISO Build Broke Host Terminal (2026-05-09)

## Incident Summary

During the first real-world test of `scripts/alpine/build-khy-os-iso.sh`,
the build process was interrupted multiple times (via `kill` / task stop).
Each interruption left bind-mounts active, and the cleanup routine's `rm -rf`
destroyed host device nodes through the live bind-mount, rendering the
Linux terminal unusable.

## Timeline

| Time  | Event |
|-------|-------|
| 11:17 | First ISO build launched; killed due to DNS failure |
| 11:17 | `rm -rf /tmp/khy-os-rootfs.*` ran while `/dev` was still bind-mounted → **`/dev/null` destroyed** (became a regular file) |
| 11:49 | User reported: "Failed to open PTY: No such file or directory" |
| 11:53 | `/dev/null` restored via `mknod /dev/null c 1 3; chmod 666 /dev/null` |
| 11:54 | Second build launched; `mount -t devpts devpts …/dev/pts` **overwrote host `/dev/pts`** with `ptmxmode=000` |
| 11:54 | User reported: "Text was empty (or contained only whitespace)" |
| 11:56 | Host `/dev/pts` remounted with correct options; GNOME Terminal profile had `use-custom-command=true` but empty command; reset to `false` |
| 11:58 | Terminal fully restored |

## Root Causes

### 1. `rm -rf` through live bind-mount

```bash
# DANGEROUS: if /dev is still bind-mounted, this deletes HOST device nodes
mount --bind /dev "${ROOTFS}/dev"
# ... build interrupted ...
rm -rf "${STAGING}"   # deletes ${ROOTFS}/dev/* = HOST /dev/*
```

**Impact**: `/dev/null` and `/dev/ptmx` were deleted from the host.

### 2. `devpts` mount clobbered host

```bash
# DANGEROUS: mounts a NEW devpts over the host's /dev/pts
mount -t devpts devpts "${ROOTFS}/dev/pts"
```

Without `newinstance`, this replaces the host's devpts mount with a new
instance using restrictive defaults (`ptmxmode=000`), preventing any new
PTY allocation system-wide.

### 3. No safety check before destructive cleanup

The `cleanup()` function assumed `umount` always succeeds, then
unconditionally ran `rm -rf`. On interrupted builds, `umount` can fail
silently (target busy, already unmounted by sub-process, etc.).

## Fixes Applied (commit 3f70078)

### Fix 1: Mount isolation

```bash
mount --bind /dev "${ROOTFS}/dev"
mount --make-rslave "${ROOTFS}/dev"          # NEW: prevent propagation
mount -t devpts devpts "${ROOTFS}/dev/pts" \
  -o newinstance,gid=5,mode=620,ptmxmode=666 # NEW: isolated instance
```

- `--make-rslave`: unmount events inside chroot do not propagate to host
- `newinstance`: creates a separate devpts filesystem, host `/dev/pts` untouched

### Fix 2: Safety check before rm

```bash
cleanup() {
  umount -lf "${ROOTFS}/dev/pts" 2>/dev/null || true
  umount -lf "${ROOTFS}/dev"     2>/dev/null || true
  # ...
  if mountpoint -q "${ROOTFS}/dev" 2>/dev/null; then
    warn "dev still mounted! Skipping rm to protect host."
    return 1
  fi
  rm -rf "${STAGING}"
}
```

### Fix 3: Proper unmount order

Changed to unmount in reverse order: `proc → sys → dev/pts → dev`.

## Recovery Commands

If this happens again, restore with:

```bash
# Restore /dev/null
sudo mknod /dev/null c 1 3
sudo chmod 666 /dev/null

# Restore /dev/ptmx
sudo mknod /dev/ptmx c 5 2
sudo chmod 666 /dev/ptmx

# Fix /dev/pts permissions
sudo mount -o remount,gid=5,mode=620,ptmxmode=666 /dev/pts

# If GNOME Terminal shows "Text was empty"
dconf write /org/gnome/terminal/legacy/profiles:/:b1dcc9dd-5262-4d8d-a863-c897e6d979b9/use-custom-command false
```

## Lessons Learned

1. **Never `rm -rf` a directory tree that contains bind-mounts** — always
   verify with `mountpoint -q` first.
2. **Always use `newinstance` when mounting devpts in chroot** — otherwise
   the host PTY subsystem is silently replaced.
3. **Use `--make-rslave`** on bind-mounts to prevent event propagation
   between chroot and host.
4. **Interrupted builds are the norm, not the exception** — cleanup must be
   robust against partial state.
