Metadata-Version: 2.4
Name: unrender
Version: 0.2.8
Summary: Reconstruct editor-friendly timelines from flattened final renders.
Author: Muhammad Hadi Yusufali
Maintainer: Muhammad Hadi Yusufali
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/mhadifilms/unrender
Project-URL: Repository, https://github.com/mhadifilms/unrender
Project-URL: Issues, https://github.com/mhadifilms/unrender/issues
Project-URL: Changelog, https://github.com/mhadifilms/unrender/blob/main/CHANGELOG.md
Keywords: audio,video,source-separation,diarization,speaker-identification,post-production,film,dubbing,media
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.24
Requires-Dist: opencv-python>=4.8
Requires-Dist: Pillow>=10
Requires-Dist: requests>=2.31
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Provides-Extra: face
Requires-Dist: insightface; extra == "face"
Requires-Dist: onnxruntime; extra == "face"
Provides-Extra: voice
Requires-Dist: pyannote.audio; extra == "voice"
Requires-Dist: torchaudio; extra == "voice"
Provides-Extra: transcription
Requires-Dist: whisperx; extra == "transcription"
Provides-Extra: scripts
Requires-Dist: openpyxl; extra == "scripts"
Requires-Dist: python-docx; extra == "scripts"
Provides-Extra: clone
Requires-Dist: mlx-audio>=0.4.4; extra == "clone"
Provides-Extra: timeline
Requires-Dist: opentimelineio>=0.17; extra == "timeline"
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: opentimelineio>=0.17; extra == "dev"
Requires-Dist: pre-commit>=3.5; extra == "dev"
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

# Unrender

[![CI](https://github.com/mhadifilms/unrender/actions/workflows/ci.yml/badge.svg)](https://github.com/mhadifilms/unrender/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)

Unrender reconstructs an editor-friendly set of media artifacts from a flattened
final render. It separates the final mix into useful layers, identifies speakers,
cuts dialogue into reusable units, maps those units back onto shots, and writes
JSON/CSV plans plus stem files that an editor or downstream tool can consume.

The target shape is:

- video organized by shot
- dialogue as dialogue-line and shot-level stems
- music as its own source layer
- effects as their own source layer
- speaker identity tracked through one shared speaker database

## Install

Unrender requires Python 3.10+ and a working `ffmpeg` on your `PATH`.

Install the latest version directly from GitHub:

```bash
python3 -m pip install "git+https://github.com/mhadifilms/unrender.git"
```

For development from a local checkout:

```bash
python3 -m venv .venv
./.venv/bin/python -m pip install -e ".[dev]"
```

Install only the media extras you need:

```bash
./.venv/bin/python -m pip install -e ".[face,voice,transcription,scripts]"
```

Optional extras:

- `face`: real face detection and embeddings
- `voice`: real voice embeddings
- `transcription`: WhisperX dialogue transcription
- `scripts`: XLSX/DOCX dub script parsing
- `clone`: MLX/Qwen3-TTS voice clone synthesis on Apple Silicon
- `timeline`: OpenTimelineIO timeline reconstruction (`unrender timeline build`)

The optional BandIt/MSST source separation backend is installed separately
(its dependency is not on PyPI, so it cannot be a package extra).

AudioShake separation requires an API key:

```bash
export AUDIOSHAKE_API_KEY=...
```

The BandIt backend installs upstream Music-Source-Separation-Training and does
not require a local checkout:

```bash
./.venv/bin/python -m pip install "music-source-separation-training[bandit] @ git+https://github.com/ZFTurbo/Music-Source-Separation-Training.git@ccc011abf7f89dd7922bb2888d48493b575c0289"
unrender audio separate -p show-a --source-backend bandit
```

By default, `unrender` uses the BandIt/MSST package config and downloads the
BandIt Plus checkpoint into the user cache, verifying its checksum before use.
Use `--bandit-config` or `--bandit-checkpoint` to override those paths. On Apple
Silicon, the runtime casts unsupported float64 model state to float32 before
moving the model to MPS, so no patched MSST checkout is required.

## Inputs

Most runs start from these inputs:

- `speakers.json`: canonical speaker names and aliases
- `proxy_master.mov`: reviewable master/proxy for face matching
- `full_audio.wav`: full mix or full audio stem
- `shots.csv`: shot IDs, video paths, and start/end seconds

If you already have an isolated dialogue stem, use `dx_stem` or `--dx-stem`
instead of `full_audio`.

Example `speakers.json`:

```json
{
  "speakers": {
    "JORDAN": { "aliases": [] },
    "ALEX": { "aliases": ["ALEKS"] }
  }
}
```

Manual labels are constrained to the configured speaker set. Aliases resolve to
canonical names.

Example `shots.csv`:

```csv
shot_id,video_path,start_sec,end_sec
001,/path/to/shot_001_proxy.mov,102.25,106.75
002,/path/to/shot_002_proxy.mov,215.10,218.30
```

For tighter face matching, add a target face box:

```csv
shot_id,video_path,start_sec,end_sec,target_face_box
001,/path/to/shot_001_proxy.mov,102.25,106.75,"[320, 140, 180, 220]"
```

Supported target box columns are `target_face_box`, `face_box`, `bbox`, or split
columns `target_x,target_y,target_w,target_h`.

## Project Config

Use `-p` / `--project` to avoid repeating paths. Project configs live under
`config/` by default, and the `.json` suffix is optional.

```bash
unrender doctor -p demo-spanish
unrender audio separate -p demo-spanish
unrender shots match -p demo-*
```

Example `config/demo-spanish.json`:

```json
{
  "speakers": {
    "JORDAN": { "aliases": [] },
    "ALEX": { "aliases": [] }
  },
  "paths": {
    "run_dir": "runs/demo-spanish",
    "proxy_master": "/path/to/proxy_master.mov",
    "full_audio": "/path/to/full_audio.wav",
    "dx_stem": "/path/to/dialogue_stem.wav",
    "dub_script": "/path/to/dub_script.csv",
    "shots": "config/demo-spanish-shots.csv",
    "separated_stems": "/path/to/stems/*.wav"
  },
  "stem_map": {
    "JORDAN": "/path/to/stems/JORDAN_stem.wav",
    "ALEX": "/path/to/stems/ALEX_stem.wav"
  },
  "audio": {
    "audioshake_variant": "n_speaker",
    "silence_threshold_db": -100,
    "max_gap_sec": 0.5,
    "handle_sec": 0.18,
    "clip_resolver": "cluster",
    "min_margin": 0.05,
    "override_sec": 1.5,
    "override_margin": 0.15,
    "cut_shot_stems": false
  },
  "face": {
    "interval": 2.0,
    "min_confidence": 0.5,
    "min_face_px": 36
  },
  "shots": {
    "samples_per_shot": 8,
    "sim_threshold": 0.45,
    "min_confidence": 0.5,
    "min_votes": 2,
    "min_margin": 0.05
  },
  "voice": {
    "cluster_threshold": 0.35,
    "min_cluster_size": 2
  },
  "timeline": {
    "dialogue": "auto",
    "shot_dx": false,
    "no_markers": false
  }
}
```

Every CLI tuning flag can also be set in the matching config section
(`audio`, `face`, `shots`, `voice`, `voice_clone`, `timeline`); explicit CLI
flags always win.

If `speakers` is omitted and `paths.dub_script` is set, Unrender derives the
speaker list from the script. CSV, TSV, JSON, XLSX, and DOCX dub scripts are
supported when the relevant extras are installed.

Explicit CLI flags override project config values.

## Main Workflow

With a project config:

```bash
unrender doctor -p demo-spanish
unrender face build -p demo-spanish
unrender audio separate -p demo-spanish
unrender audio transcribe-lines -p demo-spanish
unrender audio build-clips -p demo-spanish
unrender voice build -p demo-spanish
unrender labels interactive -p demo-spanish
unrender shots match -p demo-spanish
unrender audio resolve-clips -p demo-spanish
unrender audio map-dialogue -p demo-spanish
unrender voice clone -p demo-spanish
unrender timeline build -p demo-spanish
unrender export artifacts -p demo-spanish
unrender status -p demo-spanish
```

Without a project config:

```bash
RUN=/tmp/unrender-show

unrender face build --video proxy_master.mov --run-dir "$RUN"
unrender audio separate --full-audio full_audio.wav --run-dir "$RUN"
unrender audio transcribe-lines --dx-stem dialogue.wav --shots shots.csv --run-dir "$RUN"
unrender audio build-clips --run-dir "$RUN"
unrender voice build --run-dir "$RUN"
unrender labels interactive --run-dir "$RUN" --config speakers.json
unrender shots match --shots shots.csv --speaker-db "$RUN/speaker_db.json" --run-dir "$RUN"
unrender audio resolve-clips --speaker-db "$RUN/speaker_db.json" --run-dir "$RUN"
unrender audio map-dialogue --shots shots.csv --run-dir "$RUN"
unrender timeline build --shots shots.csv --run-dir "$RUN"
unrender export artifacts --run-dir "$RUN"
unrender status --run-dir "$RUN"
```

The primary deliverable is per-speaker full stems split into labeled,
speaker-named dialogue-line clips — that is what the timeline carries.

## Optional Per-Shot Outputs

Two optional final steps produce per-shot artifacts on top of the primary
dialogue-line flow:

```bash
# One DX clip per shot, cut from the full dialogue stem. The DX source is
# resolved automatically: an explicit --dx-stem, the source-separation
# output, or (when neither exists) a stem merged from the separated
# per-speaker stems. Add to the timeline as one track with --shot-dx.
unrender audio shot-dx -p demo-spanish
unrender timeline build -p demo-spanish --shot-dx

# Per-shot speaker stems: one file per dialogue line overlapping each shot,
# cut exactly to the shot timecode (2+ files when 2+ lines fall in a shot).
# Include them in the timeline explicitly with --dialogue shots.
unrender audio map-dialogue -p demo-spanish --cut-shot-stems
```

## Commands

```text
unrender audio separate         Separate full audio/DX into source and speaker stems
unrender audio transcribe-lines Build dialogue_lines.json from WhisperX or a dub script
unrender audio build-clips      Cut per-line voice clips from full-length speaker stems
unrender audio resolve-clips    Resolve dialogue clips to labeled voices. The default
                                cluster resolver groups every line's voice globally and
                                jointly assigns clusters to speakers (long, confident
                                clips can override); --resolver clip matches each clip
                                independently with a similarity threshold and margin
unrender audio map-dialogue     Map resolved dialogue lines onto shots; optionally cut
                                shot-timecode-accurate per-line stems (--cut-shot-stems)
unrender audio shot-dx          Cut a per-shot DX stem from the full dialogue stem
                                (given, separated, or merged from speaker stems)

unrender face build             Build face_db.json and face review grids
unrender voice build            Build voice_db.json and voice review samples
unrender voice match            Match arbitrary clips to labeled voices
unrender voice clone            Build voice clone references and optional samples

unrender labels template        Write labels.csv for manual review
unrender labels apply           Apply labels.csv to speaker_db.json
unrender labels interactive     Prompt through unlabeled face/voice clusters

unrender shots match            Identify on-screen speakers per shot
unrender timeline build         Reconstruct an OpenTimelineIO timeline (.otio)
unrender export artifacts       Copy final JSON artifacts under exports/
unrender status                 Show expected run artifacts
unrender doctor                 Validate inputs and local prerequisites
```

Run `unrender <group> <command> --help` for command-specific flags.

## Run Directory

A run directory is the working output area for one project:

```text
runs/show/
  face_db.json
  voice_db.json
  speaker_db.json
  dialogue_lines.json
  dialogue_lines.csv
  labels.csv
  shot_speaker_matches.json
  shot_speaker_matches.csv
  voice_speaker_matches.json
  clip_stem_plan.json
  dialogue_stem_plan.json
  shot_dialogue_map.json
  shot_stem_plan.json
  timeline.otio
  audio/
    source_stems/
    unmapped_speakers/
    voice_clips/
    dialogue_mapped/
    _unmapped/
    mapped/
    voice_clone_data/
  review/
    face/
    voice/
  voice_clones/
  exports/
```

Build steps are restartable. Existing artifacts are reused by default; pass
`--force` when you intentionally want to rebuild a step.

## Timeline Reconstruction

The final step reassembles the reconstructed media into an editor timeline using
[OpenTimelineIO](https://opentimeline.io):

```bash
./.venv/bin/python -m pip install -e ".[timeline]"
unrender timeline build -p demo-spanish
```

The resulting `timeline.otio` contains:

- a video track per shot, placed at each shot's `start_sec`/`end_sec`, marked
  with the matched on-screen speaker(s)
- one dialogue track per speaker, ordered by speaking time, with line-level
  stems (or per-shot stems) placed at their global time; overlapping lines fan
  out onto additional lanes
- music (`MX`) and effects (`FX`) tracks from the full-length source stems
- editorial markers on each dialogue clip showing the speaker and spoken line,
  color-coded: green (matched), orange (off-screen / unresolved), red (missing
  stem)
- a program start timecode and provenance metadata (Unrender version, run
  directory, fps, dialogue source) on the timeline

Clip durations are probed from the real media with `ffprobe` (falling back to
the stdlib `wave` module for `.wav` stems, then to artifact timing if `ffprobe`
is unavailable), so each clip carries an accurate `available_range`. Clips
reference the stem and proxy files on disk; OTIO stores the cut, not the media.
Lines without a matched stem are kept as placeholders so the editorial
structure stays intact.

Useful flags:

- `--dialogue {auto,lines,shots}` — choose the dialogue source granularity
- `--fps` — edit rate (default 24, or `audio.fps` from the project config)
- `--start-timecode HH:MM:SS:FF` — set the timeline origin (e.g. `01:00:00:00`)
- `--dialogue-bed` — add the full-length dialogue source stem as a track
- `--no-source-stems` / `--no-markers` — drop music/effects tracks or markers
- `--dry-run` — report the plan without writing
- `--bundle` — also write a self-contained `.otiod` media bundle

`.otio` is written by the built-in adapter. To export other formats (EDL,
Final Cut Pro XML, AAF), install the adapters and pass a matching `--out`
suffix:

```bash
pip install OpenTimelineIO-Plugins
unrender timeline build -p demo-spanish --out runs/demo-spanish/cut.edl
```

## Development

Run checks:

```bash
./.venv/bin/ruff check .
./.venv/bin/black --check .
./.venv/bin/pytest -q
```

Format:

```bash
./.venv/bin/black src tests
```

Type-check:

```bash
./.venv/bin/mypy
```

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and
the checks that must pass, and [SECURITY.md](SECURITY.md) for how to report
vulnerabilities privately.

## License

Unrender is licensed under the [Apache License 2.0](LICENSE).

Optional, separately installed backends (BandIt/MSST, pyannote.audio, WhisperX,
InsightFace, and others) are governed by their own licenses and model
terms. They are not bundled with Unrender; review each project's terms before
use, especially for commercial work. See [NOTICE](NOTICE) for details.
