# Fidesplus

Fidesplus is a monorepo containing the various premium offerings of the Fides ecosystem. These offerings are primarily distributed as Docker images, and therefore each published image is considered its own application. The available applications are as follows:

- **Slim**: This is the standard extension of Fides that includes the visual datamap, lightweight-masking regex engine, lightweight Fidescls (_classify_) implementation, and more.
- **Fidesplus**: This is a superset of `slim`. It contains all of Fidesplus as well as some more advanced Natural Language Processing libraries to increase the accuracy and functionality of the classification related features.
- **Masking**: (DEPRECATED) This is a lightweight wrapper around PrivateAI, an application that provides additional functionality around masking PII. This application has been deprecated as of August 2025, but as it is still in use by one Ethyca client, we've not fully removed it from the codebase yet.

_Note: Most local development is done using the `slim` application for performance reasons, especially on M1 machines._

All of these applications can be built and tested using [`uv`](https://docs.astral.sh/uv/) and the `nox` sessions in this repo (`uv run nox …`).

## Monorepo layout

This repository is a **monorepo**: core OSS Fides and Fidesplus live in one tree. You do not need a separate checkout of [ethyca/fides](https://github.com/ethyca/fides).

| Component | Path |
|---|---|
| Fides (Python package `fides`) | `src/fides/` |
| Fidesplus server | `src/fidesplus/` |
| Admin UI | `clients/admin-ui/` |
| Privacy Center | `clients/privacy-center/` |
| Tests (Fidesplus) | `tests/fidesplus/` |
| Tests (Fides core) | `tests/fides/` |

Dependencies are managed with **uv** from a single root `pyproject.toml` (`uv.lock`). Install everything with:

```sh
uv sync --all-groups
```

## Getting Started

Follow these steps for our standard development environment:

1. Install these system requirements locally:
   * [Docker](https://www.docker.com/products/docker-desktop) (version 20.10.11 or later)
   * [Python](https://www.python.org/downloads/) (version 3.13)
   * [uv](https://docs.astral.sh/uv/getting-started/installation/)
   * [Cursor](https://cursor.com/home) or [VSCode](https://code.visualstudio.com/) as your IDE
   * [1Password CLI](https://developer.1password.com/docs/cli/get-started/) (`op`) — required for automatic environment seeding (see [Environment Setup & Seeding](#environment-setup--seeding) below). If you don't have 1PW CLI, you can still run the dev environment without seeding by passing `-- no-seed`.

2. Clone [ethyca/fidesplus](https://github.com/ethyca/fidesplus) and sync dependencies from the repo root:

```sh
git clone https://github.com/ethyca/fidesplus.git
cd fidesplus
uv sync --all-groups
```

3. Open the repository root in your editor.

4. Build and run the stack in hot-reloading dev mode (slim backend + Admin UI + Privacy Center):

```sh
uv run nox -s 'build(slim)' 'dev(slim)' -- dev fides-admin-ui fides-privacy-center
```

Or use the command palette (`Cmd+Shift+P`) → `Tasks: Run Task` → `Run Fidesplus & Fides (DEV)` if your editor is configured with the repo tasks.

| Project | Source directory | Local URL |
|---|---|---|
| `fides` package | `src/fides/` | n/a |
| `fidesplus` server | `src/fidesplus/` | http://localhost:8080 |
| Admin UI | `clients/admin-ui/` | http://localhost:3000 |
| Privacy Center | `clients/privacy-center/` | http://localhost:3001 |

Changes under `src/fides`, `src/fidesplus`, and the mounted client trees reload in the running containers.

WARNING: The client applications (`fides-admin-ui` and `fides-privacy-center`) are built on-demand when you first visit them in your web browser. You will see log messages in the Docker logs like the example below, and these can be quite slow to compile within the Docker container:
```
2025-08-18 13:26:09 admin-ui:dev:  ○ Compiling /login ...
2025-08-18 13:26:49 admin-ui:dev:  ✓ Compiled /login in 40.4s (6498 modules)
```
...as a result, you may find this approach too cumbersome to quickly work on the client applications. For instructions on how to build and run the UIs outside Docker, see `clients/admin-ui/README.md` and `clients/privacy-center/README.md`.

NOTE: When using `uv run nox -s "dev(slim)" -- demo`, the Admin UI and Privacy Center run from **pre-built Docker images** that do not hot-reload or rebuild automatically. To incorporate local changes, you must manually rebuild the images:
```sh
docker compose -f docker-compose.yml -f demo/docker-compose.demo.yml build fides-admin-ui fides-privacy-center
```
The Admin UI also runs a Next.js production build on container startup that takes a few minutes, so it won't be immediately accessible even after the container reports healthy — look for `✓ Ready in Xms` in the container logs. This build step is memory-intensive — allocate at least **12 GB of memory to Docker Desktop**.

## Developing Fidesplus

This section is a brief overview of the available development commands and resources available in this repository. Remember that you can also rely on `uv run nox` (no args) and `uv run nox -s usage -- <session_name>` to learn more about the available commands.

### Shell Setup (optional, zsh)

Tired of typing `uv run nox`? Install the shell integration once:

```sh
uv run nox -s shell_setup
```

This adds a `nox` wrapper function to your `~/.zshrc` that runs `uv run nox` under the hood (so sessions always see a synced project venv), plus tab-completion for session names — `nox -s <TAB>` lists every available session. The integration lives in `scripts/nox-uv.zsh`, so improvements ship automatically with `git pull`; re-run `shell_setup` only if you move your clone.

### Listing the Applications and their Information

The full Application list, including all related information, can be viewed with the following command:

```sh
uv run nox -s display_applications
```

### Building the Applications

For the sake of simplicity, one can build _all_ permutations of a specific Application by running:

```sh
uv run nox -s "build(<application>)" -- all
```

Real Example:

```sh
uv run nox -s "build(slim)" -- all
```

Or if you want to build a specific version:

```sh
uv run nox -s "build(<application>)" -- <build_name>
```

Real Example:

```sh
uv run nox -s "build(slim)" -- dev
```

### Running the Applications

To run the default local dev stack (slim image + worker), use:

```sh
uv run nox -s dev
```

`dev` always starts the **slim monorepo stack** — it does not iterate over every application image. For other images (full fidesplus, masking, sidecar), use:

```sh
uv run nox -s "dev_app(<application>)"
```

`dev(slim)` remains a backward-compatible alias for `dev`.

By default, `uv run nox -s dev` starts services **in the background** (detached mode), waits for them to become healthy, and then **automatically invokes the seed API** to configure the environment. Once complete, it prints a "DEV ENVIRONMENT READY" banner with useful commands (tail logs, stop containers, re-seed, etc.) and exits — leaving the containers running.

The automatic seeding is **best-effort**: if 1Password CLI is not available or seeding fails for any reason, a warning is printed but the environment stays up and usable. To skip seeding entirely:

```sh
uv run nox -s dev -- no-seed
```

See [Environment Setup & Seeding](#environment-setup--seeding) below for details on how seeding and `.env` validation work.

Due to `dev` no longer handling the build process, you can chain commands together for ease-of-use. The following command will build and then spin up the slim application.

_Note: order matters here!_

```sh
uv run nox -s "build(slim)" dev -- dev
```

---

### Running Multiple Instances in Parallel (Git Worktrees)

When working across multiple git worktrees simultaneously — for example with agentic coding workflows or parallel feature branches — each worktree can run its own fully isolated stack with no port conflicts.

**It just works from a worktree.** When `nox -s "dev(slim)"` is run from a linked worktree (any path other than the main `fidesplus/` checkout), it automatically detects the worktree, derives a unique `WORKTREE_ID` from the branch name, and assigns isolated ports and Docker resource names. No configuration required.

```sh
# In your main checkout — uses default ports (8080, 5432, 6379, …)
cd ~/<where-your-code-lives>/fidesplus
nox -s dev

# In a linked worktree — auto-detected, gets its own ports
cd ~/<where-your-code-lives>/.worktrees/fidesplus-my-feature
nox -s dev
# Logs will show:
#   WORKTREE_ID:          my-feature
#   PORT_OFFSET:          <hash-derived>
#   COMPOSE_PROJECT_NAME: fidesplus-my-feature
#   API port:             <offset port>
```

**All Docker resources are fully isolated per worktree:**
- Named volumes (`fidesplus-my-feature_postgres`) — each worktree has its own database
- Container names (`fidesplus-slim-my-feature`)
- Docker network (`fidesplus-my-feature_default`)
- Host ports — offset from defaults so nothing collides

**Manual override** (pin a specific slot or resolve a hash collision):

```sh
# in fidesplus/.env within the worktree
WORKTREE_ID=my-feature   # short slug; auto-detected from branch if omitted
PORT_OFFSET=100          # explicit slot; hash-derived from WORKTREE_ID if omitted
```

**Slot reference table:**

| Slot | PORT_OFFSET | API  | Admin UI | Privacy Center | DB   | Redis |
|------|-------------|------|----------|----------------|------|-------|
| 0 (main) | 0   | 8080 | 3000     | 3001           | 5432 | 6379  |
| 1    | 100         | 8180 | 3100     | 3101           | 5532 | 6479  |
| 2    | 200         | 8280 | 3200     | 3201           | 5632 | 6579  |
| 3    | 300         | 8380 | 3300     | 3301           | 5732 | 6679  |

Each worktree seeding is independent — run `nox -s seed` in each worktree after first start.

---

### Connecting to the Application

To connect to the running slim dev container:

```sh
uv run nox -s shell
```

For other application images:

```sh
uv run nox -s "shell_app(<application>)"
```

### Working on Fides core (`src/fides`)

The Fidesplus server reloads when you change `src/fidesplus` or `data/`. Edits to **`src/fides/`** are picked up the same way in dev: the repo is bind-mounted into the container and uvicorn reload watches the tree. No sibling fides checkout or `FIDES_REPO_PATH` is required.

Start the stack as usual:

```sh
uv run nox -s dev
```

You can use `pdb.set_trace()` in either package while the slim container is running. Fides core and Fidesplus changes ship together in one PR on this monorepo.

### Environment Setup & Seeding

The default `uv run nox -s dev` session enforces a reference `.env` as the baseline local dev state. This ensures that all required environment variables — API keys, service credentials, feature flags — are present before seeding runs. The flow works as follows:

#### 1Password CLI (prerequisite)

The [1Password CLI](https://developer.1password.com/docs/cli/get-started/) (`op`) is effectively a prerequisite for local Fidesplus development with seeding enabled. It is used for two things:

1. **`.env` validation** — On startup, the dev session fetches a reference `.env` from 1Password (vault: `fidesplus-demo-seed`, item: `fidesplus-demo-dot-env`) and validates that every variable it defines is present in your local environment (either in your shell or in the repo-root `.env` file).
2. **Seed API invocation** — The seed API reads secrets (e.g., Mailgun API keys, S3 credentials) from 1Password at runtime to configure the environment.

To install and configure:

```sh
# Install (macOS)
brew install 1password-cli

# Sign in to your Ethyca account
op signin
```

If you don't have 1PW CLI installed or configured, you can bypass seeding entirely:

```sh
uv run nox -s "dev(slim)" -- no-seed
```

For full details on seed prerequisites, troubleshooting, and the seed API, see [`src/fidesplus/seed/README.md`](src/fidesplus/seed/README.md).

#### Reference `.env` and validation

When `uv run nox -s dev` runs (without `no-seed`), it:

1. Fetches the reference `.env` from 1Password
2. Compares its variables against your local `.env` file and shell environment
3. If variables are missing, **offers to pull the full reference `.env`** from 1Password automatically (backing up your existing `.env` first)
4. If you decline or the pull fails, aborts with a clear error listing the missing variables

This ensures your local environment matches the expected configuration before containers start and seeding runs.

> **Note:** After a `uv run nox -s teardown` (or `docker compose down`), the next `uv run nox -s dev` will re-seed the environment automatically, since the seed state is not persisted across container recreations.

#### Demo mode

The `demo` posarg starts a full demo environment that includes the Fidesplus backend, Admin UI, Privacy Center, sample databases, and the Cookie House sample application — everything needed for a working end-to-end demonstration. Unlike the default dev mode, demo mode attaches to container logs after startup (Ctrl+C to stop). See [`demo/README.md`](demo/README.md) for demo infrastructure details and [`src/fidesplus/seed/README.md`](src/fidesplus/seed/README.md) for seed configuration and troubleshooting.

```sh
# Full demo environment with seeding
uv run nox -s dev -- demo

# Demo environment without seeding
uv run nox -s dev -- demo no-seed
```

#### Quick reference

| Command | Behavior |
|---|---|
| `uv run nox -s dev` | Start detached, validate `.env`, seed (best-effort), print ready banner |
| `uv run nox -s dev -- no-seed` | Start detached, skip all seeding and `.env` validation |
| `uv run nox -s dev -- demo` | Full demo environment, seed (required), attach to logs |
| `uv run nox -s dev -- demo no-seed` | Full demo environment, skip seeding, attach to logs |
| `uv run nox -s "dev_app(fidesplus)"` | Dev stack for a non-slim application image |
| `uv run nox -s seed` | Manually seed a running environment |
| `uv run nox -s teardown` | Stop containers |
| `uv run nox -s teardown -- volumes` | Stop containers and wipe all state |

### Running tests

Note: The `pytest` suite in Fidesplus currently relies on a `BIGQUERY_CREDENTIALS` environment variable being set in the test runtime. At the time of this writing, this should be set to the value of the `BigQuery (bqextract-service-acct-2) service account private key` vault entry found in OnePassword. Search for this vault in OnePassword, copy the `BIGQUERY_CREDENTIALS environment variable` value, and then add it to your local `.env` file like this:
```
BIGQUERY_CREDENTIALS='{ "type": "service_account", ...
```

To run the `pytest` test suite locally, you can either use the dedicated `pytest` nox session, or manually run `pytest` on your already-running local application container (see ["Running the Application"](##running-the-applications) section above)

#### Examples: Running fidesplus tests

Fidesplus backend tests use the `pytest` nox session (slim Docker stack, `tests/fidesplus/`):

```sh
uv run nox -s pytest
```

Merged **fides** core tests (`tests/fides/`) use separate sessions — build the CI image first, then run one group:

```sh
docker build -f Dockerfile --target slim_test -t ethyca/fidesplus:local-slim .
uv run nox -s "pytest_fides(api)"
```

To run only a specific set of tests, you can supply a test path as an argument. For example, to run just the classification tests:

```sh
uv run nox -s pytest tests/cls/
```

You can also pass individual test files or use any [pytest supported selectors](https://docs.pytest.org/en/latest/how-to/usage.html):

```sh
uv run nox -s pytest tests/api/test_plus.py
```

To rerun only the tests that failed in the previous pytest run (uses `.pytest_cache` on the bind-mounted repo):

```sh
uv run nox -s pytest_unit -- failed
# or
uv run nox -s pytest -- --lf
# or
PYTEST_LAST_FAILED=1 uv run nox -s "pytest_fides(api)"
```

The same `failed` / `--lf` shorthands work on other nox pytest sessions (`pytest_local`, `pytest_fides`, etc.).

#### Examples: Running tests _inside the application container_

There are two common approaches for running the backend test suite directly on your running Fidesplus backend container:

##### 1. Run tests using a single command from your host machine

This approach runs tests via a one-liner without needing to manually shell into the container. For the default backend container (`fidesplus-slim`), you can do:

```sh
docker exec fidesplus-slim bash -c "pytest --no-cov"
```

To run a specific test file (example: only run tests/ops/schemas/test_partitioning_schema.py):

```sh
docker exec fidesplus-slim bash -c "pytest --no-cov tests/ops/schemas/test_partitioning_schema.py"
```


##### 2. Run tests by shelling into the container

Alternatively, you can open a shell in the running backend container, and then invoke `pytest` directly within that shell. First, shell into the container

```sh
uv run nox -s "shell(<application>)"
```

Once inside the container, you can run any `pytest` command. For example:

```sh
pytest --no-cov
```

Or to run a specific test file or folder:

```sh
pytest --no-cov tests/api/test_plus.py
```

## Configuration

Some of the additional functionality within Fidesplus requires additional, specific configuration. The following section explains what is required and how to go about setting up the various additional services available.

### Dictionary service

Fidesplus can be pointed to a dictionary service (Fides Compass). There are three relevant variables:

```sh
FIDESPLUS__DICTIONARY__ENABLED=true
FIDESPLUS__DICTIONARY__DICTIONARY_SERVICE_URL=http://host.docker.internal:8081
FIDESPLUS__DICTIONARY__DICTIONARY_SERVICE_API_KEY=sample_api_key
```

These can alternatively be set in `fides.toml`:

```toml
[dictionary]
enabled = false
dictionary_service_url = "http://host.docker.internal:8081"
dictionary_service_api_key = "sample_api_key"
```

#### Hosted dictionary

To run against the hosted dictionary in our staging environment, you can find an API key in [1p](https://start.1password.com/open/i?a=5LPILPRDUBGWBPJWYS6QHHHLRE&v=p3bpsusghc3t45pzpsebehrd4m&i=rxpdmufsbwazfuc2qdalgn6arq&h=ethyca.1password.com) under "Staging API Key".

The service URL for the environment is `https://compass.dev.ethyca.com`.

#### Local dictionary

Alternatively, you can run the [dictionary service locally](https://github.com/ethyca/fides-services/tree/main/services/dictionary) Once you have the service running, the default credentials provided in the `fides.toml` file should work.


## Core (OSS) `fides` in this monorepo

Fides core lives in-tree at `src/fides/` as part of the unified **`fidesplus`** Python package. You do **not** maintain a separate `requirements.txt` pin or a sibling `ethyca/fides` checkout for day-to-day development.

### Typical workflow

1. Change code under `src/fides/` and/or `src/fidesplus/` on one branch.
2. Sync deps when `pyproject.toml` / `uv.lock` change: `uv sync --all-groups`.
3. Build and run locally: `uv run nox -s "build(slim)" "dev(slim)" -- dev`.
4. Run tests:
   - Fidesplus: `uv run nox -s pytest`
   - Fides core: `docker build -f Dockerfile --target slim_test -t ethyca/fidesplus:local-slim .` then `uv run nox -s "pytest_fides(api)"` (see [Running tests](#running-tests)).

CI uses the same monorepo layout (`backend_checks.yml` for `tests/fides/`, `backend_pr_checks.yml` for `tests/fidesplus/`).

### Prerelease Docker images (optional)

For integration testing against **published** prerelease builds (not the in-tree workspace), use nox `dev_prerelease` / `prod_prerelease` targets and `uv run nox -s "push(slim)" -- prerelease` after tagging. See `uv run nox -s usage -- push` and the `docker_fidesplus_prerelease` workflow.

**Note:** Nightly **double-edge** images (Fidesplus `main` plus a separate Fides `main` checkout) are no longer built; both trees ship in one revision in this repo.
