# Composable Sample Data Sources for the Seed Service

**Source:** [ENG-3764](https://ethyca.atlassian.net/browse/ENG-3764) / [ENG-3769](https://ethyca.atlassian.net/browse/ENG-3769)

## Problem Statement

Seed data today is a single fixed data set (`data/seed/resources/`, ~13 systems, no users,
no system groups), so demo environments feel sparse and can't exercise stewardship, RBAC,
system groups, or Data Map Report completeness. There is no way to select *which data*
gets seeded. Two rich synthetic data sets exist (publisher and insurance verticals) but
live outside the product as ad-hoc scripts.

## Design: profiles compose, sources parameterize

A seed *profile* (today `secret_profile`) is already a **stored `SeedRequest`** — the 1PW
payload is a full request body (task flags + configs) merged field-by-field with the
caller's request. We deliberately do **not** introduce a second, parallel "data profile"
concept. Instead:

- **Sample data sources** are named, on-disk data bundles (`data/seed/sample_data/<name>/`)
  containing fideslang manifests plus user and system-group definitions. They are
  config-free, 1PW-free, and additive by construction.
- **Data tasks select their sources** via the service's established sibling-config idiom
  (boolean flag in `SeedTasksConfig` + optional `Optional[BaseModel]` config field on
  `SeedRequest`, like `discovery_monitors`). Omitted config = `["default"]` (exact
  back-compat).
- **Profiles remain the composition layer**: a stored profile payload sets the task flags
  *and* the per-task sources, so `POST /seed {"secret_profile": "publisher-demo"}`
  provisions a complete vertical demo. This is the first half of ENG-3769.

Sibling configs are auto-discovered by `SeedRequest._credential_field_names()`, so the
1PW merge needs **zero changes**.

`secret_profile` is misnamed by its own success (profiles compose data selection, not
secrets): add `profile` as the canonical field name with `secret_profile` as a
deprecated alias (story 6). A config-only (non-1PW) profile variant is a follow-up,
out of scope here.

## Requirements Summary

- Named **sample data sources**, selectable per data task, **composable and additive**
  (e.g. `["default", "publisher"]`) with no cross-source conflicts
- Vertical naming: `publisher` (media group), `insurance` (risk partners)
- New **users** seed task: per-user randomly generated passwords, logged at INFO
  (server logs only)
- New **system_groups** seed task (groups carry color labels + steward assignments)
- Data defined as JSON/YAML files checked into the repo
- Scope: users + systems first; datasets ride along where already converted

## Data Model

**No migrations.** Everything writes to existing tables:

| Resource | Model | Write path |
|---|---|---|
| Systems + privacy declarations | `System`, `PrivacyDeclaration` | existing `_upsert_resource` / `_upsert_privacy_declarations` |
| Users | `FidesUser` (hashed_password+salt), `FidesUserPermissions` (roles) | new `setup_users` step |
| Steward assignments | `SystemManager` (m2m user↔system) | within `setup_users`, reusing the `_seed_dashboard_stewards` pattern |
| System groups | `SystemGroup` + `TaxonomyElement` + `SystemGroupMember` | new `setup_system_groups` step via `SystemGroupService.create_system_group` (handles the taxonomy element — never raw model inserts) |

Verified: the insurance data set's `label_color` values (`taxonomy_blue`, `sandstone`,
`minos`, …) all exist in `CustomTaxonomyColor` (`src/fides/api/models/system_group.py`).

### Sample data directory layout

```
data/seed/sample_data/
  default/                    # moved from data/seed/resources (git mv)
    source.yml                # description metadata
    systems/*.yml             # fideslang manifests
    datasets/*.yml
    policies/*.yml
    organization.yml
  publisher/                  # media group vertical
    source.yml
    systems/*.yml
    users.yml
    system_groups.yml
  insurance/                  # risk partners vertical
    ...same shape, plus datasets/
```

`default` is **not special-cased** — the existing `data/seed/resources/` directory is
physically moved (the path is referenced only by the `SEED_RESOURCES_PATH` constant in
`src/fidesplus/seed/service.py`; no packaging or Dockerfile references).
`data/seed/email_templates/` is untouched.

`users.yml` entry shape (no password field by design):

```yaml
users:
  - username: mchen
    email_address: mchen@meridian.example
    first_name: Maya
    last_name: Chen
    role: viewer            # validated against fides.api.oauth.roles
    systems:                # steward assignments, by fides_key
      - subscriber_crm
```

## Security Requirements

- **Passwords**: never committed, never in request payloads. `setup_users` generates a
  cryptographically random, policy-compliant password per user and logs it at INFO in the
  seed step output — server logs only, explicitly **not** written to the Redis execution
  tracker, so the seed status API never exposes credentials. Re-runs never touch an
  existing user's password. Day-2 access: root/owner resets any seeded user via the
  existing `PUT /user/{user_id}/force-reset-password` endpoint.
  - Trade-off (accepted): passwords in server logs flow to log aggregation. Acceptable
    for demo-env users by design; any individual credential is resettable, and seeded
    users are confined to non-production environments per the seed service's existing
    warnings.
- **Authorization**: unchanged — seed endpoints already require the `CONFIG_UPDATE`
  scope; user creation happens server-side within the already-privileged seed flow.
- **Role validation**: seeded roles validated against the role registry at load time;
  reject unknown roles rather than silently creating broken permissions.
- **Input validation**: source names resolved against the discovered on-disk registry
  only (no path traversal from user-supplied names — never join paths from raw input).
- **Confidentiality**: sample data uses fictional names/domains (`.example`) per repo
  conventions; upstream user files with plaintext passwords are *not* imported as-is —
  user definitions are regenerated sanitized.
- **Threat triage**: run `/threat-triage` on the first PR. Expected low-moderate (no new
  auth surface); seeding users with `owner` role is the one privilege-relevant element —
  mitigated by the existing scope gate and the docs' non-production warning.

## Scalability Considerations

Not a concern at this scale (≤60 systems, ≤20 users per source). Systems upsert through
the existing per-resource loop; users/groups are small loops. The seed already runs as a
FastAPI background task with Redis-tracked step status — new steps inherit that.

## Error Handling & Recovery

- **Idempotency** (the core property): systems upsert by `fides_key` (existing), users
  upsert by `username` (create-or-update permissions/stewards, never reset an existing
  password), groups upsert by `fides_key` via service get-then-create/update. Re-running
  any combination of sources is safe.
- **Additivity guarantee**: a unit test loads every built-in source and asserts
  pairwise-disjoint `fides_key`/username sets across sources (`default` included), so
  `default + publisher + insurance` can never conflict. New sources can't merge without
  passing it.
- **Partial failure**: follow existing per-step semantics — a step failure marks that
  step `ERROR` in the execution tracker and halts the run; data written so far is
  consistent because each resource write is independent and idempotent.
- **Dangling references**: steward assignments and group memberships referencing unknown
  systems/users log a warning and skip (matching the dashboard seeder's behavior) rather
  than failing the run — supports seeding `publisher` users without its systems, or
  mixed-source selections.
- **Unknown source**: 422 with the available-sources list, mirroring the existing
  `UnknownSeedProfileError` pattern.

## Implementation Approach

All changes in `src/fidesplus/seed/` plus data files; follows the documented
"Adding a New Seed Step" recipe (`service.py` module docstring).

1. **Sample data registry** (new `sample_data.py`): discover
   `data/seed/sample_data/*/source.yml`, expose resolution by name with allowlist
   semantics + typed errors; compute per-source resource counts for the listing endpoint.
2. **Schemas** (`schemas.py`): `SampleResourcesConfig(sources=["default"])`,
   `UsersConfig(sources=[...])`, `SystemGroupsConfig(sources=[...])` as sibling configs;
   `SeedTasksConfig.users` / `.system_groups` boolean flags.
3. **Service** (`service.py`): `setup_sample_resources` resolves its config's sources and
   iterates their dirs (existing parse/upsert path unchanged); add `setup_users` and
   `setup_system_groups`. Dispatch order in `run_seed`: `sample_resources` → `users` →
   `system_groups` (groups reference both systems and steward usernames).
4. **Routes**: `GET /plus/seed/sample-data` listing available sources with metadata +
   resource counts (`GET /seed/profiles` untouched — it lists stored 1PW profiles).
5. **Data**: `insurance/` from the already-converted fideslang manifests + ported group
   definitions + regenerated sanitized users; `publisher/` converted from the ENG-3764
   source data.
6. **Periphery**: `profile` alias for `secret_profile`; nox `seed` session sources param;
   admin-ui POC page source multi-select (regenerate API types); update `seed/README.md`.

## API Design

```jsonc
// POST /api/v1/plus/seed — raw request
{
  "tasks": {
    "sample_resources": true,
    "users": true,                                  // NEW
    "system_groups": true                           // NEW
  },
  "sample_resources": { "sources": ["default", "publisher"] },  // NEW sibling config
  "users":            { "sources": ["publisher"] },             // NEW sibling config
  "system_groups":    { "sources": ["publisher"] }              // NEW sibling config
}

// 1PW profile "publisher-demo" — a stored SeedRequest composing the above;
// POST /seed {"secret_profile": "publisher-demo"} provisions the full vertical.

// GET /api/v1/plus/seed/sample-data
[
  { "name": "default",   "description": "Standard sample data",   "systems": 13, "users": 0,  "system_groups": 0 },
  { "name": "publisher", "description": "Media group vertical",   "systems": 51, "users": 19, "system_groups": 9 },
  { "name": "insurance", "description": "Risk partners vertical", "systems": 43, "users": 19, "system_groups": 9 }
]
```

## Story Breakdown

| # | Story | Type | Estimate | Dependencies |
|---|-------|------|----------|--------------|
| 1 | Sample-data-source infrastructure: registry/discovery, move `data/seed/resources` → `data/seed/sample_data/default`, `SampleResourcesConfig.sources`, generalized `setup_sample_resources`, `GET /seed/sample-data` | Story | M | - |
| 2 | `users` seed task: `UsersConfig.sources`, `setup_users` with generated passwords (logged), roles, permissions, steward assignments; idempotent | Story | M | #1 |
| 3 | `system_groups` seed task: `SystemGroupsConfig.sources`, via `SystemGroupService`; ordering after users | Story | S | #1, #2 |
| 4 | `insurance` source data: manifests, system groups, sanitized users + cross-source collision test | Task | M | #1, #2, #3 |
| 5 | `publisher` source data: convert ENG-3764 source to manifests + users + groups | Task | M | #1, #2, #3 |
| 6 | Periphery: `profile` alias for `secret_profile`, nox param, admin-ui POC source selector + type regen, seed README/docs | Task | S | #1 |

Each of 2/3 is a complete vertical slice (schema → service step → tests); 4/5 are
data-only and independently mergeable.

## Status & Handoff (2026-06-11)

**Done:**

- Story 1 — [PR #3927](https://github.com/ethyca/fidesplus/pull/3927) (draft, base `dev`),
  branch `adam/ENG-3769-seed-data-profiles`
- Stories 2+3 — [PR #3934](https://github.com/ethyca/fidesplus/pull/3934) (draft,
  stacked on #3927), branch `adam/ENG-3769-seed-users-task`

**Remaining: stories 4, 5, 6.** Stack new branches on `adam/ENG-3769-seed-users-task`.

### Building a new source (stories 4/5)

Create `data/seed/sample_data/<name>/` with `source.yml` (description),
`resources/` (fideslang YAML manifests — systems/datasets/policies),
`users.yml`, `system_groups.yml`. Shapes are documented in
`src/fidesplus/seed/sample_data.py` (module docstring) and validated by
`SampleUser` / `SampleSystemGroup` in `seed/schemas.py` (roles checked against
`ROLES_TO_SCOPES_MAPPING`; colors against `CustomTaxonomyColor`). No password
fields anywhere — passwords are generated at seed time.
`tests/fidesplus/seed/test_sample_data.py::TestBuiltInSources` already
enforces cross-source fides_key/username disjointness and will pick up new
source dirs automatically; add a per-source count assertion when adding data.

### Source material

- **insurance** (story 4): fideslang manifests already converted at
  `dsill-ethyca/demo-env-setup` → `01-sterling-synthetic-data/fideslang_manifests/`
  (43 systems, 21 datasets). System group definitions: `SYSTEM_GROUPS` constant
  in `seed_synthetic_systems.py` (9 groups; color values verified to exist in
  `CustomTaxonomyColor`). The users file (`synthetic-users.json`) is
  **gitignored upstream** (plaintext passwords) — regenerate 19 sanitized
  user definitions from the shape documented in that repo's `USERS.md`
  (username/email/name/department/role/steward systems), or get the canonical
  copy from the demo-env owner and strip passwords.
- **publisher** (story 5): `synthetic-data-newscorp (1).zip` attached to
  [ENG-3764](https://ethyca.atlassian.net/browse/ENG-3764) (51 systems,
  20 datasets, 19 users, 9 groups, steward map). JSON → fideslang conversion
  precedent: `convert_to_fideslang_manifests.py` in the demo-env-setup repo.
  Datasets conversion is deferred (out of scope); users + systems + groups first.
  Use fictional `.example` domains; never import upstream passwords.

### Gotchas learned in stories 1–3

- **Tooling**: run `uv run ruff format` / `uv run ruff check` / `uv run pyrefly check`
  (project-pinned, project-wide). A pyenv-installed ruff (0.15.4 vs pinned
  0.15.16) reformatted differently and broke an inline `# pyrefly: ignore`,
  causing red CI on both PRs.
- The `system_group` Taxonomy row comes from a data migration; tests need the
  `system_groups_taxonomy` fixture (`tests/fidesplus/conftest.py`).
- `fidesplus.seed.service` cannot be imported standalone (pre-existing circular
  import via discovery-monitor routes); import `fidesplus.main` first, or rely
  on pytest. Tests are the right smoke check.
- `parse_manifests` ingests **every** `.yml`/`.yaml` recursively — fideslang
  manifests must stay under `resources/` inside a source so `users.yml` /
  `system_groups.yml` aren't swept up.
- Seed run order is `sample_resources` → `users` → `system_groups`; dangling
  references (unknown stewards/systems) warn and skip by design, so data-only
  PRs can be tested with any subset of tasks.

## Testing Strategy

- **Schema unit tests** (`test_schemas.py`): sources-config defaults and validation,
  1PW merge with the new sibling fields (should pass with zero merge changes).
- **Service unit tests** (`test_service.py`): `setup_users` (create/update/skip paths,
  role validation, dangling steward refs), `setup_system_groups` (create vs update,
  missing steward), source resolution (unknown source, multi-source iteration).
- **Integration** (`test_service_integration.py`): extend `TestSeedTaskCoverage` (every
  task flag must have a wired step); per-source load test asserting counts; double-run
  idempotency test; composability test (`default` + `insurance` together).
- **Static data test**: cross-source fides_key/username disjointness; every source's
  YAML parses against fideslang + the users/groups schemas (catches bad data files in CI
  without a DB).
- **Route tests**: `GET /seed/sample-data`.

## Out of Scope

- **Privacy-request enrichment** (synthetic identities, denial/cancel reasons, locations,
  source weights) — ENG-3764's "areas of improvement" section; separate follow-up against
  the dashboard seeder.
- **Publisher datasets conversion** — deferred; insurance datasets included since already
  converted.
- **Tenant-manager exposure** of seed profiles — second half of ENG-3769.
- **Config-only (non-1PW) profiles** — worthwhile follow-up so pure-data profiles don't
  require the 1PW token; not in this slice.
- **Default-source content changes** — the publisher data is *not* merged into
  `default` (supersedes the interim approach discussed on ENG-3764).
