# Technical Spike: Cloud Infra System Grouping & Promotion

## Context

The cloud infra monitor currently discovers AWS resources (S3, RDS, DynamoDB) and stages them as `CloudInfraStagedResource` records. The next step is enabling users to **group** multiple resources into a single Fides System and **promote** them — persisting the assignment in the System Inventory.

Unlike IDP monitors (Okta/Entra) where promotion is 1:1 (one app → one System), cloud infra uses a **many-to-many** model: N AWS resources can belong to M Systems, since a shared resource (e.g. a common RDS instance) may serve multiple business applications.

This spike covers the backend work needed for grouping and promotion. The approach is **iterative by design** — this domain is loosely defined enough that we'd rather take small steps, validate, and let the solution evolve based on what we learn. The phasing reflects this:

- **Phase 1: Manual Grouping & Promotion** — users manually create groups, assign resources to them, and promote resources individually. Includes bulk actions (mute/unmute) and the full promotion lifecycle. This gives us something testable immediately.
- **Phase 2: Rule-based Grouping** — deterministic rules leveraging AWS tags (e.g. "all resources tagged `Application=checkout` → Checkout System"). Rules create draft groups during the scan that users review and promote.
- **Phase 3 (if needed): LLM-assisted Grouping** — only if we see gaps that rules can't cover, and ideally with real enterprise data to develop effective prompts against.

**Note on classification:** Unlike IDP monitors where Compass/LLM can infer descriptions and data uses (because apps are known products), cloud infra resources (S3 buckets, RDS instances) don't carry enough metadata to classify at scan time. Data uses and descriptions are best determined after an integration is created and we can actually inspect the data. This is why the integration CTA post-promotion is important — it bridges the gap between grouping and classification.

## What Exists Today

| Layer | Status |
|-------|--------|
| `CloudInfraStagedResource` ORM model | ✅ Own table (`cloud_infra_staged_resource`) with `service`, `tags`, `location`, `cloud_account_id`, `source_id` |
| Detection & reconciliation | ✅ Scan → diff → persist → reconcile disappeared resources |
| `GET /cloud-infra-results` | ✅ Paginated query with filters (`diff_status`, `location`, `cloud_account_id`) |
| Pydantic schemas & domain entities | ✅ `CloudInfraStagedResource`, `CloudInfraResourceMetadata`, `CloudInfraCategory` |
| `promote_staged_resource()` | ❌ Raises `NotImplementedError` |
| Bulk actions (promote/mute/unmute) | ❌ No endpoints or service methods |
| Resource grouping | ❌ No grouping mechanism exists |

**Reference implementation:** The datastore monitor has the most relevant promotion and bulk action patterns. Its architecture uses an actions controller layer (`discovery_monitor_actions_controller.py`), a filtered-field-actions system (`discovery_monitors_filtered_field_actions.py`), and Celery tasks for async promotion.

## Schema Changes

### New table: `cloud_infra_group`

A group represents a named logical collection of cloud infra resources that maps to a System. A group can be created targeting a new System (no `system_id` yet) or an existing one (`system_id` set at creation).

```python
class CloudInfraGroup(Base):
    id = Column(UUID, primary_key=True, default=uuid4)
    monitor_config_id = Column(String, nullable=False, index=True)
    draft_system_name = Column(String, nullable=True)  # used as System name on promotion; only relevant when system_id is NULL
    system_id = Column(String, ForeignKey(System.id_field_path), nullable=True, index=True)  # NULL = new System pending, set = targets existing System
    created_at = Column(DateTime, default=now)
    updated_at = Column(DateTime, default=now, onupdate=now)
```

- `draft_system_name` is the provisional System name, only relevant when `system_id` is `NULL`. Used as the System name on first promotion. Once `system_id` is set, the authoritative name is on the System itself — `draft_system_name` is ignored. The user provides it on group creation (Phase 1), or rules derive it from tag values (Phase 2, e.g. `Application=checkout` → "checkout"). Nullable because groups targeting an existing System don't need it.
- `system_id` is a FK to `ctl_systems.id`. `NULL` means the group targets a new System that will be created on first promotion. A value means the group targets an existing System (set at creation or after first promotion). Note: `system_id` being set does NOT mean all resources in the group are promoted.
- `monitor_config_id` scopes groups to a specific monitor.
- Unique constraint `(monitor_config_id, system_id)` where `system_id IS NOT NULL` — a System can only have one group per monitor. Groups from different monitors can target the same System.

### New join table: `cloud_infra_group_assignment`

A resource can belong to **multiple groups** (and therefore multiple Systems). The relationship is modeled via a join table.

```python
class CloudInfraGroupAssignment(Base):
    id = Column(UUID, primary_key=True, default=uuid4)
    resource_id = Column(UUID, ForeignKey("cloud_infra_staged_resource.id"), nullable=False, index=True)
    group_id = Column(UUID, ForeignKey("cloud_infra_group.id"), nullable=False, index=True)
    promoted = Column(Boolean, nullable=False, default=False)  # whether this resource has been promoted within this group
    created_at = Column(DateTime, default=now)
```

- A resource with no assignments is ungrouped.
- A resource can appear in multiple groups.
- `promoted` tracks whether the resource has been promoted within this specific group. A resource can be promoted in one group but not yet in another. When a resource is first promoted in any group, its `diff_status` changes to `MONITORED` and stays that way — subsequent promotions in other groups only set `promoted = true` on the new assignment without changing the resource's status.
- Unique constraint `(resource_id, group_id)` — a resource can only appear once per group.

**No `system_id` on the resource.** With multi-group support, a single `system_id` column is insufficient to represent all System associations. The join table + group's `system_id` is the source of truth. Querying "all resources for System X" goes through: `assignment JOIN group WHERE group.system_id = X AND assignment.promoted = true`.

### Query service: Add missing filters

The current query service filters on `diff_status`, `location`, `cloud_account_id`, and `service`. Additional filters needed per the PRD:

| Filter | Column | Notes |
|--------|--------|-------|
| `group_name` | `group.draft_system_name` or `system.name` | Filter by group's draft name or System name (via join table); ungrouped resources have no assignments |
| `system_id` | `group.system_id` | Filter by specific System (via join table: assignment → group) |


The existing search (regex on `name` and `urn`) should be extended to also search tag keys and values. This becomes important in Phase 2 where tags drive rule-based grouping.

## Grouping & Promotion Design

### Concepts

A **group** is a named logical collection of resources that maps to a System (existing or future).

- `group.system_id = NULL`: the group targets a new System that doesn't exist yet. The `name` is the intended System name.
- `group.system_id` set: the group targets an existing System. This can be set at group creation time (user picks an existing System) or at the first promotion (System created from `group.draft_system_name`). The `system_id` can be changed as long as no assignments in the group have `promoted = true` — once a resource has been promoted, the group's System is locked.

**A resource can belong to multiple groups** (and therefore multiple Systems). This supports shared resources.

**Promotion is per-resource, not per-group.** A group can have a mix of promoted and unpromoted resources. A resource's promotion status is tracked per-group via the `promoted` flag on the assignment — not when it is assigned to a group, even if the group already has a `system_id`.

This is important because in Phase 2, rules may add new resources to a group that already has a System during a scan — those new resources should remain `ADDITION` until the user reviews and promotes them individually.

### User flow: Promoting to a new System

```mermaid
sequenceDiagram
    actor User
    participant UI
    participant API

    User->>UI: Select resources
    UI->>API: POST /groups {draft_system_name, resource_urns}
    API-->>UI: Group created
    User->>UI: Review / adjust group
    User->>UI: Promote resources
    UI->>API: POST /promote {resource_urns}
    API->>API: Create System from draft_system_name
    API->>API: Set group.system_id
    API->>API: Set assignments promoted + resources MONITORED
    API-->>UI: Promotion complete
```

### User flow: Promoting to an existing System

```mermaid
sequenceDiagram
    actor User
    participant UI
    participant API

    User->>UI: Select resources
    UI->>API: POST /groups {system_id, resource_urns}
    API-->>UI: Group created (linked to existing System)
    User->>UI: Review / adjust group
    User->>UI: Promote resources
    UI->>API: POST /promote {resource_urns}
    API->>API: Set assignments promoted + resources MONITORED
    API-->>UI: Promotion complete
```

### Resource lifecycle

```mermaid
flowchart TD
    Scan["Scan discovers resource"] --> Addition["ADDITION\n(ungrouped)"]

    Addition -->|assign to group| Grouped["ADDITION\n(grouped)"]
    Addition -->|mute| Muted["MUTED"]

    Grouped -->|promote in group| Monitored["MONITORED\n(promoted in ≥1 group)"]
    Grouped -->|add to another group| Grouped
    Grouped -->|remove from group| Addition
    Grouped -->|mute| Muted

    Monitored -->|promote in another group| Monitored
    Monitored -->|disappears from AWS| Removal["REMOVAL"]
    Monitored -->|System deleted| Addition

    Removal -->|confirm-removal| Deleted["Deleted"]

    Muted -->|unmute| Addition
```

### State machine

```
Ungrouped resource (ADDITION, no assignments)
  ├── assign to group ──► Grouped resource (ADDITION, assignment created)
  └── mute ─────────────► MUTED

Grouped resource (ADDITION, has assignments)
  ├── promote (in group) ► MONITORED (assignment.promoted = true; if draft group, System created first)
  ├── remove from group ─► assignment deleted (if no assignments left, resource is ungrouped)
  ├── add to another group► new assignment created (resource now in multiple groups)
  └── mute ──────────────► MUTED (all assignments removed)

MONITORED resource (promoted in at least one group)
  ├── add to another group► MONITORED (new assignment created, promoted = false)
  ├── promote (in another group) ► MONITORED (assignment.promoted = true)
  ├── disappears ────────► REMOVAL (existing reconciliation)
  └── System deleted ────► assignment.promoted cleared, group reverts to draft
                           ├── if promoted in other groups ► stays MONITORED
                           └── if no other promoted assignments ► ADDITION

REMOVAL resource
  └── confirm-removal ───► deleted (assignments removed, staged resource deleted)

MUTED ──unmute───────────► ADDITION (no assignments)
```

### Promotion logic

Promotion is **per-resource** in Phase 1. Selected resources within a group are promoted individually. The intent is to keep Phase 1 as simple as possible to validate the core idea.

Per-resource promotion is also needed regardless: when a scan adds a new resource to an already-promoted group, that resource needs to be promoted individually. Promoting an entire group at once (all `ADDITION` resources in one action) is straightforward to add later.

```
POST /{monitor_config_key}/cloud-infra-results/promote
Body: {
    "resource_urns": ["urn1", "urn2", ...]
}
```

1. Validate all URNs have at least one assignment
2. For each group the resource belongs to:
   - **If the group has no `system_id`** (new System): create the System automatically from `group.draft_system_name`, set `group.system_id`
     - `fides_key`: sanitized from `draft_system_name`
     - `system_type`: `"service"`
     - `description`, `privacy_declarations`, `data_uses`: left empty — populated later via integration
   - **If the group already has `system_id`** (existing System): reuse it
   - Set `assignment.promoted = true`
3. Set resource `diff_status = MONITORED` (if not already)

**Note:** Resources in the same group that are not in the selection remain `ADDITION` — they are not promoted automatically.

### Re-grouping

An `ADDITION` resource can be added to or removed from groups at any time. A resource can belong to multiple groups simultaneously. Once a resource is `MONITORED`, it cannot be removed from groups where it has been promoted — but it can still be assigned to or removed from other groups where it has not been promoted.

## Bulk Actions

### Mute / Unmute

Follows the datastore monitor pattern — mute/unmute are **synchronous** (no Celery tasks), returning `MonitorActionResponse(task_ids=[])`.

```
POST /{monitor_config_key}/cloud-infra-results/mute
POST /{monitor_config_key}/cloud-infra-results/unmute
Body: { "resource_urns": ["urn1", "urn2", ...] }
```

- **Mute:** `ADDITION` → `MUTED`, all group assignments removed
- **Unmute:** `MUTED` → `ADDITION` (single bulk UPDATE with status check)

Cloud infra resources are flat (no parent/child hierarchy), so unmute is simpler than the datastore case — always restores to `ADDITION`.

### Allowed Actions Endpoint

```
POST /{monitor_config_key}/cloud-infra-results/allowed-actions
Body: { "resource_urns": ["urn1", "urn2", ...] }
```

Returns intersection of valid actions for the selected resources:

| Current Status | Allowed Actions |
|----------------|-----------------|
| `ADDITION` (ungrouped) | `group`, `mute` |
| `ADDITION` (grouped) | `promote`, `ungroup`, `mute` |
| `MONITORED` | — (managed from System Inventory) |
| `MUTED` | `unmute` |
| `REMOVAL` | `confirm-removal` (removes from System and deletes staged resource) |

## Service Layer Architecture

Following the datastore monitor pattern, the cloud infra actions integrate into the existing actions infrastructure:

```
cloud_infra_monitor/
├── base.py                  # existing — implement promote_staged_resource
├── repository.py            # existing — add group/promotion/mute/unmute methods
├── entities.py              # existing
└── aws/
    └── monitor.py           # existing

routes/discovery_monitor/
├── discovery_monitors.py    # existing — add cloud infra action endpoints
└── cloud_infra_actions_controller.py  # NEW — action wrappers
```

### Action wrappers (new)

Following the datastore pattern, all action wrappers share the same signature and return `MonitorActionResponse`:

- `create_group_wrapper(db, monitor_config_key, draft_system_name=None, system_id=None, resource_urns=None)` → `MonitorActionResponse` (group can be created empty or with resources; provide `draft_system_name` for new System or `system_id` for existing)
- `promote_resources_wrapper(db, urns, monitor_config_key)` → `MonitorActionResponse`
- `mute_cloud_infra_resources_wrapper(db, urns, monitor_config_key)` → `MonitorActionResponse`
- `unmute_cloud_infra_resources_wrapper(db, urns, monitor_config_key)` → `MonitorActionResponse`
- `confirm_removal_wrapper(db, urns, monitor_config_key)` → `MonitorActionResponse`

**Promotion can be synchronous initially.** Unlike datastore monitors which promote schema→table→field hierarchies and upsert datasets (requiring Celery), cloud infra promotion is a bulk UPDATE + System creation. If latency becomes an issue, we can move to Celery later.

### Repository additions

- `create_group(monitor_config_id, draft_system_name=None, system_id=None)` — insert into `cloud_infra_group` (provide `draft_system_name` for new System, or `system_id` for existing)
- `assign_resources_to_group(urns, group_id)` — insert assignments into join table
- `remove_resources_from_group(urns, group_id)` — delete assignments from join table
- `promote_resources(urns)` — for each resource, promote in all its groups (create System if needed), set `assignment.promoted = true`, update `diff_status = MONITORED`
- `create_system_from_group(group)` — create System from `draft_system_name`
- `bulk_mute_resources(urns, monitor_config_key)` — set MUTED, remove all assignments
- `bulk_unmute_resources(urns, monitor_config_key)` — set ADDITION with MUTED check
- `confirm_removal(urns)` — remove all assignments, delete staged resources

## API Endpoints Summary (Phase 1)

All under `/{monitor_config_key}/cloud-infra-results/`:

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/` | ✅ Exists — paginated results with filters |
| `POST` | `/groups` | Create a new group with selected resources |
| `GET` | `/groups` | List groups for this monitor (draft + promoted) |
| `PATCH` | `/groups/{group_id}` | Update group (rename, add/remove resources) |
| `DELETE` | `/groups/{group_id}` | Delete a group (only if no assignments have been promoted; removes all assignments) |
| `POST` | `/promote` | Promote selected resources (creates System if group is draft) |
| `POST` | `/confirm-removal` | Confirm removal of disappeared resources |
| `POST` | `/mute` | Bulk mute |
| `POST` | `/unmute` | Bulk unmute |
| `POST` | `/allowed-actions` | Valid actions for selection (POST because the body can contain a large list of URNs) |

## Phase 2: Rule-based Grouping

> **Out of scope for Phase 1.** Included for context so Phase 1 design doesn't block it.

Deterministic grouping based on AWS tags. Large enterprises enforce strict tagging — resources are tagged with keys like `Application`, `Service`, `Project`, `Team`.

### Approach

- User configures tag keys that represent application grouping (e.g. `Application`, `Service`)
- On scan completion, rules create **draft groups** from resources sharing the same tag value
- Users review, modify, or dismiss draft groups before promoting

### Configuration

Where tag key rules live is TBD:
- **Monitor config** — per-monitor rules (simplest, most flexible)
- **Organization settings** — global defaults
- **Hardcoded defaults with override** — start with `Application`, `App`, `Service`, `Project`

### Phase 1 decisions that enable Phase 2

- The `tags` JSONB column is already stored and queryable — rules can group by tag values without schema changes
- No coupling between the grouping mechanism and the promotion flow
- A `source` column could be added to the group table later to distinguish manual vs rule-created groups

## Phase 3 (if needed): LLM-assisted Grouping

> **Only pursue if Phase 2 reveals gaps that deterministic rules can't cover.** Requires real enterprise data to develop effective prompts — without it, prompt development would be inefficient.

Potential use cases where LLM could add value over rules:
- Resources with inconsistent or missing tags
- Inferring application boundaries from name patterns, service combinations, and resource relationships
- Generating human-readable group descriptions

This phase warrants its own spike with real customer data to evaluate feasibility.

## Deduplication

The `urn` (format: `{monitor_key}:{arn}`) and `source_id` (the AWS ARN) already serve as unique identifiers. On subsequent scans, existing resources are matched by URN — promoted resources (`MONITORED`) are recognized and not re-staged. This is already implemented in the detection flow.

## Future Enhancement: Integration CTA

> **Not tied to a specific phase.** Can be added at any point after Phase 1 lands.

After resources are promoted to a System, the UI could surface a CTA to create an integration for applicable resources (e.g. RDS → PostgreSQL connection, DynamoDB → DynamoDB connector). The connection type can be inferred from the AWS service. If an integration already exists for the System, offer to link the resource instead.

This is where real classification would happen — data categories and data uses are determined by actually connecting to and scanning the datastore, not from resource metadata alone.

**Backend support needed:** An endpoint or field that returns whether a promoted resource's service type has a corresponding Fides connector, and whether the System already has an integration configured.

## Future Enhancement: Show cloud resources in System Inventory

> Currently, promoted cloud infra resources are linked to a System via `system_id` but are not visible within the System Inventory UI. We need a way to display which cloud resources belong to a System.

## Future Enhancement: Moving resources between Systems from the Inventory

> Once a resource is `MONITORED`, it is managed from the System Inventory, not the action center. Managing a resource's group assignments from within the inventory (adding/removing from groups) is out of scope for this spike and would need its own design.

## Future Enhancement: Integration ↔ System 1:N refactor

> Currently an Integration can only be associated with a single System. With multi-group support, a shared resource (e.g. a common RDS instance) may end up associated with multiple Systems that each need their own integration. The current 1:N restriction on Integration→System will need to be refactored to avoid duplicate/identical integrations. This is a prerequisite for the Integration CTA to work fully with multi-group resources.

## Open Questions (Phase 1)

1. **Re-grouping audit trail** — The PRD says "History is recorded on the System audit log." Do we need to log which resources were added/removed from a System?
2. **Create Assets on promotion?** — Should promotion also create Assets under the System (similar to how website monitors promote resources as Assets)?

## Decisions

1. **System deletion cascade** — When a System is deleted, the associated group reverts to draft (`system_id` cleared) and its assignments are marked as not promoted (`promoted = false`). Resources that are still promoted in other groups remain `MONITORED`; resources with no remaining promoted assignments revert to `ADDITION`.
