# Purpose-Based Access Control (PBAC): Data Purposes, Consumers & Producers

## Overview

We've introduced three new core entities — **Data Purposes**, **Data Consumers**, and **Data Producers** — as the foundation of our Purpose-Based Access Control (PBAC) system. Together, these entities answer three fundamental questions about data access:

| Entity | Question it answers | Example |
|---|---|---|
| **Data Purpose** | *Why* is this data being accessed? | "Marketing Analytics", "Billing Operations" |
| **Data Consumer** | *Who* is accessing the data? | "Analytics Team", "ML Pipeline" |
| **Data Producer** | *Who is responsible* for the data? | "Data Engineering Team", "Revenue Team" |

The system evaluates SQL queries in real time: when a consumer runs a query, PBAC checks whether the consumer's declared purposes overlap with the purposes allowed on the dataset. Mismatches are flagged as **violations**.

## How It Works (End-to-End Flow)

```
1. Register consumers        →  Who accesses data (teams, services, individuals)
2. Define purposes           →  Why data is accessed (tied to data uses from the taxonomy)
3. Assign purposes to consumers  →  Link consumers to their declared reasons for access
4. Annotate datasets         →  Tag datasets/collections/fields with allowed purposes
5. Evaluate queries          →  SQL queries are parsed; consumer purposes are checked
                                 against dataset purposes. Mismatch = violation.
```

## Data Purpose

A Data Purpose defines *why* data is being accessed. It is tied to the Fides taxonomy via the `data_use` field, grounding purposes in the organization's existing privacy taxonomy.

Purposes are **universal** — they are not scoped to any platform. A purpose like "Marketing Analytics" applies regardless of whether the consumer is a GCP IAM role or a Google Workspace group.

Several fields were carried over from Privacy Declarations to Data Purposes, elevating legal and compliance metadata so it can be managed independently at the purpose level:

| Privacy Declaration Field | Data Purpose Field | Notes |
|---|---|---|
| `special_category_legal_basis` | `special_category_legal_basis` | Legal basis when processing special category data |
| `features` | `features` | List of processing features associated with this purpose |
| `impact_assessment_location` | `impact_assessment_location` | Currently a free-text string (URL to external platform); could be enhanced to link to Fides assessments |
| `flexible_legal_basis_for_processing` | `flexible_legal_basis_for_processing` | Whether the legal basis can be overridden by consent |

**Key attributes:**
- `fides_key` — Unique identifier (e.g., `marketing_analytics`)
- `name` — Human-readable label
- `description` — Optional description
- `organization_fides_key` — Optional organization scope
- `data_use` — Link to the Fides data use taxonomy (e.g., `marketing.advertising`); validated against the taxonomy on create/update
- `data_subject` — Optional data subject scope; validated against the taxonomy if provided
- `data_categories` — Which categories of data this purpose covers
- `legal_basis_for_processing` — Legal basis (e.g., consent, legitimate interest)
- `flexible_legal_basis_for_processing` — Whether the legal basis can be overridden by consent (default: `true`)
- `special_category_legal_basis` — Legal basis for special category data processing
- `impact_assessment_location` — URL or reference to a data protection impact assessment
- `features` — Processing features associated with this purpose
- `retention_period` — How long data may be retained under this purpose
- `tags` — Freeform labels for organization

**API endpoints:**
| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/api/v1/data-purpose` | Create a new purpose |
| `GET` | `/api/v1/data-purpose` | List purposes (filterable by `data_use`, `data_subject`) |
| `GET` | `/api/v1/data-purpose/{fides_key}` | Get a single purpose |
| `PUT` | `/api/v1/data-purpose/{fides_key}` | Update a purpose |
| `DELETE` | `/api/v1/data-purpose/{fides_key}?force=true` | Delete (force removes consumer assignments) |

**Deletion safety:** Deleting a purpose that is still assigned to consumers requires `?force=true`. Without it, the API raises a `ResourceInUseError` to prevent accidental removal of active purpose assignments.

## Data Consumer

A Data Consumer represents *who* accesses data — a team, service account, project, or individual. Consumers are assigned purposes that declare their legitimate reasons for accessing data.

**Key attributes:**
- `name` — Display name (e.g., "Analytics Team")
- `description` — Optional description
- `type` — Classification: `group`, `project`, `system`, `gcp_service_account`, `gcp_iam_role`, `google_group`, or `unresolved`
- `scope` — Scope dictionary carrying the full namespace chain for identity resolution (e.g., `{"domain": "example.com", "project_id": "my-project", "role": "roles/bigquery.dataViewer"}`)
- `contact_email` — Used for identity resolution (matching query authors to consumers)
- `contact_slack_channel` — Slack channel for notifications
- `contact_details` — Additional contact metadata (dict)
- `purposes` / `purpose_fides_keys` — The declared purposes this consumer operates under (response only)
- `tags` — Freeform labels for filtering
- `egress` / `ingress` — Data flow metadata
- `data_shared_with_third_parties` — Third-party sharing flag
- `third_parties` — Description of third parties data is shared with
- `shared_categories` — Data categories shared with third parties
- `system_fides_key` — When the consumer is a registered Fides system (dual representation; response only)
- `vendor_id` — Vendor identifier when consumer is a system (response only)

**API endpoints:**
| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/api/v1/data-consumer` | Create a consumer |
| `GET` | `/api/v1/data-consumer` | List (filterable by `type`, `purpose_fides_key`, `tags`, `scope`) |
| `GET` | `/api/v1/data-consumer/{id}` | Get a single consumer |
| `PUT` | `/api/v1/data-consumer/{id}` | Update a consumer |
| `DELETE` | `/api/v1/data-consumer/{id}` | Delete a consumer |
| `PUT` | `/api/v1/data-consumer/{id}/purpose` | Batch-assign purposes (replace) |
| `POST` | `/api/v1/data-consumer/{id}/purpose/{fides_key}` | Add a single purpose |
| `DELETE` | `/api/v1/data-consumer/{id}/purpose/{fides_key}` | Remove a single purpose |

**Identity resolution:** When a SQL query is intercepted, the system identifies the consumer through a resolution chain:

1. **Fast path — email match:** exact match on `DataConsumer.contact_email`
2. **Platform path — prefetched attributes:** If Identity Group Providers are active (e.g., GCP, Google Workspace), the `CompositeIdentityGroupResolver` uses prefetched IdentityGroups from all providers to perform strict matching on both `consumer.type` AND `consumer.scope` keys
3. **Multi-match — union purposes:** If multiple consumers match a single identity (e.g., a user belongs to multiple Google Groups and has a GCP IAM role), purposes are **unioned** across all matched consumers (most permissive)

If no consumer matches, the user is "unresolved" with zero purposes — all dataset accesses become violations (recorded as gaps).

**Consumer types endpoint:** The `GET /api/v1/plus/pbac/consumer-types` endpoint returns the supported consumer types across all registered Identity Group Providers. Each `ConsumerTypeDescriptor` includes a `platform` field (e.g., `gcp`, `google_workspace`) and `scope_keys` describing the required scope dimensions. The UI uses this to populate the type dropdown and scope fields when creating consumers.

**Dual representation:** Existing Fides systems can also act as consumers. The `DataConsumerEntity` supports construction from both a `DataConsumer` record and a `System` record, allowing organizations to leverage their existing system registry.

## Data Producer

A Data Producer represents *who is responsible* for data — the team or individual that owns, registers, and manages datasets. Producers can be linked to monitors for automated data discovery.

**Key attributes:**
- `name` — Display name (e.g., "Data Engineering Team")
- `description` — Optional description
- `external_id` — External identifier
- `monitor_id` — Optional link to a Fides monitor for automated discovery
- `contact_email`, `contact_slack_channel`, `contact_details` — Contact information
- `member_ids` — Fides user IDs of team members

**API endpoints:**
| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/api/v1/data-producer` | Create a producer |
| `GET` | `/api/v1/data-producer` | List producers with pagination |
| `GET` | `/api/v1/data-producer/{id}` | Get a single producer |
| `PUT` | `/api/v1/data-producer/{id}` | Update a producer |
| `DELETE` | `/api/v1/data-producer/{id}` | Delete a producer |
| `PUT` | `/api/v1/data-producer/{id}/member` | Batch-set team members (replace) |
| `POST` | `/api/v1/data-producer/{id}/member/{user_id}` | Add a team member |
| `DELETE` | `/api/v1/data-producer/{id}/member/{user_id}` | Remove a team member |

## Access Control Dashboard

On top of these entities, we've built a dashboard API that provides real-time visibility into compliance:

| Endpoint | What it shows |
|---|---|
| `/access-control/summary` | Headline stats: total violations, total requests, trend, active consumers, total policies |
| `/access-control/requests` | Timeseries chart data (requests vs. violations over time); supports `interval` parameter (hours) for bucket size |
| `/access-control/consumers` | Per-consumer breakdown (request count, violation count); supports `order_by` parameter |
| `/access-control/filters` | Available filter values for cascading filters (consumers, policies, datasets, data uses, controls) |
| `/access-control/violations` | Paginated aggregate findings (grouped by policy/control); supports `sort_by` and `sort_direction` |
| `/access-control/violations/logs` | Cursor-paginated raw violation log (individual events) |
| `/access-control/violations/{id}` | Full detail for a single violation (includes `policy_description`) |
| `/access-control/violations/{id}/reason` | AI-generated human-readable explanation of a violation |

All dashboard endpoints support filtering by consumer, policy, dataset, data use, and date range. The violations and violation logs endpoints additionally support filtering by `control`.

## Identity Group Providers

The identity resolution system is extensible via the `IdentityGroupProvider` Protocol. Each cloud platform implements this protocol to provide platform-specific identity discovery. Identity Group Providers are **separate from SSO/Identity Providers** — each provider has its own credentials.

**Core types:**
- `ConsumerTypeDescriptor` — Declares a type of identity grouping a provider supports (e.g., `google_group`, `gcp_iam_role`, `gcp_service_account`). Includes `scope_keys: dict[str, str]` describing the required scope dimensions for consumers of that type. Surfaced to the UI via the consumer-types endpoint.
- `IdentityGroup` — Represents a group/role/service-account that a user belongs to, with `type` (matches `ConsumerTypeDescriptor.key`) and `value`.

**Protocol interface:**
- `supported_consumer_types` — Declares what consumer types this provider can resolve
- `prefetch(user_identities)` — Bulk-fetch identity attributes for a batch of users (consumer-driven: only queries groups/roles registered as DataConsumers)
- `resolve(identity)` — Resolve a user identity to a `DataConsumerEntity`
- `list_scopes()` — Enumerate all discoverable scope combinations (powers the consumer form dropdown)

**GCP implementation** (`GCPIdentityGroupProvider`) supports:
- `gcp_iam_role` — GCP IAM role bindings (fetched via Cloud Resource Manager API); uses `browser` role, no delegation needed
- `gcp_service_account` — Service accounts extracted from IAM policy bindings (no extra API call)

**Google Workspace implementation** (`GoogleWorkspaceIdentityGroupProvider`) supports:
- `google_group` — Google Workspace groups; members fetched via Cloud Identity API, group listing via Admin Directory API. Requires `delegation_subject` for domain-wide delegation. Configured as a separate integration type (`google_workspace`) with its own connection and credentials.

**Provider configuration:** Each provider is backed by a persistent `IdentityGroupProviderConfig` (stored in Redis) that pairs a `provider_type` with a `ConnectionConfig`. Created via the "Identity resolution" tab on the integration page. CRUD endpoints at `/api/v1/identity-group-provider`.

**Available scopes:** `GET /api/v1/plus/pbac/available-scopes` queries all enabled providers and returns discoverable identities (groups, roles, service accounts) not yet assigned to a consumer. The consumer form uses this to populate the scope dropdown.

**Composite resolution:** The `CompositeIdentityGroupResolver` chains multiple providers together in the poll flow, aggregating identity groups across all registered platforms in a single prefetch/resolve cycle. Dataset purposes are loaded from the DB during the poll flow.

Prefetch results are cached in Redis with a 15-minute TTL.

**OSS fallback:** Fides OSS provides a simpler `RedisIdentityResolver` that resolves via `contact_email` matching only, without platform API calls.

## Security Model

Every endpoint is protected by granular OAuth scopes:

| Scope | Grants |
|---|---|
| `data_purpose:create/read/update/delete` | Full CRUD on purposes |
| `data_consumer:create/read/update/delete` | Full CRUD on consumers + consumer-types + available-scopes |
| `data_producer:create/read/update/delete` | Full CRUD on producers |
| `identity_group_provider:create/read/update/delete` | Full CRUD on identity group provider configs + test + available-scopes |
| `policy_violation:read` | Access to the dashboard and violation logs |

## Architecture Notes
- **Evaluation engine:** The core PBAC evaluation logic (purpose overlap checking) lives in Fides OSS. Fidesplus adds platform connectors (BigQuery, Snowflake), Identity Group Providers for advanced identity resolution, and the dashboard.
- **Persistence:** Data Purposes, Data Consumers, and Data Producers are stored in Redis (not the relational database) using a `RedisRepository[T]` base class with JSON serialization and Redis sorted set indexes for filtering. Violations are also persisted in Redis alongside query access log entries.
- **Identity Group Providers:** Fidesplus implements the `IdentityGroupProvider` Protocol for each supported platform (currently GCP and Google Workspace). Each provider handles consumer-driven prefetching, scope-based IdentityGroup matching, multi-match purpose union, and scope enumeration (`list_scopes`). Providers are chained via `CompositeIdentityGroupResolver`. Each provider is backed by a persistent `IdentityGroupProviderConfig` entity that references a `ConnectionConfig` for credentials.
- **Integration types:** Google Workspace is a dedicated integration type (`connection_type=google_workspace`) with its own secrets schema (keyfile + delegation_subject + domain). BigQuery integrations support both query logging and GCP identity resolution. The "Identity resolution" tab on the integration page manages the provider config.
- **Service-repository pattern:** All entities follow the Routes → Services → Repositories → Entities layered architecture with DI via `deps.py` factory functions.
