# PBAC — Fidesplus Extensions

Fidesplus extends the Fides OSS PBAC engine with Identity Group Providers for platform-specific identity resolution, query log ingestion, violation/gap persistence, and a compliance counter.

For the core evaluation engine, types, and OSS extension points, see `fides/src/fides/service/pbac/README.md`.

## What fidesplus adds

The OSS engine evaluates a single `RawQueryLogEntry` and returns an `EvaluationResult`. Fidesplus wraps that with:

1. **Query log connectors** — fetch query audit logs from cloud platforms (BigQuery, Snowflake, etc.)
2. **Identity Group Providers** — query cloud IAM/RBAC APIs to resolve user roles and group memberships across multiple platforms
3. **Identity Group Provider Configs** — persistent CRUD entities that pair a provider type with a connection's credentials
4. **Available scopes** — API to enumerate discoverable identities (groups, roles, service accounts) from configured providers, powering the consumer form's scope dropdown
5. **Persistence** — store violations, gaps, and compliant query counts
6. **Consumer types API** — surface available identity groupings per platform for the UI
7. **Polling orchestration** — Celery tasks + APScheduler for periodic log ingestion

## Full flow

```
Celery task fires
  │
  ▼
QueryLogService.poll_source(config_key)
  │
  ├─ 1. QueryLogConnector.fetch_entries()
  │    BigQuery Jobs API / Snowflake QUERY_HISTORY / Mock
  │    Returns: list[RawQueryLogEntry]
  │
  ├─ 2. Build evaluation service for this batch
  │    ├─ Load dataset purposes from DB
  │    ├─ Create IdentityGroupProviders for each registered platform
  │    │    ├─ GCPIdentityGroupProvider (IAM roles, service accounts)
  │    │    └─ GoogleWorkspaceIdentityGroupProvider (Google Groups)
  │    ├─ Wrap in CompositeIdentityGroupResolver (chains multiple providers)
  │    ├─ prefetch(user_identities) — bulk IAM/group lookup across all providers
  │    └─ Inject the composite resolver into InProcessPBACEvaluationService
  │
  ├─ 3. Per entry:
  │    ├─ Deduplicate (skip if external_job_id already processed)
  │    ├─ InProcessPBACEvaluationService.evaluate(entry)
  │    │    → identity resolution → dataset resolution → engine → policy
  │    │
  │    ├─ COMPLIANT (no gaps): increment ComplianceCounterRepository
  │    ├─ VIOLATION: store QueryAccessLogEntity + PolicyViolationEntity
  │    └─ GAP: store QueryAccessLogEntity + CoverageGapEntity
  │
  └─ 4. Update watermark for next poll
```

## Key concepts

### Identity Group Providers

The OSS `IdentityResolver` matches by email — you register consumers manually and it does exact matches. Fidesplus extends this with `IdentityGroupProvider`, which queries cloud platform APIs to discover what roles, groups, and service accounts a user belongs to.

Identity Group Providers are **separate from SSO/Identity Providers** — each provider has its own credentials and resolves a specific platform's identity groupings. For example, GCP IAM roles are resolved by `GCPIdentityGroupProvider` using a service account with the `browser` role, while Google Workspace groups are resolved by `GoogleWorkspaceIdentityGroupProvider` using domain-wide delegation via the Admin Directory API.

The resolution is **consumer-driven**: we only query the platform for groups/roles that are registered as DataConsumers. This bounds API calls by O(registered_consumers), not O(groups_in_org).

The poll flow uses `CompositeIdentityGroupResolver` to chain multiple providers together, aggregating results across all registered platforms.

See `identity/README.md` for the full guide including how to add new platforms.

### Identity Group Provider Configs

Each provider is backed by a persistent `IdentityGroupProviderConfig` entity (stored in Redis) that pairs a `provider_type` with a `ConnectionConfig` for credentials. This decouples the provider's identity-specific settings (domain, delegation_subject) from the raw credential storage.

Each integration type maps to a provider type:
- **BigQuery** integration → `gcp` provider (IAM roles + service accounts)
- **Google Workspace** integration → `google_workspace` provider (Google Groups)

The UI creates these configs via the "Identity resolution" tab on the integration page. The config is then used by both the poll flow (to instantiate providers) and the available-scopes endpoint (to enumerate discoverable identities).

CRUD endpoints: `POST/GET/PUT/DELETE /api/v1/identity-group-provider`
Test: `POST /api/v1/identity-group-provider/{id}/test`
Per-provider scopes: `GET /api/v1/identity-group-provider/{id}/available-scopes`

### Available scopes

`GET /api/v1/plus/pbac/available-scopes` queries all enabled identity group providers and returns the discoverable identities (Google Groups, IAM roles, service accounts) that haven't already been assigned to a consumer. The consumer form uses this to populate the scope dropdown.

Each provider implements `list_scopes()`:
- **GCP**: fetches the project IAM policy and extracts roles + service accounts (single API call)
- **Google Workspace**: lists all groups in the domain via the Admin Directory API

### Consumer types

Each Identity Group Provider declares `ConsumerTypeDescriptor`s — the types of identity groupings it supports. These are surfaced via `GET /api/v1/plus/pbac/consumer-types` so the UI can show the right options when creating a DataConsumer.

GCP types: `gcp_iam_role`, `gcp_service_account`
Google Workspace types: `google_group`

Each `ConsumerTypeDescriptor` includes `scope_keys: dict[str, str]` that describe the scope dimensions required for consumers of that type (e.g., `{"domain": "Google Workspace domain", "project_id": "GCP project ID"}`).

When a provider prefetches, it builds a list of `IdentityGroup(type, value)` per user — e.g., `IdentityGroup(type="google_group", value="analytics-team@co.com")`. Matching is **strict**: both `consumer.type` AND `consumer.scope` keys must match an `IdentityGroup`. When a user matches multiple consumers, purposes are **unioned** (most permissive).

### Scope-based consumers

DataConsumers have a `scope: dict[str, str]` that carries the full namespace chain for the consumer's identity. The scope always includes the domain at the top level plus any platform-specific identifiers. For example:

- A Google Group consumer: `{"domain": "example.com", "group_email": "analytics@example.com"}`
- A GCP IAM role consumer: `{"domain": "example.com", "project_id": "my-project", "role": "roles/bigquery.dataViewer"}`
- A GCP service account: `{"domain": "example.com", "project_id": "my-project", "email": "etl@my-project.iam.gserviceaccount.com"}`

This replaces the previous `connection_config_key` + `external_id` approach, enabling consumers to be scoped precisely without being tied to a single connection configuration.

### Purposes

Purposes are **universal** — they are not scoped to any platform. A purpose like "Marketing Analytics" applies regardless of whether the consumer is a GCP IAM role or a Google Workspace group. Dataset purposes are loaded from the DB during the poll flow.

### Storage strategy

| Outcome | What's stored | Why |
|---------|--------------|-----|
| Compliant | Counter per source per hour (`ComplianceCounterRepository`) | Most queries pass — individual records would be wasteful |
| Violation | `QueryAccessLogEntity` + `PolicyViolationEntity` per violation | Need full audit trail for drill-down |
| Gap | `QueryAccessLogEntity` + `CoverageGapEntity` per gap | Immutable records for tracking coverage over time |

Gaps are never backfilled. When config is addressed, future queries evaluate correctly. Old gaps stay as historical records.

### Prefetch lifecycle

Multiple `IdentityGroupProvider` instances are wrapped in a `CompositeIdentityGroupResolver` that handles both `prefetch()` and `resolve()` within a batch. `QueryLogService._build_eval_service_for_batch()` creates the providers, wraps them in the composite resolver, calls prefetch with the batch's user identities, then injects it into a new `InProcessPBACEvaluationService` for that batch.

Prefetched data is cached in Redis with a configurable TTL (default 15 minutes) so subsequent batches within the window skip the API calls.

## Extension points (fidesplus-specific)

| Interface | Kind | What it does |
|-----------|------|-------------|
| `IdentityGroupProvider` | Protocol | Cloud IAM/RBAC resolution with `prefetch()` + `resolve()` |
| `QueryLogConnector` | ABC | Fetch query logs from a platform |
| `ComplianceCounterRepository` | ABC | Count compliant queries (Redis now, Postgres later) |

## Package structure

```
fidesplus/service/pbac/
  identity/
    README.md           How to add new Identity Group Providers
    interface.py        IdentityGroupProvider Protocol + ConsumerTypeDescriptor + IdentityGroup
    gcp.py              GCPIdentityGroupProvider (IAM roles, service accounts via Resource Manager API)
    google_workspace.py GoogleWorkspaceIdentityGroupProvider (Google Groups via Admin Directory API)
    composite.py        CompositeIdentityGroupResolver (chains multiple providers)
    mock.py             MockIdentityGroupProvider (sample data, no credentials needed)
    __init__.py         Provider registry + instantiate_from_config factory
    config/
      entities.py       IdentityGroupProviderConfig dataclass
      repository.py     Redis-backed CRUD repository
      service.py        Validation + facade over repository

  connectors/
    interface.py        QueryLogConnector ABC
    types.py            PlatformQueryIdentity, PlatformQueryLogEntry
    bigquery_connector.py   Fetches query logs from BigQuery Jobs API
    mock_connector.py       Generates synthetic entries for demos

  query_log/
    service.py          QueryLogService (orchestrates polling + evaluation + persistence)
    counters.py         ComplianceCounterRepository ABC + Redis implementation
    entities.py         QueryLogConfig, QueryAccessLogEntity
    repository.py       Redis repositories for configs + access logs
    tasks.py            Celery task for periodic polling
    scheduler.py        APScheduler job management

  violations/
    entities.py         PolicyViolationEntity
    repository.py       ViolationRedisRepository
    service.py          Violation dashboard queries (summary, timeseries, logs)

  gaps/
    entities.py         CoverageGapEntity
    repository.py       CoverageGapRedisRepository

  consumers/
    entities.py         DataConsumerEntity (with scope: dict[str, str])
    repository.py       DataConsumerRedisRepository
    service.py          DataConsumerService (schema→entity conversion)
```
