# ENG-2787: PBAC Evaluation Service Boundary + OSS Architecture

## Context

The PBAC evaluation logic currently lives entirely in fidesplus (PR #3247). The goal is to architect this so the **entire evaluation stack** is open source in Fides, extractable to a standalone Go service later.

**Open source in Fides OSS:**
- SQL parser (generic SQL -> `RawQueryLogEntry`)
- `PBACEvaluationService` Protocol + flat result types
- PBAC engine (purpose-based evaluation)
- Policy v2 engine (priority-based match/unless evaluation)
- Policy v2 models + CRUD + `/evaluate` endpoint
- Consumer/purpose resolution

**Stays in fidesplus:**
- Premium connectors (BigQuery, Snowflake audit log ingestion)
- Platform identity resolution (query BigQuery IAM / Snowflake RBAC for user roles)
- Access control dashboard (routes, UI, violation queries)
- Connector config management

---

## Identity Resolution

`IdentityResolver` is a Protocol -- OSS provides a base implementation, fidesplus extends with platform IAM lookups.

**OSS `BasicIdentityResolver`:**
1. `contact_email` match -- direct consumer lookup
2. `external_id` match -- if query carries a principal/role identifier
3. `members` list match -- admin-managed group membership on the consumer

**Fidesplus `PlatformIdentityResolver`:**
1. Query BigQuery IAM / Snowflake RBAC to get the user's roles/groups
2. Match roles against consumer `external_id` (e.g., `external_id="analytics-viewer"`)
3. Falls back to OSS resolution chain

The consumer's `external_id` is the bridge -- it references an external role or group. In fidesplus the connector resolves email to platform roles automatically. In OSS the admin configures membership manually via a `members` field on the consumer.

`DataConsumerEntity` needs a `members: list[str]` field for OSS group membership.

---

## Architecture

### System Overview

```
+---------------------------------------------------------------------------------+
|                              Fides OSS                                          |
|                                                                                 |
|  +---------------------+     +----------------------------------------------+   |
|  |   SQL Parser         |     |         PBACEvaluationService               |   |
|  |                      |     |         (Protocol + InProcess impl)         |   |
|  |  raw SQL             |     |                                              |   |
|  |    |                 |     |  RawQueryLogEntry                            |   |
|  |    v                 |     |    |                                          |   |
|  |  sqlglot extracts    |---->|    v                                          |   |
|  |  table references    |     |  +------------------------------------+       |   |
|  |    |                 |     |  |  1. Identity Resolution            |       |   |
|  |    v                 |     |  |     IdentityResolver Protocol      |       |   |
|  |  RawQueryLogEntry    |     |  |     +-------------------------+    |       |   |
|  +---------------------+     |  |     | BasicIdentityResolver   |    |       |   |
|                               |  |     |  a. contact_email match |    |       |   |
|                               |  |     |  b. external_id match   |    |       |   |
|                               |  |     |  c. members list match  |    |       |   |
|                               |  |     +-------------------------+    |       |   |
|                               |  |          |                         |       |   |
|                               |  |          v ResolvedConsumer        |       |   |
|                               |  |                                    |       |   |
|                               |  |  2. Dataset Resolution             |       |   |
|                               |  |     TableRef --> fides dataset key |       |   |
|                               |  |          |                         |       |   |
|                               |  |          v dataset_keys            |       |   |
|                               |  |                                    |       |   |
|                               |  |  3. Purpose Map                    |       |   |
|                               |  |     consumer --> purpose_keys      |       |   |
|                               |  |     dataset  --> purpose_keys      |       |   |
|                               |  |          |                         |       |   |
|                               |  |          v                         |       |   |
|                               |  |                                    |       |   |
|                               |  |  4. PBAC Engine                    |       |   |
|                               |  |     consumer_purposes              |       |   |
|                               |  |       intersection                 |       |   |
|                               |  |     dataset_purposes               |       |   |
|                               |  |       |                            |       |   |
|                               |  |       +-- overlap --> COMPLIANT    |       |   |
|                               |  |       |                            |       |   |
|                               |  |       +-- no overlap --> VIOLATION |       |   |
|                               |  |              |                     |       |   |
|                               |  |              v                     |       |   |
|                               |  |                                    |       |   |
|                               |  |  5. Policy v2 Engine               |       |   |
|                               |  |     AccessPolicyEvaluator Protocol |       |   |
|                               |  |       |                            |       |   |
|                               |  |       +-- ALLOW --> suppress       |       |   |
|                               |  |       +-- DENY  --> confirm        |       |   |
|                               |  |       +-- NO_DECISION --> confirm  |       |   |
|                               |  |                                    |       |   |
|                               |  +------------------------------------+       |   |
|                               |          |                                    |   |
|                               |          v                                    |   |
|                               |    EvaluationResult                           |   |
|                               |      query_id, consumer, dataset_keys,        |   |
|                               |      is_compliant, violations[]               |   |
|                               +----------------------------------------------+   |
|                                                                                 |
|  +--------------------------+   +--------------------------------+              |
|  |  Policy v2 CRUD          |   |  Consumer / Purpose CRUD       |              |
|  |  POST /api/v1/policy     |   |  POST /api/v1/data-consumer   |              |
|  |  GET  /api/v1/policy     |   |  POST /api/v1/data-purpose    |              |
|  |  POST /api/v1/evaluate   |   |  PUT  .../purposes            |              |
|  |  POST /api/v1/control    |   +--------------------------------+              |
|  +--------------------------+                                                   |
+---------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------+
|                              Fidesplus                                          |
|                                                                                 |
|  +----------------------------------+   +------------------------------------+  |
|  |  Platform Connectors              |   |  Platform Identity Resolution     |  |
|  |                                    |   |                                    |  |
|  |  BigQuery audit logs --+           |   |  BigQuery IAM  --+                |  |
|  |  Snowflake query hist -+           |   |  Snowflake RBAC -+                |  |
|  |  Databricks logs ------+           |   |  Databricks ACLs +                |  |
|  |                        |           |   |         |                          |  |
|  |                        v           |   |         v                          |  |
|  |              RawQueryLogEntry      |   |  user email --> platform roles    |  |
|  +------------------+-----------------+   |  platform roles --> external_id   |  |
|                     |                     |  external_id --> consumer          |  |
|                     |                     |                                    |  |
|                     |                     |  PlatformIdentityResolver          |  |
|                     |                     |  (extends IdentityResolver)        |  |
|                     |                     +------------------------------------+  |
|                     |                                                            |
|                     v                                                            |
|  +--------------------------------------------------------------------------+    |
|  |  QueryLogService (thin orchestrator)                                     |    |
|  |                                                                          |    |
|  |  1. Connector polls/ingests --> RawQueryLogEntry                         |    |
|  |  2. Calls PBACEvaluationService.evaluate(entry)                          |    |
|  |  3. Persists access log                                                  |    |
|  |  4. Records violations --> dashboard                                     |    |
|  +--------------------------------------------------------------------------+    |
|                                                                                  |
|  +--------------------------------------------------------------------------+    |
|  |  Access Control Dashboard                                                |    |
|  |                                                                          |    |
|  |  GET /access-control/summary      GET /access-control/violations/logs    |    |
|  |  GET /access-control/requests     GET /access-control/violations/{id}    |    |
|  |  GET /access-control/consumers    GET /access-control/filters            |    |
|  +--------------------------------------------------------------------------+    |
|                                                                                  |
+---------------------------------------------------------------------------------+
```

### OSS User Flow

```
                    OSS User
                       |
                       |  "SELECT email FROM customers.orders"
                       |  user: analyst@company.com
                       v
              POST /api/v1/pbac/evaluate
                       |
                       v
               +---------------+
               |  SQL Parser    |  sqlglot extracts table refs
               |  (sqlglot)     |  --> TableRef(dataset="customers", table="orders")
               +-------+-------+
                       |
                       v  RawQueryLogEntry
               +-----------------------------------------------+
               |         PBACEvaluationService                  |
               |                                                |
               |  +-----------------------------------------+   |
               |  | Identity: analyst@company.com            |   |
               |  |   members match --> "Analytics Team"     |   |
               |  |   purposes: [marketing_analytics]        |   |
               |  +-----------------------------------------+   |
               |                                                |
               |  +-----------------------------------------+   |
               |  | Dataset: customers                       |   |
               |  |   purposes: [billing_operations]         |   |
               |  +-----------------------------------------+   |
               |                                                |
               |  +-----------------------------------------+   |
               |  | PBAC: {marketing_analytics}              |   |
               |  |   intersect {billing_operations}         |   |
               |  |     = {} --> VIOLATION                   |   |
               |  +-----------------------------------------+   |
               |                                                |
               |  +-----------------------------------------+   |
               |  | Policy v2: any override?                 |   |
               |  |   NO_DECISION --> violation stands       |   |
               |  +-----------------------------------------+   |
               |                                                |
               +-------------------+---------------------------+
                                   |
                                   v
                           EvaluationResult
                           {
                             is_compliant: false,
                             consumer: "Analytics Team",
                             violations: [{
                               dataset: "customers",
                               reason: "purposes do not overlap"
                             }]
                           }
```

### Fidesplus User Flow

```
                    Fidesplus User
                       |
                       |  BigQuery audit log entry
                       |  user: analyst@company.com
                       |  tables: project.dataset.table
                       v
               +---------------+
               |  BQ Connector  |  Fetches from INFORMATION_SCHEMA.JOBS
               |  (fidesplus)   |  Produces RawQueryLogEntry
               +-------+-------+
                       |
                       v
               +---------------+
               |  BQ IAM        |  Queries IAM bindings for user
               |  Resolution    |  analyst@company.com --> roles:
               |  (fidesplus)   |    ["roles/bigquery.dataViewer",
               |                |     "analytics-viewer"]
               +-------+-------+
                       |
                       v  PlatformIdentityResolver
               +-----------------------------------------------+
               |  Match "analytics-viewer" --> consumer         |
               |  external_id="analytics-viewer"                |
               |  --> "Analytics Team" consumer                 |
               +-------+---------------------------------------+
                       |
                       v  RawQueryLogEntry + resolved identity
               +-----------------------------------------------+
               |  PBACEvaluationService.evaluate(entry)         |
               |  (same engine as OSS)                          |
               +-------+---------------------------------------+
                       |
                       v
               +-----------------------------------------------+
               |  QueryLogService                               |
               |    persist access log                          |
               |    record violations --> dashboard             |
               +-----------------------------------------------+
```

---

## Deliverables

### 1. Service boundary types (Fides OSS)

New package: `src/fides/service/pbac/evaluation/`

```python
# types.py -- zero dependencies, JSON/protobuf-portable

@dataclass(frozen=True)
class ResolvedConsumer:
    id: str | None
    name: str
    email: str | None
    type: str                           # "group", "project", "system", "unresolved"
    external_id: str | None             # reference to outside user groups or roles
    system_fides_key: str | None
    purpose_fides_keys: tuple[str, ...]

@dataclass(frozen=True)
class EvaluationViolation:
    dataset_key: str
    collection: str | None
    consumer_purposes: tuple[str, ...]
    dataset_purposes: tuple[str, ...]
    data_use: str | None
    reason: str
    control: str                        # "purpose_restriction" or policy control key

@dataclass(frozen=True)
class EvaluationResult:
    query_id: str
    consumer: ResolvedConsumer
    dataset_keys: tuple[str, ...]
    is_compliant: bool
    violations: tuple[EvaluationViolation, ...]
    total_accesses: int
    timestamp: datetime
    query_text: str | None
```

```python
# interface.py

class PBACEvaluationService(Protocol):
    def evaluate(
        self,
        entry: RawQueryLogEntry,
        dataset_purpose_overrides: dict[str, list[str]] | None = None,
    ) -> EvaluationResult: ...
```

### 2. SQL parser (Fides OSS)

New module: `src/fides/service/pbac/sql_parser.py`

Uses `sqlglot` (new dependency -- pure Python, no native extensions). Needs to be added to Fides requirements.

### 3. RawQueryLogEntry + TableRef (Fides OSS)

Move from `fidesplus/service/pbac/connectors/types.py` to `src/fides/service/pbac/types.py`. Fidesplus updates its imports directly.

### 4. PBAC engine (Fides OSS)

Move from `fidesplus/service/pbac/engine/` to `src/fides/service/pbac/engine/`. Already zero-dependency.

### 5. Policy v2 interface (Fides OSS)

Move `AccessPolicyEvaluator` Protocol and `NoOpPolicyEvaluator` to `src/fides/service/pbac/policies/`.

### 6. Refactored QueryLogService (fidesplus)

Thin orchestrator that depends on `PBACEvaluationService`. Keeps: connectors, access log persistence, violation recording, dashboard.

### 7. InProcessPBACEvaluationService (fidesplus, initially)

Wraps existing resolution + engine + policy code. Moves to OSS as consumer/purpose management migrates.

---

## File Changes

### Fides OSS (new files)

| File | Purpose |
|------|---------|
| `src/fides/service/pbac/__init__.py` | Package |
| `src/fides/service/pbac/types.py` | `RawQueryLogEntry`, `TableRef` |
| `src/fides/service/pbac/sql_parser.py` | Generic SQL -> `RawQueryLogEntry` |
| `src/fides/service/pbac/engine/__init__.py` | PBAC engine (moved from fidesplus) |
| `src/fides/service/pbac/engine/types.py` | Engine types |
| `src/fides/service/pbac/engine/evaluate.py` | `evaluate_access()` |
| `src/fides/service/pbac/engine/reason.py` | Violation reason generation |
| `src/fides/service/pbac/evaluation/__init__.py` | Evaluation service package |
| `src/fides/service/pbac/evaluation/types.py` | `EvaluationResult`, `ResolvedConsumer`, `EvaluationViolation` |
| `src/fides/service/pbac/evaluation/interface.py` | `PBACEvaluationService` Protocol |
| `src/fides/service/pbac/identity/__init__.py` | Identity resolution package |
| `src/fides/service/pbac/identity/interface.py` | `IdentityResolver` Protocol |
| `src/fides/service/pbac/identity/basic.py` | `BasicIdentityResolver` (email + external_id + members) |
| `src/fides/service/pbac/policies/__init__.py` | Policy v2 package |
| `src/fides/service/pbac/policies/interface.py` | `AccessPolicyEvaluator` Protocol + types |
| `src/fides/service/pbac/policies/noop.py` | `NoOpPolicyEvaluator` |

### Fidesplus (modified)

| File | Change |
|------|--------|
| `service/pbac/connectors/types.py` | Remove -- imports from `fides.service.pbac.types` |
| `service/pbac/engine/` | Remove -- imports from `fides.service.pbac.engine` |
| `service/pbac/policies/` | Remove -- imports from `fides.service.pbac.policies` |
| `service/pbac/evaluation/in_process.py` | New -- `InProcessPBACEvaluationService` |
| `service/pbac/query_log/service.py` | Refactored -- thin orchestrator |
| `api/deps.py` | Updated factories |

---

## Migration Path to Go

1. **Today**: `InProcessPBACEvaluationService` in Python (fidesplus)
2. **Later**: Go service implements the same `evaluate(RawQueryLogEntry) -> EvaluationResult` contract
3. **Swap**: Replace `InProcessPBACEvaluationService` with `GrpcPBACEvaluationClient` in `deps.py`
4. **Data sync**: Go service owns its own consumer/purpose/policy stores, Fides syncs via API
