# Policy Engine Sidecar: Stateful Cache Design

## Status
Draft — for review before implementation.

## Context

The Go policy engine sidecar currently accepts fixture data per-request. Every evaluation call requires the fidesplus Python service to serialize consumers, purposes, datasets, and policies into JSON and send them over HTTP. At batch scale (thousands of query log entries per poll cycle), this per-request serialization is the bottleneck — not the evaluation itself.

The sidecar needs to hold evaluation-relevant data in memory and serve evaluations directly, with Python only involved for CRUD writes, cache invalidation, and result persistence.

## Architecture

```
Python (fidesplus)                         Go (sidecar, in fidesplus repo)
┌──────────────────────┐                  ┌──────────────────────────┐
│                      │                  │                          │
│  CRUD APIs           │  invalidate      │  In-memory fixture cache │
│  - POST consumer     │ ──────────────►  │  - consumers  map[email] │
│  - PUT policy        │  POST /v1/cache  │  - purposes   map[key]   │
│  - DELETE dataset    │                  │  - datasets   + table idx│
│                      │                  │  - policies   []sorted   │
│  Query log poller    │  evaluate        │                          │
│  - fetch from BQ     │ ──────────────►  │  Evaluation endpoints    │
│  - send identities   │  evaluate-batch  │  - /v1/evaluate-batch    │
│    + table refs      │                  │  - /v1/evaluate-pipeline │
│  - persist results   │ ◄──────────────  │  - /v1/evaluate-purpose  │
│                      │  results JSON    │  - /v1/evaluate-policies │
└──────────────────────┘                  └──────────────────────────┘
```

## Source of Truth

- **Redis**: consumers, purposes, identity provider configs
- **Postgres**: datasets (with data_purposes), access policies (with YAML versions), system privacy declarations

The sidecar cache is a **read replica**. Fidesplus Python is the only writer. The sidecar never writes back to Redis or Postgres.

## Cache API

All cache endpoints live on the sidecar (`policy-engine-sidecar/`). Auth uses the same JWE validation as evaluation endpoints but with a `cache:write` scope.

### `POST /v1/cache` — Full cache load

Called at sidecar startup and on full invalidation. Python serializes the complete fixture set from Redis + Postgres and pushes it.

```json
{
  "consumers": {
    "alice@example.com": {
      "name": "Analytics Team",
      "members": ["alice@example.com", "priya@example.com"],
      "purposes": ["analytics"]
    }
  },
  "purposes": {
    "analytics": {
      "fides_key": "analytics",
      "name": "Analytics",
      "data_use": "analytics.reporting"
    }
  },
  "datasets": {
    "purposes": {
      "sales": {
        "dataset_key": "sales",
        "purpose_keys": ["billing"],
        "collection_purposes": {"orders": ["billing"]}
      }
    },
    "tables": {"orders": "sales", "invoices": "sales"}
  },
  "policies": [
    {
      "key": "allow-analytics-on-billing",
      "priority": 100,
      "enabled": true,
      "decision": "ALLOW",
      "match": {"data_use": {"any": ["essential.service.payment_processing"]}},
      "action": {"message": "Analytics permitted on billing data"}
    }
  ]
}
```

Response: `200 OK` with cache metadata:
```json
{
  "consumers": 5,
  "purposes": 3,
  "datasets": 4,
  "policies": 2,
  "loaded_at": "2026-04-16T12:00:00Z"
}
```

### `PATCH /v1/cache` — Partial invalidation

Called when a single entity type changes. Only the specified keys are replaced; unspecified top-level keys are left untouched.

```json
{
  "consumers": {
    "alice@example.com": { "name": "Analytics Team", "purposes": ["analytics", "billing"] }
  }
}
```

To delete a consumer (member left the org):
```json
{
  "consumers": {
    "alice@example.com": null
  }
}
```

`null` values remove the key from the cache. Non-null values upsert.

Response: same metadata shape as `POST /v1/cache`.

### `DELETE /v1/cache` — Full cache clear

Forces the sidecar to drop all cached data. Evaluation calls return an error until `POST /v1/cache` repopulates it.

Response: `204 No Content`

### `GET /health` — Health check (updated)

Returns cache state alongside the existing health status. The sidecar reports unhealthy until the cache is populated — k8s won't route traffic until then.

```json
{
  "status": "healthy",
  "cache": {
    "populated": true,
    "cache_size_bytes": 48210,
    "loaded_at": "2026-04-16T12:00:00Z",
    "last_invalidation": "2026-04-16T14:30:00Z"
  }
}
```

When cache is empty: `{"status": "unhealthy", "cache": {"populated": false}}` with HTTP 503.

## Evaluation API

### `POST /v1/evaluate-batch` — Batch evaluation (new, primary production endpoint)

Accepts a list of inputs and evaluates each against the **cached** fixtures. No fixture data in the request.

```json
{
  "inputs": [
    {
      "query_id": "bq-job-123",
      "identity": "alice@example.com",
      "tables": [
        {"collection": "orders", "qualified_name": "project.schema.orders"}
      ],
      "context": {}
    },
    {
      "query_id": "bq-job-124",
      "identity": "bob@example.com",
      "tables": [
        {"collection": "campaigns", "qualified_name": "project.schema.campaigns"}
      ],
      "context": {}
    }
  ]
}
```

Response:
```json
{
  "records": [
    {
      "query_id": "bq-job-123",
      "identity": "alice@example.com",
      "consumer": "Analytics Team",
      "dataset_keys": ["sales"],
      "is_compliant": true,
      "violations": [...],
      "gaps": [],
      "total_accesses": 1
    }
  ]
}
```

Error when cache is not populated:
```json
{"error": "cache not populated — call POST /v1/cache first"}
```

### Existing endpoints (unchanged)

- `POST /v1/evaluate-pipeline` — single evaluation with fixtures in the request (for CLI / testing)
- `POST /v1/evaluate-purpose` — purpose-only primitive
- `POST /v1/evaluate-policies` — policy-only primitive

## Go Implementation

### Cache struct

New package in the sidecar: `internal/cache/cache.go`

```go
type Cache struct {
    mu       sync.RWMutex
    fixtures pipeline.Fixtures
    loaded   bool
    loadedAt time.Time
    lastInv  time.Time
}

func (c *Cache) Load(f pipeline.Fixtures)
func (c *Cache) Patch(partial PartialUpdate)
func (c *Cache) Clear()
func (c *Cache) Get() (pipeline.Fixtures, error)  // error if not loaded
func (c *Cache) Status() CacheStatus
```

Thread-safe via `sync.RWMutex`. Evaluation reads take a read lock; cache writes take a write lock. Evaluations are never blocked by other evaluations — only by cache updates, which are infrequent and fast.

### Handler additions

```go
func (h *Handler) Health(c echo.Context) error         // GET /health (updated: includes cache state)
func (h *Handler) LoadCache(c echo.Context) error      // POST /v1/cache
func (h *Handler) PatchCache(c echo.Context) error     // PATCH /v1/cache
func (h *Handler) ClearCache(c echo.Context) error     // DELETE /v1/cache
func (h *Handler) EvaluateBatch(c echo.Context) error  // POST /v1/evaluate-batch
```

`EvaluateBatch` acquires a read lock, iterates inputs calling `pipeline.Evaluate()` per input against cached fixtures, releases the lock, returns all records.

### Auth scopes

| Endpoint | Scope |
|---|---|
| Cache mutations (POST/PATCH/DELETE /v1/cache) | `cache:write` |
| Health (GET /health) | No auth (k8s probes need unauthenticated access) |
| Evaluation endpoints | `policy:evaluate` |

## Python Integration (fidesplus)

### New module: `fidesplus/service/pbac/engine/cache_sync.py`

Responsible for loading the full fixture set and pushing it to the sidecar.

```python
def populate_sidecar_cache(db: Session, cache: FidesopsRedis) -> None:
    """Load all PBAC data from Redis + Postgres and push to sidecar."""
    sidecar = get_sidecar_client()

    consumers = load_consumers_from_redis(cache)
    purposes = load_purposes_from_redis(cache)
    datasets = load_datasets_from_postgres(db)
    policies = load_policies_from_postgres(db)

    sidecar.load_cache({
        "consumers": consumers,
        "purposes": purposes,
        "datasets": datasets,
        "policies": policies,
    })
```

### Invalidation hooks

Each CRUD service that changes evaluation-relevant data calls `PATCH /v1/cache` after the write succeeds.

| CRUD operation | Invalidation |
|---|---|
| Create/update/delete consumer | `PATCH {"consumers": {email: data_or_null}}` |
| Create/update/delete purpose | `PATCH {"purposes": {key: data_or_null}}` |
| Create/update/delete dataset | `PATCH {"datasets": {full rebuild}}` — the table index (collection name → dataset key) is derived, so a surgical patch is fragile. Simpler to push the full datasets on any dataset change. |
| Create/update/delete policy | `PATCH {"policies": [full sorted list]}` — policies are priority-sorted and the full set is needed for first-decisive-match evaluation. |

Datasets and policies use full-category pushes because their internal data structures (table index, priority-sorted list) make surgical patching fragile. These changes are operator-driven and infrequent — the cost is negligible.

### Integration points for invalidation

| Fidesplus module | Trigger | Invalidation |
|---|---|---|
| `DataConsumerService.create/update/delete` | Consumer CRUD | Patch consumers |
| `DataPurposeService.create/update/delete` | Purpose CRUD | Patch purposes |
| Dataset API routes | Dataset data_purposes changed | Patch datasets (full) |
| `AccessPolicyService.create/update/delete` | Policy CRUD | Patch policies (full) |
| `IdentityGroupProviderConfigService` | Provider config changed | Patch consumers (full — consumer resolution may change) |
| Sidecar startup / health recovery | Sidecar restarts | Full `POST /v1/cache` |

### Batch evaluation in QueryLogService

```python
# Current (per-statement, Python in the loop):
for entry in entries:
    result = eval_service.evaluate(entry)  # N HTTP calls, Python serialization

# Target (one call, Go does everything):
inputs = [
    {
        "query_id": entry.external_job_id,
        "identity": entry.identity,
        "tables": [
            {"collection": ref.table, "qualified_name": ref.qualified_name}
            for ref in entry.referenced_tables
        ],
    }
    for entry in entries
]
records = sidecar.evaluate_batch(inputs)  # 1 HTTP call, Go evaluates all
for record in records:
    persist_result(record)  # Python: Redis/Postgres writes only
```

Python's role in the batch path becomes: call the connector (fetch query logs), send one HTTP batch, persist results.

### Sidecar client additions

```python
# sidecar_client.py — new methods

def load_cache(self, fixtures: dict) -> dict:
    resp = self._client.post("/v1/cache", json=fixtures)
    resp.raise_for_status()
    return resp.json()

def patch_cache(self, partial: dict) -> dict:
    resp = self._client.patch("/v1/cache", json=partial)
    resp.raise_for_status()
    return resp.json()

def clear_cache(self) -> None:
    resp = self._client.delete("/v1/cache")
    resp.raise_for_status()

def evaluate_batch(self, inputs: list[dict]) -> list[dict]:
    resp = self._client.post("/v1/evaluate-batch", json={"inputs": inputs})
    resp.raise_for_status()
    return resp.json()["records"]
```

## Consistency Model

**Eventual consistency with bounded staleness.** Between a CRUD write and the `PATCH /v1/cache` arriving at the sidecar, evaluations may use stale data. The staleness window is the latency of the patch call — typically <10ms on localhost (k8s pod network).

This is acceptable because query log evaluation is already asynchronous (polling on a schedule). A few milliseconds of staleness doesn't change outcomes.

If strong consistency is ever needed (e.g., real-time enforcement where a policy change must take effect immediately), the CRUD endpoint can block until the sidecar acknowledges the cache update before returning the API response to the caller.

## Failure Modes

| Failure | Behavior |
|---|---|
| Sidecar restarts | Cache empty. `GET /health` returns unhealthy (k8s won't route traffic). Startup hook calls `POST /v1/cache`. Health goes healthy once cache is populated. |
| Invalidation call fails | Log warning, retry with backoff. Stale data used until retry succeeds. |
| Sidecar unreachable | Evaluation fails. No silent fallback to Python — surface the error. |
| Cache write interrupted | `sync.RWMutex` ensures the in-memory update is atomic. HTTP failure = no partial state. |

## Migration Path

1. **Implement `internal/cache`** in the sidecar (thread-safe fixture cache with Load/Patch/Clear/Get/Status)
2. **Add cache + batch endpoints** to the sidecar handler
3. **Add `cache_sync.py`** to fidesplus (population + invalidation helpers)
4. **Wire invalidation hooks** into CRUD services (consumer, purpose, dataset, policy)
5. **Add cache population** to sidecar startup / readiness flow
6. **Rewrite `QueryLogService.poll_source()`** to use `/v1/evaluate-batch`
7. **Remove stepwise Python evaluation** from `sidecar_service.py`

Steps 1-2 can ship as one PR (Go-only). Steps 3-5 as a second (Python integration). Steps 6-7 as a third (batch migration + cleanup).

## Decisions

1. **Dataset table index rebuild on PATCH** — full-category push on any dataset change. The table index is derived and surgical patching is fragile. This is sufficient for current scale. Incremental index updates are a possible future performance enhancement if dataset CRUD frequency increases.

2. **Cache warming on deploy** — the sidecar's readiness probe (`GET /health`) returns unhealthy until the cache is populated. Kubernetes won't route traffic to the pod until the startup hook calls `POST /v1/cache` and the sidecar acknowledges. No evaluation requests can hit an empty cache.

3. **Cache size monitoring** — `GET /health` includes `cache_size_bytes` (approximate in-memory size). The health endpoint returns metadata only, never the cache contents themselves.
