# Mock Query Log Connector

**Source:** feat/pbac-enforcement branch

## Problem Statement

Testing and demoing the PBAC query log flow currently requires either a live BigQuery connection or manually calling the `POST /query-log/ingest` endpoint with handcrafted payloads. There's no way to generate a continuous, realistic stream of query log data from the system itself.

We need a mock/test query log connector that conforms to the `QueryLogConnector` interface but generates large volumes of synthetic query logs based on the datasets and data consumers that already exist in the system.

## Requirements

- Implements `QueryLogConnector` (same interface as BigQuery)
- Generates realistic `RawQueryLogEntry` data using existing datasets and data consumers from Redis
- Produces a configurable volume of entries per poll (e.g., 50-500)
- Mixes compliant and non-compliant queries (some consumers query datasets they don't have purpose authorization for)
- Watermark-aware: each poll returns new entries, not duplicates
- No external dependencies (no credentials needed)
- Registered in the connector registry so it can be used via the standard query log config flow

## Current State

**Connector interface** (`connectors/interface.py`):
- `QueryLogConnector` ABC with three methods: `connection_type`, `fetch_entries(since, limit)`, `test_connection()`
- Returns `tuple[list[RawQueryLogEntry], dict]` from `fetch_entries`

**Registry** (`connectors/__init__.py`):
- `QUERY_LOG_CONNECTORS_BY_TYPE` maps `ConnectionType` to connector class
- Currently only `ConnectionType.bigquery` is registered

**Existing test connection types** (`fides/models/connectionconfig.py`):
- `ConnectionType.test_datastore` and `ConnectionType.test_website` already exist for test monitors
- We can reuse `test_datastore` for the mock query log connector (or add a new one if needed)

## Implementation Approach

### 1. Create `MockQueryLogConnector` (`connectors/mock_connector.py`)

A new connector that implements `QueryLogConnector`:

```python
class MockQueryLogConnector(QueryLogConnector):
    """Generates synthetic query log entries from existing datasets and consumers."""

    connection_type = ConnectionType.test_datastore

    def fetch_entries(self, since, limit=1000):
        # 1. Load all DataConsumers from Redis
        # 2. Load all datasets from DB (or use a static pool)
        # 3. Generate `limit` random RawQueryLogEntry objects:
        #    - Pick a random consumer → use their contact_email
        #    - Pick 1-3 random datasets → build TableRef list
        #    - Generate realistic SELECT queries
        #    - Timestamp = now minus random offset (monotonically increasing)
        # 4. Return entries + new watermark (counter-based)

    def test_connection(self):
        return True  # always succeeds
```

Key generation logic:
- **Consumer selection**: Weighted random from all `DataConsumerEntity` objects in Redis. Includes some queries from "unknown" emails (not mapped to any consumer) to simulate unresolved identities.
- **Dataset/table selection**: Pull dataset fides_keys from the DB or from a seeded list. Build `TableRef` objects with realistic project/dataset/table names.
- **Query text**: Generate plausible `SELECT` statements referencing the chosen tables (e.g., `SELECT col1, col2 FROM project.dataset.table WHERE ...`).
- **Compliance mix**: ~70% of generated queries should be from consumers querying datasets they have purpose authorization for, ~30% should be violations (consumer lacks required purposes for the dataset).
- **Volume**: Configurable via `limit` parameter. Default poll returns ~100 entries.

### 2. Register in connector registry (`connectors/__init__.py`)

```python
QUERY_LOG_CONNECTORS_BY_TYPE: Dict[ConnectionType, Type[QueryLogConnector]] = {
    ConnectionType.bigquery: BigQueryQueryLogConnector,
    ConnectionType.test_datastore: MockQueryLogConnector,
}
```

### 3. Watermark strategy

Use a simple counter-based watermark since there's no external system to track:

```python
watermark = {"mock_counter": <int>}
```

Each poll increments the counter. The connector generates entries with timestamps based on `now - (limit - i) seconds` so they appear chronologically ordered.

### 4. Data sourcing

The mock connector needs access to existing consumers and datasets to generate realistic data:

- **Consumers**: Read from `DataConsumerRedisRepository` (injected via constructor or loaded from cache)
- **Datasets**: Query `DatasetConfig` from the DB session, or fall back to a hardcoded pool of realistic table names if no datasets exist

If no consumers or datasets exist yet, fall back to a built-in pool of synthetic names so the connector always produces output.

### 5. No secrets required

Since `ConnectionType.test_datastore` exists and the mock needs no credentials, `test_connection()` always returns `True` and `fetch_entries()` needs no authentication. The `ConnectionConfig.secrets` can be empty or contain optional config like `{"entries_per_poll": 100, "violation_rate": 0.3}`.

## Story Breakdown

| # | Story | Repo | Type | Estimate | Dependencies |
|---|-------|------|------|----------|--------------|
| 1 | Create `MockQueryLogConnector` implementing the `QueryLogConnector` interface | Fidesplus | Story | M | - |
| 2 | Register mock connector in `QUERY_LOG_CONNECTORS_BY_TYPE` for `test_datastore` | Fidesplus | Task | S | #1 |
| 3 | Unit tests for mock connector (deterministic output with seeded random) | Fidesplus | Task | S | #1 |
| 4 | Update demo script to support a `--mock` flag that uses the mock connector | Fidesplus | Task | S | #1, #2 |

## Testing Strategy

- Unit tests: seed the random generator, verify entry count matches limit, verify entries reference real consumer emails and dataset names when available, verify watermark advances
- Integration test: create a query log config with `test_datastore` connection type, trigger poll, verify violations are recorded
- Verify `test_connection()` always returns True

## Out of Scope

- Adding a new `ConnectionType` enum value (reuse `test_datastore`)
- Frontend integration (no UI for this yet)
- Configurable query complexity (simple SELECTs are sufficient)
- Snowflake or other platform mock connectors (BigQuery mock pattern can be extended later)
