# Questionnaire System Architecture

End-to-end architecture for privacy assessment questionnaires: from assessment templates through LLM-powered chat to versioned answer persistence.

## Data Model

```
AssessmentTemplate (versioned DPIA/PIA definitions)
 └── AssessmentQuestion (grouped by requirement_key)
      │
      ▼
PrivacyAssessment (concrete assessment for a system)
 ├── AssessmentAnswer (current answer per question)
 │    └── AnswerVersion (immutable audit trail)
 └── Questionnaire (chat session)
      └── ChatMessage (conversation history)
```

**Models live in fides OSS** (`fides/api/models/privacy_assessment.py`, `questionnaire.py`). Fidesplus owns the service layer that operates on them.

## Assessment Templates

Templates define the structure of a privacy assessment. Each template contains questions organized into groups.

```
AssessmentTemplate
├── assessment_type    e.g., "us_ca_cpra_risk_assessment"
├── version            regulatory version, e.g., "US-CA-CPPA-RA-2025-09-23"
├── fides_revision     Ethyca's iteration of that version (1, 2, ...)
├── is_active          only one active template per assessment_type
└── questions[]
     ├── requirement_key     group identifier, e.g., "processing_scope"
     ├── requirement_title   display label, e.g., "Processing Scope and Purpose(s)"
     ├── question_text       the actual question
     ├── guidance            reviewer guidance (optional)
     ├── fides_sources       system fields that can auto-answer this question
     ├── expected_coverage   what fides_sources should provide
     ├── group_order         ordering of groups
     └── question_order      ordering within group
```

When an assessment is created, it is linked to the active template for its type. The template's questions define what the questionnaire will ask.

## Answer Versioning

Answers use a two-table design for auditability:

```
AssessmentAnswer                    AnswerVersion
├── assessment_id                   ├── answer_id (FK)
├── question_id                     ├── version_number (1, 2, 3...)
├── current_version_id (FK) ──────► ├── answer_text
│                                   ├── answer_status: complete | partial | needs_input
│                                   ├── answer_source: system | ai_analysis | user_input | team_input
│                                   ├── confidence (float, optional)
│                                   ├── change_type: ai_generated | human_edited | approved | rejected
│                                   ├── created_by (email of who made this change)
│                                   ├── evidence (JSONB)
│                                   ├── source_references (JSONB)
│                                   └── diff_from_previous (text)
```

**Answer sources:**

- `system` -- auto-populated from fides system metadata (data categories, uses, etc.)
- `ai_analysis` -- generated by the LLM during assessment creation
- `user_input` -- manually edited in the Admin UI
- `team_input` -- collected via questionnaire chat (Slack or in-app)

**Questionnaire answers** are persisted as `team_input` with `source_references` containing `questionnaire_id`, `message_id`, and `question_index` for traceability back to the exact chat message.

## Questionnaire Lifecycle

```
                    ┌─────────────┐
         start      │ in_progress │
        ────────►   │             │
                    └──────┬──────┘
                           │
                 ┌─────────┼─────────┐
                 ▼                   ▼
          ┌────────────┐     ┌────────────┐
          │ completed  │     │  stopped   │
          │            │     │            │
          │ all Q's    │     │ user chose │
          │ answered   │     │ to end     │
          └────────────┘     └────────────┘
                                   │
                                   ▼
                             can start a
                             new session
```

An assessment can have multiple questionnaire sessions (retries after stopping). The `PrivacyAssessment.questionnaire` relationship returns the latest active session; the service queries by `assessment_id` ordered by `created_at desc`.

## Chat Provider Interface

The questionnaire communicates through a provider-agnostic interface (`service/chat/provider/interface.py`). Providers handle message delivery; the questionnaire service handles conversation logic.

```
QuestionnaireService
       │
       ▼
ChatProviderInterface (ABC)
       │
       ├── SlackChatProvider        real Slack API (OAuth2, threads, webhooks)
       └── FidesChatProvider        in-memory store for in-app browser chat
```

**Interface methods:**


| Method                     | Purpose                                         |
| -------------------------- | ----------------------------------------------- |
| `send_message`             | Post to a channel (with optional color sidebar) |
| `send_thread_reply`        | Reply within a thread                           |
| `update_message`           | Edit an existing message                        |
| `delete_message`           | Remove a message                                |
| `get_channels`             | List available channels                         |
| `get_user_info`            | Resolve user ID to email + display name         |
| `set_typing_indicator`     | Show "thinking..." status                       |
| `clear_typing_indicator`   | Clear status                                    |
| `verify_webhook_signature` | Validate incoming webhooks                      |
| `join_channel`             | Join a channel (if required)                    |


**Slack** uses real HTTP calls to the Slack Web API, OAuth2 tokens, thread timestamps, and HMAC-SHA256 webhook verification.

**Fides chat provider** stores messages in memory and returns them via the questionnaire chat API. No external service needed -- the browser polls for new messages.

Provider instantiation is handled by `factory.py` based on the `ChatConfig.provider_type` field.

## Two-Pass LLM Architecture

User messages are processed in two separate LLM calls, each with a focused prompt:

```
User message
     │
     ▼
┌──────────────────────────────────────┐
│  Pass 1: Intent Classification       │
│                                      │
│  Input:  user message                │
│          conversation history        │
│          current question + context  │
│          fides privacy context       │
│                                      │
│  Output: List[ToolAction]            │
│          (structured tool calls)     │
└──────────────────┬───────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│  Tool Execution (handlers)           │
│                                      │
│  For each ToolAction:                │
│    update_answers → persist answer   │
│    navigate → move question index    │
│    respond → no-op                   │
│    silence → no-op                   │
│    stop → end questionnaire          │
│                                      │
│  Returns: HandlerResult              │
│    (new position, next question)     │
└──────────────────┬───────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│  Answer Normalization (optional)     │
│                                      │
│  LLM cleans casual speech into       │
│  assessment-quality text:            │
│    "yeah we keep it for like 2yrs"   │
│    → "Data is retained for 2 years"  │
│                                      │
│  Also: rejects irrelevant answers,   │
│  translates non-English, filters     │
│  sarcasm/delegation                  │
└──────────────────┬───────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│  Pass 2: Response Generation         │
│                                      │
│  Input:  action summary from Pass 1  │
│          conversation history        │
│          next question text          │
│          handler results             │
│                                      │
│  Output: natural language response   │
│          (acknowledgment + next Q)   │
└──────────────────────────────────────┘
```

**Why two passes?** A single prompt must simultaneously choose tools, extract data, AND generate a natural response. This leads to prompt overloading: wrong tool selection, parroting user words, leaking next-question text into acknowledgments, and style violations. Separating the cognitive tasks lets each prompt focus on one job.

## Composable Tools

Pass 1 returns a `List[ToolAction]`, supporting compound actions in a single user message (e.g., "answer this and skip the next one").


| Tool             | Purpose                                              | State Change                                 |
| ---------------- | ---------------------------------------------------- | -------------------------------------------- |
| `update_answers` | Write or correct answers to one or more questions    | Persists AnswerVersion, may advance position |
| `navigate`       | Move position: skip, go back, jump to index, restart | Updates current_question_index               |
| `respond`        | Send a conversational message (clarification, help)  | None                                         |
| `silence`        | Stay silent when users are talking to each other     | None                                         |
| `stop`           | End questionnaire early with optional reason         | Marks questionnaire stopped                  |


Handlers in `handlers.py` execute each tool action and return a `HandlerResult` containing position changes and the next question text for Pass 2.

## End-to-End Request Flow

A complete cycle from user message to persisted answer:

```
1.  User sends message (Slack thread reply or in-app chat input)
         │
2.  Route receives request
    ├── Slack: POST /slack/events → webhook handler
    └── In-app: POST /chat/questionnaire/reply → questionnaire route
         │
3.  QuestionnaireService.process_reply()
    ├── Acquires DB lock (SELECT FOR UPDATE on questionnaire row)
    ├── Builds conversation history from ChatMessage table (last 10)
    ├── Loads current question + fides privacy context
         │
4.  Pass 1: classify_intent()
    ├── LLM extracts structured List[ToolAction]
    └── Returns tool name, parameters, confidence per action
         │
5.  Tool execution (handlers)
    ├── update_answers → persist_answer_version()
    │   ├── Creates AnswerVersion (source=team_input, created_by=email)
    │   ├── Updates AssessmentAnswer.current_version_id
    │   └── Recalculates assessment completeness %
    ├── navigate → updates current_question_index
    └── stop → marks questionnaire stopped with reason
         │
6.  Answer normalization (if update_answers was called)
    └── LLM cleans raw text → assessment-quality prose
         │
7.  Pass 2: generate_response()
    └── LLM produces natural acknowledgment + next question
         │
8.  Response delivery
    ├── Slack: send_thread_reply() via Slack API
    └── In-app: store ChatMessage, returned on next poll
         │
9.  ChatMessage persisted with sender_email, question_index, timestamp
```

## Multi-Session Continuity

When a stopped questionnaire is restarted:

- New `Questionnaire` row created (old one stays as history)
- Continuation intro references prior progress by question group
- Question summary includes answers from ALL prior sessions so LLM has full context
- Only `team_input` answers from prior sessions are considered (ignores system/AI answers)

## User Resolution

Users are identified differently across providers. `user_resolution.py` normalizes to a consistent identity:

**Trust hierarchy** (first match wins):

1. FidesUser full name (admin-managed, authoritative)
2. FidesUser first name
3. FidesUser username
4. Root user config match
5. Identifier as-is

Display names from Slack are never trusted (users can set anything). Email is the canonical identifier; display names are resolved at render time.

## Key Files

### Questionnaire Service (`service/questionnaire/`)


| File                       | Responsibility                                                       |
| -------------------------- | -------------------------------------------------------------------- |
| `service.py`               | Orchestration: lifecycle, reply processing, question navigation      |
| `handlers.py`              | Tool execution: persists state changes from each ToolAction          |
| `intent_classification.py` | Pass 1: LLM intent extraction and ToolAction parsing                 |
| `tools.py`                 | Tool definitions, schemas, and action dataclasses                    |
| `prompts.py`               | All LLM prompt templates (classification, generation, normalization) |
| `constants.py`             | Shared constants (colors, limits)                                    |
| `repository.py`            | Data access layer for questionnaire entities                         |
| `__init__.py`              | Public API surface and module wiring                                 |


### Chat Providers (`service/chat/provider/`)


| File                     | Responsibility                                                     |
| ------------------------ | ------------------------------------------------------------------ |
| `interface.py`           | `ChatProviderInterface` ABC                                        |
| `models.py`              | Provider-agnostic data models (ChatMessage, ChatUser, ChatChannel) |
| `slack_provider.py`      | Slack Web API implementation                                       |
| `fides_chat_provider.py` | In-memory implementation for browser-based chat                    |
| `factory.py`             | Provider instantiation by type                                     |


### Privacy Assessment (`service/privacy_assessment/`)


| File                 | Responsibility                                           |
| -------------------- | -------------------------------------------------------- |
| `service.py`         | Assessment CRUD, questionnaire status in detail response |
| `answer_utils.py`    | `persist_answer_version()` -- creates versioned answers  |
| `user_resolution.py` | Normalize user identity across providers                 |
| `repository.py`      | Data access for assessments and answers                  |
| `entities.py`        | Dataclass entities (decoupled from ORM)                  |
| `tasks.py`           | Celery task for async assessment generation              |


### API Routes


| File                                | Responsibility                                |
| ----------------------------------- | --------------------------------------------- |
| `api/routes/privacy_assessments.py` | Assessment CRUD + questionnaire send endpoint |
| `api/routes/chat/questionnaire.py`  | In-app chat: start, reply, get messages       |
| `api/schemas/questionnaire_chat.py` | Request/response schemas for chat API         |


### Models (fides OSS)


| Model                | Table                 | Purpose                           |
| -------------------- | --------------------- | --------------------------------- |
| `AssessmentTemplate` | `assessment_template` | Versioned DPIA/PIA definitions    |
| `AssessmentQuestion` | `assessment_question` | Questions grouped by requirement  |
| `PrivacyAssessment`  | `privacy_assessment`  | Concrete assessment instance      |
| `AssessmentAnswer`   | `assessment_answer`   | Current answer state per question |
| `AnswerVersion`      | `answer_version`      | Immutable version history         |
| `Questionnaire`      | `questionnaire`       | Chat session linked to assessment |
| `ChatMessage`        | `chat_message`        | Per-message conversation record   |


