# Using the `ConfigurableTestMonitor`

The Configurable Test Monitor generates predictable datastore discovery data without requiring external database connections. It simulates the discovery of databases, schemas, tables, and fields with configurable hierarchical structures and nested field support. This monitor mimics the structure returned by BigQueryMonitor and is particularly useful for testing the discovery monitor framework and generating predictable test data for performance testing in local environments.

It supports a number of configuration parameters, including how many databases, schemas, tables, and fields to generate, as well as nested field configuration.

The ConfigurableTestMonitor is only available when development mode is enabled (`FIDES__DEV_MODE=true`)

## Complete Configuration Example

This is a complete configuration example, all the parameters are explained in the Configuration Options section.

```json
{
  "datasource_params": {
    "num_databases": 3,
    "num_schemas_per_db": 2,
    "num_tables_per_schema": 4,
    "num_fields_per_table": 15,
    "field_sample_data": ["sample_value_1", "sample_value_2", "sample_value_3"],
    "max_nesting_level": 3,
    "nested_field_percentage": 0.3,
    "num_fields_per_nested": 3,
    "sharded_table_pattern": null
  }
}
```

## Setup Steps 

Because this is a testing utility, it cannot be configured through our usual integrations menu in the Admin UI. Instead, you have to rely on API calls to set up the monitor. Once you've successfully executed the `ConfigurableTestMonitor`, then you will be able to see the results on the Admin UI Action Center as usual.

### Step 1: Create Test Datastore Connection

**Endpoint:** `PATCH /api/v1/connection`

**Request Body:**
```json
[
  {
    "name": "Test Datastore Connection",
    "key": "test_datastore_connection",
    "connection_type": "test_datastore",
    "access": "write",
    "disabled": false,
    "description": "Connection for configurable test datastore monitor",
    "secrets": {}
  }
]
```

Note that no secrets are required for the test connection since it doesn't connect to any real datastore.

### Step 2: Create Discovery Monitor

**Endpoint:** `PUT /api/v1/plus/discovery-monitor`

#### Option A: Use Default Parameters

To use all default configuration values, omit the datasource_params field and use the request body below (the connection_config_key is the key returned in the response of the request from Step 1):

```json
{
  "name": "Default Test Datastore Monitor",
  "connection_config_key": "test_datastore_connection",
  "enabled": true
}
```

#### Option B: Override Specific Parameters

To customize the monitor behavior, add a datasource_params object with the parameters you want to override. You only need to specify the parameters you want to change - any omitted parameters will use their default values. The configuration options section lists all available configuration parameters.

**Example - Override Database Structure:**
```json
{
  "name": "Custom Structure Monitor",
  "connection_config_key": "test_datastore_connection",
  "enabled": true,
  "datasource_params": {
    "num_databases": 5,
    "num_schemas_per_db": 3,
    "num_tables_per_schema": 8
  }
}
```

**Example - Override Field Configuration:**
```json
{
  "name": "Complex Fields Monitor",
  "connection_config_key": "test_datastore_connection",
  "enabled": true,
  "datasource_params": {
    "num_fields_per_table": 25,
    "nested_field_percentage": 0.4,
    "max_nesting_level": 4,
    "num_fields_per_nested": 4
  }
}
```

### Step 3: Execute Monitor

**Endpoint:** `POST /api/v1/plus/discovery-monitor/{monitor_id}/execute`

You'll need to use the monitor key from the response on Step 2.

### Step 4: View results

You can now go the Admin UI Action center to view the results.

## Configuration Options

The configurable test datastore monitor supports extensive customization through the datasource_params field. All parameters are optional and will use defaults if not specified.

### Database Structure Configuration

Controls the hierarchical structure of the generated datastore:

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `num_databases` | integer | 2 | Number of databases to generate |
| `num_schemas_per_db` | integer | 2 | Number of schemas to generate per database |
| `num_tables_per_schema` | integer | 2 | Number of tables to generate per schema |
| `num_fields_per_table` | integer | 10 | Total number of fields to generate per table |

**Example:**
```json
{
  "num_databases": 4,
  "num_schemas_per_db": 3,
  "num_tables_per_schema": 6,
  "num_fields_per_table": 20
}
```

### Field Configuration

Controls field generation and sample data:

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `field_sample_data` | array | `["sample1", "sample2", "sample3"]` | Sample data values returned for field sampling |

**Example:**
```json
{
  "field_sample_data": [
    "test_value_alpha",
    "test_value_beta", 
    "test_value_gamma",
    "test_value_delta"
  ]
}
```

### Nested Fields Configuration

Controls the generation of nested fields (mimics BigQuery STRUCT/RECORD types):

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `nested_field_percentage` | float | 0.2 | Percentage of fields that should be nested objects (0.0 to 1.0) |
| `max_nesting_level` | integer | 2 | Maximum depth of field nesting |
| `num_fields_per_nested` | integer | 2 | Number of sub-fields to generate for each nested field |

**Example:**
```json
{
  "nested_field_percentage": 0.35,
  "max_nesting_level": 4,
  "num_fields_per_nested": 5
}
```

**Understanding Nested Fields:**

- **`nested_field_percentage`**: Determines what fraction of the total fields per table will be nested objects vs flat fields
- **`max_nesting_level`**: Controls how deeply nested the structure can go (nested fields can contain other nested fields)
- **`num_fields_per_nested`**: Each nested field will contain this many sub-fields

For example, with `num_fields_per_table: 10` and `nested_field_percentage: 0.3`:
- 7 flat fields: `field_1`, `field_2`, ..., `field_7`
- 3 nested fields: `nested_field_8`, `nested_field_9`, `nested_field_10`
- Each nested field contains `num_fields_per_nested` sub-fields

### Table Pattern Configuration

Inherits from the base datastore parameters:

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `sharded_table_pattern` | string | null | Regex pattern to identify sharded tables (inherited from DatasourceParamsBase) |

**Example:**
```json
{
  "sharded_table_pattern": "^(.+?)_\\d{8}$"
}
```

## Generated Data Structure

The monitor generates a predictable hierarchy:

### Naming Convention
- **Databases**: `db1`, `db2`, `db3`, ...
- **Schemas**: `schema_1`, `schema_2`, `schema_3`, ...
- **Tables**: `table_1`, `table_2`, `table_3`, ...
- **Fields**: 
  - Flat fields: `field_1`, `field_2`, `field_3`, ...
  - Nested fields: `nested_field_X`, `nested_field_Y`, ...
  - Sub-fields: `nested_field_X_nested_1`, `nested_field_X_nested_2`, ...

### URN Structure
All resources follow a hierarchical URN pattern:
- **Database**: `{monitor_key}.db1`
- **Schema**: `{monitor_key}.db1.schema_1`
- **Table**: `{monitor_key}.db1.schema_1.table_1`
- **Field**: `{monitor_key}.db1.schema_1.table_1.field_1`

### Example Generated Structure
With default parameters (2 databases, 2 schemas per DB, 2 tables per schema, 10 fields per table):

```
db1/
├── schema_1/
│   ├── table_1/ (8 flat fields + 2 nested fields)
│   │   ├── field_1 (string)
│   │   ├── field_2 (string)
│   │   ├── ...
│   │   ├── field_8 (string)
│   │   ├── nested_field_9 (object)
│   │   │   ├── nested_field_9_nested_1 (string)
│   │   │   └── nested_field_9_nested_2 (string)
│   │   └── nested_field_10 (object)
│   │       ├── nested_field_10_nested_1 (string)
│   │       └── nested_field_10_nested_2 (string)
│   └── table_2/ (same structure)
└── schema_2/
    ├── table_1/
    └── table_2/
db2/ (same structure as db1)
```

## Use Cases

### Performance Testing
Generate large datasets for testing discovery monitor performance:

```json
{
  "datasource_params": {
    "num_databases": 10,
    "num_schemas_per_db": 5,
    "num_tables_per_schema": 20,
    "num_fields_per_table": 50,
    "nested_field_percentage": 0.25,
    "max_nesting_level": 3
  }
}
```

### Testing Nested Field Support
Test complex nested structures similar to BigQuery:

```json
{
  "datasource_params": {
    "num_databases": 2,
    "num_schemas_per_db": 2,
    "num_tables_per_schema": 3,
    "num_fields_per_table": 15,
    "nested_field_percentage": 0.6,
    "max_nesting_level": 5,
    "num_fields_per_nested": 4
  }
}
```

### Minimal Test Data
Generate minimal data for unit tests:

```json
{
  "datasource_params": {
    "num_databases": 1,
    "num_schemas_per_db": 1,
    "num_tables_per_schema": 1,
    "num_fields_per_table": 5,
    "nested_field_percentage": 0.0
  }
}
```