# Runtime Stabilization Summary (Closed Baseline: 2026-02-11)

## Objective
Stabilize Cloud Run orchestrator reliability for `/admin/resume-history` scheduler endpoint and verify keyless Domain-Wide Delegation (DWD) configuration.

## Root Cause vs Hardening (clarification)

- **Root cause fix**: Missing `DEFAULT_MACHINE_ID` in Cloud Run env. Setting it (e.g. `clinic-default`) is what made the scheduler return 200. This was deployed and verified.
- **Code hardening**: Handler changes (stable log line, exception wrapper, 403 semantics, log-once guard) improve diagnostics and safe failure mode; these are now deployed in production.

## Root Cause Analysis

**Issue**: Cloud Scheduler reported `URL_UNREACHABLE-UNREACHABLE_5xx` (HTTP 500) when calling `/admin/resume-history`.

**Traceback** (from Cloud Run logs at 2026-02-10 00:00:31):
```
File "/app/api.py", line 422, in resume_history
    result = processor.process_history_event({"historyId": history_id}, get_config())
File "/app/processor.py", line 282, in process_history_event
    _handle_message(message, gmail_client, storage_client, fs_client, config)
File "/app/processor.py", line 321, in _handle_message
    machine_id = _extract_machine_id(message, config, fs_client)
File "/app/processor.py", line 102, in _extract_machine_id
    raise RuntimeError("Unable to determine machine id for message 19c4453dbc96c88e")
RuntimeError: Unable to determine machine id for message 19c4453dbc96c88e
```

**Root Cause**: Missing `DEFAULT_MACHINE_ID` environment variable. For single-machine deployments, `_extract_machine_id` requires this env var to avoid Firestore routing lookups.

## Changes Implemented

### 1. Environment Fix (Deployed)
- **Action**: Set `DEFAULT_MACHINE_ID=clinic-default` on Cloud Run
- **Result**: Scheduler now returns **200** for `/admin/resume-history` (verified at 2026-02-10 08:35:34 UTC)
- **Verification**: 
  ```bash
  gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="medilink-gmail-orchestrator" AND httpRequest.requestUrl:"/admin/resume-history"' --project genial-analyzer-476721-f8 --freshness=2h --limit 3 --format="value(timestamp,httpRequest.status)"
  ```
  Output shows: `2026-02-10T08:35:34.951676Z	200`

### 2. Code Improvements (Deployed)

#### api.py
- **Stable log line**: Added `LOGGER.info("resume-history invoked")` at handler start for unambiguous verification
- **Admin secret validation**: Check if `admin_shared_secret` is configured; return 500 if not set. Log "ADMIN_SHARED_SECRET not set" once per process via module-level guard to avoid log spam.
- **403 semantics**: Use 403 for missing/empty or wrong secret (app-level guard, not WWW-Authenticate)
- **Exception handling**: Wrap `processor.process_history_event` in try/except; log full traceback with `exc_info=True`; return generic 500 response body (no internal details)

#### validate_and_complete_setup.py
- **Keyless DWD prereqs check**: Added `_check_keyless_dwd_prereqs()` function
  - Verifies IAM Credentials API enabled
  - Verifies SA has Token Creator on self
  - Labels as "Keyless DWD prereqs present" (not "verified working")
  - Notes that runtime log `Gmail auth mode: dwd (keyless)` is the real verification
- **ADMIN_SHARED_SECRET check**: Validates secret is set when scheduler job exists (prevents deployment drift)

### 3. Testing

#### Regression Tests Created
- **Location**: `cloud/orchestrator/tests/test_admin_resume_history.py`
- **Framework**: stdlib `unittest` + FastAPI `TestClient`
- **Coverage**:
  - Valid request with secret and history_id -> 200
  - Missing secret -> 403
  - Wrong secret -> 403
  - Secret not configured -> 500
  - Exception in process_history_event -> 500 with generic body (hardening / safer failure mode)
  - Accepts both `history_id` and `historyId` (schema compatibility)

#### How to Run
```powershell
# From cloud/orchestrator directory
py -3.11 -m unittest discover -s tests

# Specific test
py -3.11 -m unittest tests.test_admin_resume_history
```

Note: Requires `httpx` for TestClient (`pip install httpx`)

### 4. Documentation Updates

- **Deployment Status** ([docs/MEDILINK_CLOUD_MIGRATION.md](docs/MEDILINK_CLOUD_MIGRATION.md)):
  - Moved resolved issues to "Issues resolved" section
  - Updated Cloud Build workarounds
  - Verified scheduler returns 200

- **Migration Guide** ([docs/MEDILINK_CLOUD_MIGRATION.md](docs/MEDILINK_CLOUD_MIGRATION.md)):
  - Documented fix for `/admin/resume-history` 500 errors
  - Updated known issues section
  - Updated to reflect keyless DWD verification and `dwd_required` steady state

- **Operations Runbook** ([docs/runbooks/MediLink_Gmail_Orchestrator_Operations.md](docs/runbooks/MediLink_Gmail_Orchestrator_Operations.md)):
  - Added "How to verify" section with 4-step verification process
  - Commands for checking scheduler status, handler execution, and Gmail auth mode

- **Orchestrator README** ([cloud/orchestrator/README.md](cloud/orchestrator/README.md)):
  - Added "Running Unit Tests" section

## Verification Results

### Completed
1. **Validator checks**: All checks pass (20/20 OK)
   ```bash
   py -3.11 cloud/orchestrator/validate_and_complete_setup.py --validate-only
   ```
   Output: "All checks passed and setup actions are complete."

2. **Scheduler returns 200**: Verified at 2026-02-10 08:35:34 UTC
   - Before fix: HTTP 500 with RuntimeError (root cause: missing DEFAULT_MACHINE_ID)
   - After fix: HTTP 200 with successful processing (env var fix deployed)

3. **Keyless DWD verified**:
   - IAM Credentials API enabled: yes
   - SA has Token Creator on self: yes
   - Runtime proof observed in production logs: `Gmail auth mode: dwd (keyless)`

4. **Tests created**: Regression test suite covers request contract and hardening (generic 500 body, 403 semantics)

### Platform caveat (still open)

Cloud Build can intermittently return `NOT_FOUND` for `gcloud builds submit` in this project. The validator includes preflight remediation and fallback guidance; application runtime stabilization is otherwise complete.

## Definition of Done Status

From the plan:
1. **Scheduler executions show 2xx** for `/admin/resume-history` - DONE (env fix deployed)
2. **Cloud Run logs show handler ran** (verified via 200 responses) - DONE
3. **Log line shows `Gmail auth mode: dwd (keyless)`** - DONE
4. **Docs updated** after verification - DONE

## Next Steps

1. Validate notification channels and run an alert smoke drill in a change window.
2. Decide break-glass posture for user OAuth secrets (retain vs remove) and document it.
3. Continue preprocessor design for queue normalization before Firestore writes.

## Files Modified

### Code
- `cloud/orchestrator/api.py` - Handler improvements (ready for deployment)
- `cloud/orchestrator/validate_and_complete_setup.py` - Added checks
- `cloud/orchestrator/tests/test_admin_resume_history.py` - New test file
- `cloud/orchestrator/tests/__init__.py` - New test package

### Documentation
- `docs/MEDILINK_CLOUD_MIGRATION.md`
- `docs/MEDILINK_CLOUD_MIGRATION.md`
- `docs/runbooks/MediLink_Gmail_Orchestrator_Operations.md`
- `cloud/orchestrator/README.md`

### Environment
- Cloud Run env var: `DEFAULT_MACHINE_ID=clinic-default` (deployed)

## Commands Reference

```bash
# Validation
py -3.11 cloud/orchestrator/validate_and_complete_setup.py --validate-only

# Check scheduler status
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="medilink-gmail-orchestrator" AND httpRequest.requestUrl:"/admin/resume-history"' --project genial-analyzer-476721-f8 --freshness=6h --limit 3 --format="value(timestamp,httpRequest.status)"

# Run tests
py -3.11 -m unittest discover -s cloud/orchestrator/tests

# Optional deploy path (if Cloud Build submit is unavailable)
gcloud run deploy --source cloud/orchestrator --project genial-analyzer-476721-f8 --region us-central1
```

