# MediLink Gmail Orchestrator - Complete Setup Guide

**Comprehensive guide for setting up the MediLink Gmail ingestion pipeline from scratch**

## 📋 Table of Contents

1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Quick Start](#quick-start)
4. [Detailed Setup Process](#detailed-setup-process)
5. [Testing and Verification](#testing-and-verification)
6. [Troubleshooting](#troubleshooting)
7. [Advanced Configuration](#advanced-configuration)

---

## 🏗️ Overview

### Architecture

```
mariavidaud@gmail.com (Personal Gmail)
    ↓ (Gmail Forwarding Rule)
daniel@strategicimplementation.us (Workspace mailbox)
    ↓ (User OAuth token in Secret Manager)
Cloud Run Service → Gmail API → Pub/Sub → XP Daemon
```

### Key Components

- **Cloud Run Service**: Processes Gmail notifications and manages email attachments
- **Pub/Sub**: Message queue for Gmail notifications
- **Cloud Storage**: Attachment storage
- **Firestore**: Sync state and queue management
- **Gmail API**: Email access via Workload Identity Federation

### Authentication Method

**Workspace mailbox + User OAuth refresh token**:
- ✅ Works with the existing forwarding rule to `daniel@strategicimplementation.us`
- ✅ No service account key files required
- ✅ OAuth credentials are stored in Secret Manager
- ✅ Cloud Run retrieves credentials at runtime

---

## 📋 Prerequisites

Before starting, ensure you have:

- **Google Cloud Project**: e.g., `genial-analyzer-476721-f8`
- **Google Cloud SDK**: Installed and authenticated
- **Python 3.11+**: For local testing
- **PowerShell**: For running setup scripts (Windows) or bash (Linux)
- **Permissions**: Owner or Editor role on the GCP project
- **Billing**: Enabled on the GCP project

### Install Google Cloud SDK

If you don't have `gcloud` installed:

**Windows**:
1. Download from: https://cloud.google.com/sdk/docs/install
2. Run the installer
3. Open a NEW PowerShell window after installation

**Linux/Mac**:
```bash
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
```

### Authenticate

```bash
# Login to your Google account
gcloud auth login

# Set your default project
gcloud config set project YOUR_PROJECT_ID

# Verify access
gcloud projects describe YOUR_PROJECT_ID
```

---

## 🚀 Quick Start

### For Experienced Users

```powershell
# 1. Run comprehensive setup (creates all GCP resources)
cd cloud\orchestrator
py -3.11 .\validate_and_complete_setup.py --project-id YOUR_PROJECT_ID --mailbox-user daniel@strategicimplementation.us

# 2. Validate + complete setup from local IDE (cross-platform)
python validate_and_complete_setup.py --project-id YOUR_PROJECT_ID --mailbox-user daniel@strategicimplementation.us 

# 3. Confirm forwarding still targets the workspace mailbox
# Forward from your source Gmail to daniel@strategicimplementation.us

# 4. Test the pipeline
# Send test email and monitor Cloud Run logs
```

---

## 🔧 Detailed Setup Process

### Step 1: Run Infrastructure Setup Script

```powershell
cd cloud\orchestrator
py -3.11 .\validate_and_complete_setup.py --project-id YOUR_PROJECT_ID
```

**What this script does:**
- ✅ Enables required GCP APIs (Pub/Sub, Cloud Run, Firestore, etc.)
- ✅ Creates service account with proper IAM roles
- ✅ Creates Cloud Storage bucket for attachments
- ✅ Builds and deploys Cloud Run service
- ✅ Creates Pub/Sub topic and subscription
- ✅ Provides next steps for manual configuration

**Expected Duration**: 5-10 minutes

### Step 2: Confirm Workspace Mailbox

This deployment uses the workspace mailbox `daniel@strategicimplementation.us` as the monitored inbox.

1. Verify forwarding rules still send target mail into `daniel@strategicimplementation.us`
2. Confirm the mailbox receives expected source traffic
3. Keep Gmail API enabled in the same GCP project used by Cloud Run

### Step 3: Set Up Email Forwarding

Forward emails from your source Gmail to `daniel@strategicimplementation.us`.

**Automatic Setup** (Recommended):
- Follow the interactive prompts in `GMAIL_FORWARDING_SETUP.md`

**Manual Setup**:
1. Go to [Gmail Settings](https://mail.google.com/mail/u/0/#settings/fwdandpop)
2. In the "Forwarding" section, click "Add a forwarding address"
3. Enter: `daniel@strategicimplementation.us`
4. Gmail will send a verification email to the workspace mailbox
5. Open `daniel@strategicimplementation.us` and click the verification link
6. Back in the original Gmail, enable forwarding:
   - Select the forwarding address
   - Choose "Forward a copy of incoming mail"
   - Optionally: "Keep Gmail's copy in the Inbox" or "Delete Gmail's copy"

**Optional**: Set up filters to forward only specific emails (e.g., from specific senders or with certain subjects)

### Step 4: Initialize Firestore Database

**Manual Step** (Cannot be automated):

1. Go to [Firestore Console](https://console.cloud.google.com/firestore/databases?project=YOUR_PROJECT_ID)
2. Click "Create Database"
3. Select "Native mode" (not Datastore mode)
4. Choose location: `us-central1` (recommended)
5. Click "Create"

**Create Collections**:
The following collections will be created automatically on first use:
- `queues` - Email processing queue
- `sync_state` - Gmail synchronization state

**Create Composite Index** (Optional but recommended for performance):
1. In Firestore Console → Indexes
2. Click "Create Index"
3. Collection ID: `queues`
4. Fields to index:
   - `machine_id` (Ascending)
   - `acked` (Ascending)
   - `created_at` (Ascending)
5. Click "Create"

### Step 5: Register Gmail Watch

```powershell
cd cloud\orchestrator
py -3.11 .\validate_and_complete_setup.py --verify-runtime
```

**What this does:**
- ✅ Authenticates with user OAuth credentials stored in Secret Manager
- ✅ Registers Gmail watch on `daniel@strategicimplementation.us`
- ✅ Configures Pub/Sub notifications for new emails

**Gmail Watch Details**:
- Watches expire after 7 days
- Cloud Scheduler (set up in Step 6) will auto-refresh every 6 hours
- Notifications sent to `gmail-new-emails` Pub/Sub topic

### Step 6: Set Up Cloud Scheduler (Auto-refresh Gmail Watch)

Create a scheduled job to refresh the Gmail watch every 6 hours:

```powershell
$SERVICE_URL = "https://medilink-gmail-orchestrator-XXXXX-uc.a.run.app"  # From deployment output
$PROJECT_ID = "YOUR_PROJECT_ID"
$ADMIN_SECRET = "your-admin-secret"  # Saved from deployment
$SA_EMAIL = "medilink-gmail-orchestrator@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud scheduler jobs create http gmail-watch-refresh `
    --schedule="0 */6 * * *" `
    --uri="${SERVICE_URL}/admin/resume-history" `
    --http-method=POST `
    --headers="Content-Type=application/json" `
    --message-body="{\"secret\":\"${ADMIN_SECRET}\"}" `
    --oidc-service-account-email=$SA_EMAIL `
    --oidc-token-audience="${SERVICE_URL}/admin/resume-history" `
    --project=$PROJECT_ID
```

---

## ✅ Testing and Verification

### Test 1: Health Check

```bash
curl https://YOUR_SERVICE_URL/health
```

**Expected Response**:
```json
{
  "status": "ok",
  "service": "medilink-gmail-orchestrator",
  "version": "1.0.0",
  "config_loaded": "true",
  "project_id": "YOUR_PROJECT_ID",
  "mailbox_user": "daniel@strategicimplementation.us",
  "python_version": "3.11.x"
}
```

### Test 2: Send Test Email

1. **Send a test email** with an attachment to your monitored Gmail
2. **Wait for forwarding** (usually < 30 seconds)
3. **Check Cloud Run logs**:
   ```
   https://console.cloud.google.com/logs/viewer?project=YOUR_PROJECT_ID&resource=cloud_run_revision/service_name/medilink-gmail-orchestrator
   ```
4. **Verify in Firestore**:
   - Go to Firestore Console
   - Check `queues` collection - should see new document
5. **Verify in Cloud Storage**:
   - Go to Storage Console
   - Check `medilink-gmail-staging` bucket - should see attachment

### Test 3: Pipeline Status

```powershell
py -3.11 cloud/orchestrator/validate_and_complete_setup.py --validate-only
```

**Expected Output**:
```
✓ Cloud Run Service: Running
✓ Pub/Sub Topic: Active
✓ Pub/Sub Subscription: Active
✓ Gmail Watch: Registered
✓ Firestore: Connected
✓ Storage Bucket: Accessible
```

### Test 4: Full Verification

```powershell
py -3.11 cloud/orchestrator/validate_and_complete_setup.py --verify-runtime
```

This performs comprehensive checks of all components.

---

## 🔧 Troubleshooting

### Common Issues

#### "Service account does not have Gmail API access"
**Solution**: Ensure the service account has these IAM roles:
- `roles/gmail.modify`
- `roles/gmail.readonly`

Check with:
```bash
gcloud projects get-iam-policy YOUR_PROJECT_ID --flatten="bindings[].members" --filter="bindings.members:serviceAccount:medilink-gmail-orchestrator@YOUR_PROJECT_ID.iam.gserviceaccount.com"
```

#### "Gmail watch registration failed"
**Solutions**:
1. Verify the service account has a Gmail account created
2. Check that Workload Identity Federation is properly configured
3. Ensure Gmail API is enabled: `gcloud services list --enabled | grep gmail`

#### "No emails being processed"
**Solutions**:
1. Verify email forwarding is set up correctly
2. Check that the service account's Gmail account exists and is accessible
3. Check Cloud Run logs for errors
4. Verify Pub/Sub messages are being published:
   ```
   gcloud pubsub topics list-subscriptions gmail-new-emails --project=YOUR_PROJECT_ID
   ```

#### "Firestore permission denied"
**Solution**: Verify service account has `roles/datastore.user`:
```bash
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:medilink-gmail-orchestrator@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/datastore.user"
```

### Verification Commands

**Check Cloud Run service**:
```bash
gcloud run services describe medilink-gmail-orchestrator --region=us-central1 --project=YOUR_PROJECT_ID
```

**Check Pub/Sub topic**:
```bash
gcloud pubsub topics describe gmail-new-emails --project=YOUR_PROJECT_ID
```

**Check Gmail watch**:
```powershell
py -3.11 cloud/orchestrator/validate_and_complete_setup.py --verify-runtime
```

**View Cloud Run logs**:
```bash
gcloud run logs read --service=medilink-gmail-orchestrator --region=us-central1 --project=YOUR_PROJECT_ID --limit=50
```

---

## 🔐 Advanced Configuration

### Environment Variables

The Cloud Run service uses these environment variables:

**Required**:
- `PROJECT_ID`: GCP Project ID
- `MAILBOX_USER`: Service account's Gmail address
- `GCS_BUCKET`: Cloud Storage bucket for attachments

**Optional**:
- `PORT`: Port to listen on (default: 8080)
- `LOG_LEVEL`: Logging verbosity (default: INFO)

### Security Considerations

1. **No Service Account Keys**: Uses Workload Identity Federation
2. **HTTPS Only**: Cloud Run enforces HTTPS
3. **IAM Authentication**: Pub/Sub uses IAM for auth
4. **Private Networking**: Can configure VPC connector if needed

### Custom Filters

To forward only specific emails, create Gmail filters:

1. In Gmail Settings → Filters and Blocked Addresses
2. Create filter with criteria (e.g., from: `claims@insurance.com`)
3. Action: "Forward it to" → service account email

### Scaling Configuration

Cloud Run auto-scales based on traffic. To configure:

```bash
gcloud run services update medilink-gmail-orchestrator \
    --min-instances=0 \
    --max-instances=10 \
    --concurrency=80 \
    --region=us-central1 \
    --project=YOUR_PROJECT_ID
```

---

## 📊 What Was Set Up

### GCP Resources Created

| Resource | Name | Purpose |
|----------|------|---------|
| **Service Account** | `medilink-gmail-orchestrator` | Identity for services |
| **Cloud Run Service** | `medilink-gmail-orchestrator` | Email processing service |
| **Pub/Sub Topic** | `gmail-new-emails` | Email notification queue |
| **Pub/Sub Subscription** | `gmail-new-emails-sub` | Delivers to Cloud Run |
| **Cloud Storage Bucket** | `medilink-gmail-staging` | Attachment storage |
| **Firestore Collections** | `queues`, `sync_state` | State management |

### IAM Roles Assigned

| Role | Purpose |
|------|---------|
| `roles/storage.objectAdmin` | Cloud Storage access |
| `roles/datastore.user` | Firestore access |
| `roles/pubsub.publisher` | Pub/Sub publishing |
| `roles/gmail.modify` | Gmail API access |
| `roles/gmail.readonly` | Gmail API read access |

---

## 🎯 Key Differences from Old Approach

| Old Approach | New Approach |
|-------------|-------------|
| Domain-wide delegation | Direct service account access |
| Service account keys | Workload Identity Federation |
| User impersonation | Service account owns its Gmail |
| Complex OAuth flow | Simple ADC authentication |
| Security risk (keys) | Secure (no keys stored) |
| Difficult troubleshooting | Clear error messages |

---

## 📞 Support

If you encounter issues:

1. **Check this troubleshooting section** above
2. **Run diagnostics**: `py -3.11 cloud/orchestrator/validate_and_complete_setup.py --validate-only`
3. **Review Cloud Run logs** for error details
4. **Verify all manual setup steps** were completed
5. **Check GCP quotas** and billing status

### Additional Resources

- [Gmail API Documentation](https://developers.google.com/gmail/api)
- [Cloud Run Documentation](https://cloud.google.com/run/docs)
- [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation)
- [Firestore Documentation](https://cloud.google.com/firestore/docs)

---

**The new direct service account access approach is simpler, more secure, and easier to maintain!** 🚀

**Document Status**: ✅ Consolidated Setup Guide  
**Last Updated**: 2025-11-21  
**Supersedes**: FRESH_SETUP_GUIDE.md, QUICK_START.md, SETUP_GUIDE.md


---

## 🧭 Launcher insertion point (local client activation)

Yes — the correct insertion point is the MediCafe launcher/entry routing layer.

Current behavior after this update:
- `python -m MediCafe download_emails` stays backward compatible and launches legacy `MediLink_Gmail.py` by default.
- Set `MEDILINK_DOWNLOAD_MODE=cloud` to make `download_emails` launch `xp_client/medilink_cloud_daemon.py` instead.
- `python -m MediCafe cloud_daemon [config_path]` now launches the cloud daemon directly.

Recommended rollout:
1. Pilot with `MEDILINK_DOWNLOAD_MODE=cloud` on one workstation.
2. Validate end-to-end (ready -> files -> ack).
3. Flip installer/startup defaults to cloud mode for all clinics.
4. Retire legacy `MediLink_Gmail.py` launch path after stabilization window.
