Metadata-Version: 2.4
Name: custom-llm-eval
Version: 0.1.7
Summary: A comprehensive framework for evaluating Large Language Models with built-in support for bias, toxicity, relevancy metrics, custom evaluations, conversational test cases, release tracking, and token counting
Author: Atul B
Author-email: atulbmysuru@gmail.com
License: MIT
Project-URL: Homepage, https://github.com/atulbmysuru/custom-llm-eval
Project-URL: Issues, https://github.com/atulbmysuru/custom-llm-eval/issues
Project-URL: Repository, https://github.com/atulbmysuru/custom-llm-eval
Keywords: llm,evaluation,deepeval,ai,testing,bias,toxicity,nlp,token-counting
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: deepeval>=0.21.0
Requires-Dist: requests>=2.28.0
Requires-Dist: python-dotenv>=0.19.0
Requires-Dist: tiktoken>=0.5.0

# Degreed Coach Builder

[![CodeQL](https://github.com/degreed/degreed-coach-builder/actions/workflows/codeql.yml/badge.svg)](https://github.com/degreed/degreed-coach-builder/actions/workflows/codeql.yml)
[![PR Checks](https://github.com/degreed/degreed-coach-builder/actions/workflows/prcheck.yml/badge.svg)](https://github.com/degreed/degreed-coach-builder/actions/workflows/prcheck.yml)
[![Main Pipeline](https://github.com/degreed/degreed-coach-builder/actions/workflows/main.yml/badge.svg)](https://github.com/degreed/degreed-coach-builder/actions/workflows/main.yml)

![Python](https://img.shields.io/badge/Python-3.12-blue?logo=python&logoColor=white)
![FastAPI](https://img.shields.io/badge/FastAPI-Latest-009688?logo=fastapi&logoColor=white)
![Azure OpenAI](https://img.shields.io/badge/Azure%20OpenAI-GPT--4-0078D4?logo=microsoft-azure&logoColor=white)
![OpenAI](https://img.shields.io/badge/OpenAI-API-412991?logo=openai&logoColor=white)
![Kubernetes](https://img.shields.io/badge/Kubernetes-AKS-326CE5?logo=kubernetes&logoColor=white)
![LiveKit](https://img.shields.io/badge/LiveKit-WebRTC-00D4AA?logoColor=white)
![DataDog](https://img.shields.io/badge/DataDog-APM-632CA6?logo=datadog&logoColor=white)

> **The Heart and Soul of Maestro** - Core component of Degreed's AI-powered coaching platform, delivering personalized learning experiences through conversational AI.

**Made with ❤️ by the AI Team, Degreed**

## 🎯 Overview

**Degreed Coach Builder** is the central component of **Maestro**, Degreed's enterprise-scale AI coaching platform. As the heart and soul of the Maestro ecosystem, it creates intelligent, personalized coaches that guide users through learning paths and skill development. Built with modern cloud-native architecture, it provides real-time conversational AI experiences with text and voice interactions.

### ✨ Key Features

- 🤖 **AI-Powered Conversations** - GPT-4 powered coaching with context awareness
- 🎙️ **Multi-Modal Interactions** - Text and voice realtime conversations
- 📚 **RAG Integration** - Knowledge retrieval from uploaded documents
- 🔄 **Real-Time Processing** - Stream responses with Server-Sent Events
- 🌍 **Global Deployment** - Multi-region support (US, EU, Canada)
- 🔒 **Enterprise Security** - Comprehensive security scanning and compliance
- 📊 **Advanced Analytics** - DataDog monitoring with detailed observability

## 📋 Table of Contents

- [Overview](#-overview)
- [Architecture](#-architecture)
- [Features](#-features)
  - [Text-Based AI Coaching](#text-based-ai-coaching)
  - [Real-Time Voice Coaching](#real-time-voice-coaching)
  - [Quiz Generation](#quiz-generation)
  - [Document Upload & RAG](#document-upload--rag)
  - [Conversation Analysis](#conversation-analysis)
- [Coach Types](#-coach-types)
- [Environment Variables](#-environment-variables)
- [Quick Start](#-quick-start)
  - [Prerequisites](#prerequisites)
  - [Local Development](#local-development)
  - [Docker Development](#docker-development)
- [Local Development Setup](#-local-development-setup)
- [Project Structure](#-project-structure)
- [Configuration](#-configuration)
- [API Documentation](#-api-documentation)
- [Testing](#-testing)
- [Deployment](#-deployment)
  - [Deployment Environments](#deployment-environments)
  - [CI/CD Pipeline](#cicd-pipeline)
- [Security](#-security)
- [Monitoring & Observability](#-monitoring--observability)
- [Development Workflow](#-development-workflow)
- [Troubleshooting](#-troubleshooting)
- [Performance](#-performance)
- [Contributing](#-contributing)
- [Support](#-support)

## 🏗 Architecture

### Technology Stack

- **Backend**: FastAPI (Python 3.12) with async/await
- **AI/ML**: Azure OpenAI GPT-4, LangChain
- **Real-time**: LiveKit for WebRTC, Server-Sent Events
- **Vector Storage**: Azure Managed Redis with vector search
- **Cache**: Azure Cache for Redis
- **Infrastructure**: Kubernetes (AKS), Helm, Terraform
- **Monitoring**: DataDog APM, metrics, and logging
- **CI/CD**: GitHub Actions with multi-environment pipeline

### Architecture Documentation

📋 **Detailed Architecture**: [Maestro Architecture - Paragraph](https://degreedjira.atlassian.net/wiki/spaces/DSTS/pages/6762659883/Maestro+-+Architecture+-+Paragraph)

### System Components

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Frontend      │◄──►│   Coach Builder  │◄──►│   Degreed       │
│   Applications  │    │   (Maestro Core) │    │   Platform      │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │
                       ┌────────┼────────┐
                       │        │        │
               ┌───────▼──┐ ┌───▼────┐ ┌──▼──────────┐
               │ LiveKit  │ │ Azure  │ │ Azure       │
               │ Realtime │ │ Redis  │ │ Managed     │
               │          │ │ Cache  │ │ Redis       │
               └──────────┘ └────────┘ └─────────────┘
```

## ✨ Features

### Text-Based AI Coaching

Deliver intelligent conversational coaching through Server-Sent Events (SSE) for real-time streaming responses:

- **Context-Aware Conversations** - GPT-4 powered with full conversation history
- **Personalized Coaching** - Adapts to user's learning style and goals
- **Degreed Platform Integration** - Access user profiles, skills, pathways, and content
- **Streaming Responses** - Real-time SSE for progressive message delivery
- **Function Calling** - Dynamic tool use for content discovery and skill recommendations

**Endpoints**: `/dgcb/api/sse/llm-text-connect/{session_id}`, `/dgcb/api/sse/llm-text-sse/{sessionId}`

### Real-Time Voice Coaching

Immersive voice-based coaching powered by LiveKit WebRTC infrastructure:

- **Voice Conversations** - Natural voice interactions with AI coaches
- **LiveKit Integration** - Low-latency WebRTC for real-time audio
- **Noise Cancellation** - AI-powered background noise reduction
- **Real-Time Agents** - Specialized agents for voice conversation handling
- **STUNner Gateway** - WebRTC gateway for Kubernetes deployment

**Architecture**: LiveKit server + Real-time agents + STUNner gateway operator

**Endpoints**: `/dgcb/api/realtime/register-realtime-call`, `/dgcb/api/livekit/list-rooms`

### Quiz Generation

AI-powered quiz creation and inference for skills assessment with enhanced document processing:

- **Automated Quiz Generation** - Create quizzes from job roles, skills, or topics
- **Smart Question Design** - Multiple choice questions with distractor generation
- **Quiz Inference** - Generate quizzes based on existing content or pathways
- **Validation** - Security and quality validation for generated content
- **Enhanced Topic Extraction** - TOC detection and structured topic extraction from uploaded documents
- **Intelligent Question Selection** - LLM-based selection with topic diversity and concept deduplication
- **Quiz Modification** - Add questions, change difficulty, or regenerate existing quizzes
- **Semantic Deduplication** - LLM-powered detection of concept-level question duplicates
- **Efficient Validation** - Optimized validation that only checks selected questions
- **Auto-Repair** - Automatic fixing of invalid or malformed questions

**Endpoints**: `/dgcb/api/coach_builder/fill-quiz-fields`

**Enhanced Features**:
- **Document Topic Extraction**: Analyzes document structure including Table of Contents for accurate topic detection
- **Structured Topics**: Extracts topics with subtopics and page ranges for better content representation
- **Background Processing**: Handles large document sets with parallel extraction and processing
- **Backward Compatibility**: Maintains legacy string topic format while supporting new structured format

### Document Upload & RAG

Retrieval-Augmented Generation (RAG) with document knowledge bases and enhanced topic extraction:

- **Multi-Format Support** - PDF, DOCX, TXT document processing
- **Vector Search** - Semantic search using Azure Managed Redis vector store
- **Semantic Chunking** - Intelligent document segmentation for optimal retrieval
- **Knowledge Base Management** - Upload, delete, and clone document collections
- **Background Processing** - Async file processing for large documents
- **Deep Clone** - Duplicate entire knowledge bases across coaches
- **Enhanced Topic Detection** - Automatic extraction of topics including Table of Contents analysis
- **Structured Metadata** - Topic extraction includes subtopics and page ranges for better organization

**Endpoints**: `/dgcb/api/rag/upload-files`, `/dgcb/api/rag/upload-files-v1`, `/dgcb/api/coach_builder/deep-clone`

### Conversation Analysis

Extract insights and information from coaching conversations:

- **Full Conversation Extraction** - Comprehensive conversation analysis
- **Partial Extraction** - Analyze conversation segments
- **Background Processing** - Async extraction for long conversations
- **Structured Output** - Parse conversations into actionable data
- **Multi-Version Support** - v1 and v2 APIs for different use cases

**Endpoints**: `/dgcb/api/post_process/extract_conversation_info`, `/dgcb/api/post_process/extract_conversation_info_v2`

### Additional Features

- **Conversation Starters** - Generate contextual conversation opening prompts
- **Image Generation** - AI-powered image creation for coaching materials
- **Service Health Monitoring** - Comprehensive health checks and status reporting

## 🎯 Coach Types

The platform supports multiple specialized coach types optimized for different use cases:

### Career Development (CD) Coach

Guides users through career progression and development:
- Career path exploration
- Skill gap analysis
- Goal setting and achievement tracking
- Professional development recommendations

### Leadership Coach

Develops leadership skills and management capabilities:
- Leadership style assessment
- Team management strategies
- Decision-making frameworks
- Emotional intelligence development

### Skill Review Coach

Facilitates skill assessment and improvement:
- Skill level evaluation
- Personalized learning recommendations
- Practice and reinforcement activities
- Progress tracking and reporting

### Custom Coach Creation

Build coaches tailored to your organization's needs:
- **Coach Builder API** - Programmatic coach configuration
- **Personality Customization** - Define tone, style, and approach
- **Knowledge Domain** - Upload custom documents and knowledge bases
- **Validation & Security** - Automatic injection attack prevention
- **Pathway Integration** - Embed coaches within learning pathways

**Endpoints**: `/dgcb/api/coach_builder/fill-coach-fields`, `/dgcb/api/coach_builder/validate-coach-fields`

## 🔐 Environment Variables

Configure the application using environment variables grouped by functionality:

### Azure OpenAI Configuration

Required for AI-powered features:
```bash
AZURE_OPENAI_API_KEY=<your-api-key>
AZURE_OPENAI_ENDPOINT=<your-endpoint-url>
AZURE_OPENAI_API_VERSION=<api-version>        # e.g., 2024-02-15-preview
AZURE_OPENAI_DEPLOYMENT_NAME=<deployment-name> # GPT-4 deployment
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=<embed-deployment>
```

### Redis Configuration

For caching and vector storage with support for both Azure Managed Redis and local development:
```bash
MANAGED_REDIS_HOST=<redis-host>
MANAGED_REDIS_PORT=6380
MANAGED_REDIS_PASSWORD=<password>              # Optional with Entra ID
MANAGED_REDIS_USE_SSL=True
MANAGED_REDIS_AZURE_CLIENT_ID=<client-id>      # For Entra ID auth
REDIS_AUTO_REFRESH_ENABLED=True                # Enable automatic token refresh
REDIS_HEALTH_CHECK_INTERVAL=300                # Health check interval (seconds)
```

**Enhanced Features**:
- Automatic Azure AD token refresh for managed Redis
- Circuit breaker pattern for resilient authentication
- Health monitoring with configurable intervals
- Detailed logging for troubleshooting
- Support for both production (Azure Managed Redis) and local development modes

For detailed local setup instructions, see [LOCAL_DEVELOPMENT_SETUP.md](LOCAL_DEVELOPMENT_SETUP.md).

### LiveKit Configuration

For real-time voice coaching (optional if not using voice features):
```bash
LIVEKIT_API_KEY=<api-key>
LIVEKIT_API_SECRET=<api-secret>
LIVEKIT_URL=<livekit-server-url>
```

### DataDog Monitoring

For observability and metrics:
```bash
DD_API_KEY=<datadog-api-key>
DD_ENV=<environment>                           # dev, staging, production
DD_SERVICE=degreed-coach-builder
DD_VERSION=<app-version>
```

### Degreed Platform Integration

For connecting to Degreed services:
```bash
DEGREED_FRONTEND_URL=<frontend-url>
DEGREED_BACKEND_URL=<backend-api-url>
DEGREED_API_KEY=<api-key>
```

### Security & Optional Features

```bash
COOKIE_ENCRYPTION_KEY=<encryption-key>         # For secure cookie handling
DISABLE_DOCS=False                             # Set True to disable API docs in production
APP_VERBOSE=False                              # Enable verbose logging
```

**Note**: For local development, you can create a `.env` file in the `backend/` directory. For production, use Kubernetes secrets or Azure Key Vault.

## 🚀 Quick Start

### Prerequisites

- Python 3.12
- Docker & Docker Compose
- Azure CLI (for deployment and Azure services)
- kubectl & Helm (for Kubernetes)
- LiveKit CLI (optional, for voice feature development)

### Local Development

1. **Clone the repository**
   ```bash
   git clone https://github.com/degreed/degreed-coach-builder.git
   cd degreed-coach-builder
   ```

2. **Set up Python environment**
   ```bash
   cd backend
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   pip install -r requirements.txt
   ```

3. **Configure environment variables**

   Create a `.env` file in the `backend/` directory with required configuration:

   ```bash
   # Minimum required for basic functionality
   AZURE_OPENAI_API_KEY=your-azure-openai-key
   AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
   AZURE_OPENAI_API_VERSION=2024-02-15-preview
   AZURE_OPENAI_DEPLOYMENT_NAME=your-gpt4-deployment

   # Optional: Redis for caching (can use local Redis for development)
   MANAGED_REDIS_HOST=localhost
   MANAGED_REDIS_PORT=6379
   MANAGED_REDIS_USE_SSL=False

   # Optional: LiveKit for voice features
   # LIVEKIT_API_KEY=your-livekit-key
   # LIVEKIT_API_SECRET=your-livekit-secret
   # LIVEKIT_URL=ws://localhost:7880
   ```

   See [Environment Variables](#-environment-variables) section for complete configuration options.

4. **Run the application**
   ```bash
   uvicorn app.server:app --host 0.0.0.0 --port 8000 --reload
   ```

5. **Access the API**
   - API Documentation: http://localhost:8000/docs
   - Health Check: http://localhost:8000/dgcb/healthcheck
   - Interactive Docs (ReDoc): http://localhost:8000/redoc

### Docker Development

```bash
# Build and run with Docker
docker build -t coach-builder -f backend/Dockerfile backend/
docker run -p 8000:8000 coach-builder
```

## 🚀 Local Development Setup

For comprehensive step-by-step local development setup instructions, including Python environment configuration, LiveKit setup, Redis configuration, and IDE setup, refer to:

📖 **[LOCAL_DEVELOPMENT_SETUP.md](LOCAL_DEVELOPMENT_SETUP.md)** - Complete guide for setting up your development environment

### Quick Reference

**Environment Configuration:**
- Use `.env.example` as a template for your `.env` file
- Copy `cp .env.example .env` and update with your credentials
- See [Environment Variables](#-environment-variables) for all available options

**Local Redis Setup:**
- For development without Azure Managed Redis, configure local Redis instance
- Default local Redis: `MANAGED_REDIS_HOST=localhost`, `MANAGED_REDIS_PORT=6379`, `MANAGED_REDIS_USE_SSL=False`

**Verification:**
- Run health checks: `curl http://localhost:8000/dgcb/healthcheck`
- Access API docs: `http://localhost:8000/docs`

## 📁 Project Structure

```
degreed-coach-builder/
├── backend/                    # Main Python FastAPI application
│   ├── app/
│   │   ├── api/               # REST API endpoints (20+ modules)
│   │   │   ├── sse.py         # Server-Sent Events streaming
│   │   │   ├── realtime.py    # LiveKit WebRTC integration
│   │   │   ├── fill_coach_fields.py # Coach builder
│   │   │   ├── fill_quiz_fields.py  # Quiz generation
│   │   │   ├── validate_fill_coach.py # Coach validation
│   │   │   ├── post_process*.py  # Conversation analysis (v1 & v2)
│   │   │   ├── upload_files*.py  # Document upload & RAG (v1 & v2)
│   │   │   ├── conversation_starter.py # Conversation starters
│   │   │   ├── image_generator.py    # Image generation
│   │   │   ├── deep_clone.py        # Knowledge base cloning
│   │   │   ├── livekit_rooms.py     # WebRTC room management
│   │   │   ├── redis_monitoring.py  # Redis health monitoring
│   │   │   └── service_check_status.py # Service health checks
│   │   ├── dg_component/      # Degreed platform integration
│   │   │   ├── coach/         # Coach management
│   │   │   ├── mentor/        # Mentor functionality
│   │   │   ├── pathway/       # Learning pathways
│   │   │   ├── skills/        # Skills Plus integration
│   │   │   ├── find_content/  # Content discovery & search
│   │   │   ├── profile.py     # User profile integration
│   │   │   └── experience.py  # Experience management
│   │   ├── llm/               # AI model interfaces & prompt management
│   │   │   ├── llm_client.py  # Azure OpenAI client
│   │   │   ├── generate_prompt.py # Dynamic prompt generation
│   │   │   ├── agents/        # AI agents for specialized tasks
│   │   │   └── tools/         # LLM function calling tools
│   │   ├── db/                # Database & vector store managers
│   │   │   ├── redis_*.py     # Redis cache & vector operations
│   │   │   └── extended_redis_vectorstore.py # Enhanced vector store
│   │   ├── realtime_agents/   # Real-time voice AI agents
│   │   ├── services/          # Business logic services
│   │   ├── models/            # Pydantic data models
│   │   ├── dependencies/      # FastAPI dependency injection
│   │   ├── metrics_tracking/  # DataDog APM instrumentation
│   │   ├── post_process/      # Advanced conversation processing
│   │   ├── utils/             # Security, validation & utilities
│   │   │   ├── security_validation.py # Input validation
│   │   │   ├── quiz_utils/    # Quiz-related utilities
│   │   │   └── role_to_skill.py # Role/skill mapping
│   │   ├── tests/             # Unit and integration tests (70+)
│   │   └── server.py          # FastAPI application entry point
│   ├── requirements.txt       # Python dependencies (159 packages)
│   ├── Dockerfile            # Multi-stage production container
│   └── DockerfileRealtime    # Real-time service container
├── devops/                    # Infrastructure and deployment
│   ├── infra/
│   │   ├── degreed-coach-builder-api/    # Main API Helm chart
│   │   ├── degreed-coach-realtime/       # Real-time service Helm chart
│   │   ├── dg-stunner-livekit/          # LiveKit WebRTC infrastructure
│   │   ├── stunner-gateway-operator/    # WebRTC gateway operator
│   │   ├── redis/                       # Redis configurations
│   │   └── terraform/                   # Infrastructure as Code (multi-env)
│   └── scripts/              # Deployment and utility scripts
├── .github/
│   ├── workflows/            # CI/CD GitHub Actions pipelines
│   ├── SECURITY.md          # Security policy
│   └── PULL_REQUEST_TEMPLATE.md # PR template
├── AISeleniumTests/          # Comprehensive AI testing suite
│   ├── Maestro_Bias_Accuracy_test/  # AI bias & accuracy validation
│   │   ├── CD_coach/        # Career Development coach tests
│   │   ├── Leadership_coach/ # Leadership coach tests
│   │   └── Skill_Review/    # Skill Review coach tests
│   └── Maestro_Smoke_test_env/      # Smoke testing
│       ├── CD_coach/        # Career Development smoke tests
│       ├── LD_coach/        # Leadership smoke tests
│       ├── SR_coach/        # Skill Review smoke tests
│       ├── Maestro_Studio/  # Studio interface tests
│       └── Voice_concurrency_test/ # Voice load testing
├── RedisauthTest/           # Redis authentication testing
│   └── REDIS_RECOVERY_IMPLEMENTATION.md # Enhanced Redis features
├── redis_migration/          # Redis migration utilities
├── INTEGRATION_GUIDE.md     # Redis enhanced features guide
├── CODEOWNERS               # Code ownership configuration
└── README.md                # This file
```

## 🔧 Configuration

### Coach Configuration

Coaches are configured through the API with customizable:
- **Personality traits** and communication style
- **Knowledge domains** and expertise areas  
- **Interaction modes** (text and voice realtime conversations)
- **Integration settings** with Degreed platform

## 🚢 Deployment

### Kubernetes Deployment

The application deploys to Kubernetes using Helm charts with support for multiple environments:

```bash
# Deploy main API service
helm upgrade --install coach-builder ./devops/infra/degreed-coach-builder-api \
  --namespace coach-builder \
  --values ./devops/infra/degreed-coach-builder-api/values.yaml

# Deploy realtime service
helm upgrade --install coach-realtime ./devops/infra/degreed-coach-realtime \
  --namespace coach-builder \
  --values ./devops/infra/degreed-coach-realtime/values.yaml

```

### CI/CD Pipeline

Automated deployment pipeline with GitHub Actions:

1. **Build & Test** - Docker build with security scanning
2. **Security Scan** - SAST, dependency, and container vulnerability scanning
3. **Deploy to Dev** - Automated deployment and testing
4. **Stage & Test** - Staging deployment with comprehensive testing
5. **Production** - Multi-region production deployment

### Deployment Environments

The application is deployed across 8 environments spanning multiple regions:

| Environment | Cluster | Region | Purpose |
|------------|---------|--------|---------|
| **Dev** | AISCUDGDEVAKS03 | US | Development and feature testing |
| **Staging** | DGSCULOWAKS01 | US | Pre-production validation |
| **Release** | DGSCULOWAKS01 | US | Release candidate testing |
| **Beta** | DGSCUCOREAKS01 | US | Limited production rollout |
| **Beta EU** | DGGWCEUCOREAKS01 | EU | European beta deployment |
| **Canada** | DGCACCACOREAKS01 | CA | Canadian production |
| **Production** | DGSCUCOREAKS01 | US | Main production environment |
| **Production EU** | DGGWCEUCOREAKS01 | EU | European production |

### Environment Promotion

```
Development → Staging → Release → Beta (US/EU) → Production (US/EU/CA)
```

**Multi-Region Considerations**:
- Data residency compliance for EU and Canada
- Region-specific configuration in Helm values
- Global load balancing for optimal performance
- Separate monitoring dashboards per region

## 🔒 Security

### Security Measures

- **Container Security** - Trivy vulnerability scanning
- **Code Analysis** - CodeQL, Bandit, Semgrep SAST tools
- **Dependency Scanning** - Safety checks for Python packages
- **Secret Detection** - TruffleHog and detect-secrets
- **Infrastructure Security** - Checkov for Kubernetes/Docker
- **Pre-commit Hooks** - Local security validation

### Security Scanning

The project includes comprehensive security scanning:

```bash
# Run security scans locally
pre-commit run --all-files

# View security results
gh workflow run security-scan.yml
```

## 🧪 Testing

### Test Types

- **Unit Tests** - Component-level testing (70+ test files)
- **Integration Tests** - API and service integration testing
- **AI Bias & Accuracy Testing** - Selenium-based validation for different coach types
- **Smoke Tests** - End-to-end workflow validation for production-like scenarios
- **Security Testing** - Automated vulnerability assessment with pre-commit hooks
- **Voice Concurrency Testing** - Load testing for real-time voice features

### Running Tests

#### Unit & Integration Tests

```bash
# Run all backend tests
cd backend
pytest tests/ -v

# Run with coverage report
pytest tests/ --cov=app --cov-report=html

# Run specific test file
pytest tests/test_llm.py -v

# Run tests matching pattern
pytest tests/ -k "test_coach" -v
```

#### AI Bias & Accuracy Tests

Test different coach types for bias and accuracy:

```bash
cd AISeleniumTests/Maestro_Bias_Accuracy_test

# Test Career Development coach
python CD_coach/test_cd_coach.py

# Test Leadership coach
python Leadership_coach/test_leadership_coach.py

# Test Skill Review coach
python Skill_Review/test_skill_review.py
```

#### Smoke Tests

End-to-end testing for various scenarios:

```bash
cd AISeleniumTests/Maestro_Smoke_test_env

# Test Career Development coach workflow
python CD_coach/smoke_test.py

# Test Maestro Studio interface
python Maestro_Studio/studio_smoke_test.py

# Test coach inside pathway
python Coach_inside_pathway/pathway_integration_test.py
```

#### Voice Concurrency Testing

```bash
cd AISeleniumTests/Maestro_Smoke_test_env/Voice_concurrency_test
python voice_load_test.py
```

#### Security Scanning

```bash
# Run all pre-commit security hooks
pre-commit run --all-files

# Run specific security checks
pre-commit run detect-secrets --all-files
pre-commit run bandit --all-files
```

### Test Configuration

Tests use environment variables from `.env.test` or fall back to `.env`. Configure test-specific settings:

```bash
TEST_AZURE_OPENAI_API_KEY=test-key
TEST_REDIS_HOST=localhost
TEST_ENVIRONMENT=test
```

## 📊 Monitoring & Observability

### DataDog Integration

- **APM Tracing** - Distributed request tracing
- **Custom Metrics** - Business and application metrics
- **Log Aggregation** - Centralized logging with correlation
- **Real-time Dashboards** - System health and performance
- **Alerting** - Proactive issue detection

📊 **Maestro Dashboard**: [DataDog Maestro Overview](https://app.datadoghq.com/dashboard/6z8-jwk-zm6/maestro-overview)

### Health Checks

- **Liveness Probe** - `/dgcb/healthcheck`
- **Readiness Probe** - `/dgcb/readiness`
- **Metrics Endpoint** - `/metrics`

## 💻 Development Workflow

### Setting Up Development Environment

1. **Install Pre-commit Hooks**

   Pre-commit hooks enforce code quality and security standards:

   ```bash
   # Install pre-commit
   pip install pre-commit

   # Install git hooks
   pre-commit install

   # Test hooks
   pre-commit run --all-files
   ```

2. **Security Tools Configured**

   The following tools run automatically on commit:
   - **detect-secrets** - Prevents committing secrets
   - **bandit** - Python security linter
   - **checkov** - Infrastructure security scanning
   - **hadolint** - Dockerfile linting
   - **trailing-whitespace** - Code formatting

3. **Development Commands**

   ```bash
   # Run application in development mode
   uvicorn app.server:app --reload --host 0.0.0.0 --port 8000

   # Run tests with coverage
   pytest tests/ --cov=app --cov-report=html

   # Run security scan locally
   pre-commit run --all-files

   # Format code (if using black/ruff)
   black backend/app
   ```

4. **Debugging**

   - Enable verbose logging: Set `APP_VERBOSE=True` in `.env`
   - Use DataDog APM for distributed tracing
   - Check logs in `/dgcb/logs` endpoint (if enabled)
   - Use FastAPI's automatic `/docs` for API testing

5. **Common Development Tasks**

   ```bash
   # Add new API endpoint
   # 1. Create endpoint in backend/app/api/
   # 2. Add tests in backend/app/tests/
   # 3. Update this README if public-facing

   # Add new environment variable
   # 1. Update README Environment Variables section
   # 2. Add to Kubernetes secrets/ConfigMap
   # 3. Document in deployment docs

   # Update dependencies
   pip install <package>
   pip freeze > requirements.txt
   ```

### Code Quality Standards

- **Type Hints** - Use Python type hints for all functions
- **Docstrings** - Document all public functions and classes
- **Error Handling** - Comprehensive error handling with proper logging
- **Security** - Input validation, SQL injection prevention, XSS protection
- **Testing** - Minimum 80% code coverage for new code

## 🤝 Contributing

### Development Workflow

1. **Fork** the repository
2. **Create** a feature branch (`git checkout -b feature/amazing-feature`)
3. **Install** pre-commit hooks (`pre-commit install`)
4. **Make** changes with comprehensive tests
5. **Commit** with conventional commits
6. **Push** and create a Pull Request

### Code Quality

- **Pre-commit hooks** enforce code standards
- **Security scanning** validates all changes
- **Comprehensive testing** required for new features
- **Documentation** updates for API changes

### Pull Request Process

1. Ensure all tests pass and security scans are clean
2. Update documentation for any API changes  
3. Add detailed PR description with testing instructions
4. Request review from maintainers
5. Address feedback and iterate

## 📜 API Documentation

### Core API Endpoints

#### **Health & Monitoring**
- `GET /dgcb/healthcheck` - Health check with custom header validation
- `GET /dgcb/readiness` - Readiness check for Kubernetes
- `GET /dgcb/redis_connection_check` - Redis connectivity validation
- `GET /dgcb/dogstatsd_connection_check` - DataDog metrics validation
- `GET /dgcb/metrics` - Prometheus metrics endpoint

#### **Authentication**
- `POST /dgcb/login_cookies` - Cookie-based authentication endpoint

#### **Server-Sent Events (SSE)**
- `POST /dgcb/api/sse/llm-text-connect/{session_id}` - Connect for text-based LLM interactions
- `GET /dgcb/api/sse/llm-text-sse/{sessionId}` - SSE stream for real-time text responses

#### **Real-time Voice/Video (LiveKit)**
- `POST /dgcb/api/realtime/register-realtime-call` - Register realtime voice call
- `POST /dgcb/api/realtime/test-register-realtime-call` - Register test realtime call
- `GET /dgcb/api/livekit/list-rooms` - List active LiveKit rooms (conditional)
- `GET /dgcb/api/livekit/list-participants/{room_name}` - List room participants (conditional)

#### **Coach Builder & Configuration**
- `POST /dgcb/api/coach_builder/configurations` - Generate coach configuration fields
- `POST /dgcb/api/coach_builder/fill-coach-fields` - Auto-fill coach profile fields
- `POST /dgcb/api/coach_builder/validate-coach-fields` - Validate coach configuration
- `POST /dgcb/api/coach_builder/validate-coach-injection-fields` - Validate against injection attacks
- `POST /dgcb/api/coach_builder/deep-clone` - Clone knowledge base collections

#### **Document Upload & RAG**
- `POST /dgcb/api/rag/upload-files` - Upload files with SSE streaming response
- `DELETE /dgcb/api/rag/delete-files` - Delete files from knowledge base
- `POST /dgcb/api/rag/upload-files-v1` - Upload files with background processing
- `DELETE /dgcb/api/rag/delete-files-v1` - Enhanced file deletion

#### **Quiz Generation**
- `POST /dgcb/api/coach_builder/fill-quiz-fields` - Generate quiz with questions and answers
- `POST /dgcb/api/coach_builder/validate-quiz-fields` - Validate quiz configuration

#### **Conversation Processing**
- `POST /dgcb/api/post_process/extract_conversation_info` - Extract full conversation information (v1)
- `POST /dgcb/api/post_process/extract_partial_conversation_info` - Extract partial conversation info (v1)
- `POST /dgcb/api/post_process/extract_conversation_info_v2` - Enhanced extraction with background processing
- `POST /dgcb/api/post_process/extract_partial_conversation_info_v2` - Enhanced partial extraction

#### **Additional Features**
- `POST /dgcb/api/conversation-starter` - Generate contextual conversation starters
- `POST /dgcb/api/image-generator` - AI-powered image generation for coaching materials

#### **Monitoring & Health**
- `GET /dgcb/api/redis-monitoring/health` - Redis health status and metrics
- `GET /dgcb/api/redis-monitoring/stats` - Redis connection statistics
- `GET /dgcb/api/service-check-status` - Comprehensive service health check

#### **Testing**
- `POST /dgcb/test/test_gpt` - Test GPT model response (requires custom header)

### Interactive API Docs

- **Swagger UI**: `/docs`
- **ReDoc**: `/redoc`

## 🌟 Performance

### Optimization Features

- **Docker Layer Caching** - 40-60% faster builds
- **Multi-stage Builds** - Smaller container images  
- **Connection Pooling** - Efficient database connections
- **Async Processing** - Non-blocking I/O operations
- **Content Delivery** - Global edge caching
- **Auto-scaling** - Kubernetes HPA for dynamic scaling

### Benchmarks

- **Response Time** - < 200ms for API calls
- **Throughput** - 1000+ concurrent users
- **Availability** - 99.9% uptime SLA
- **Build Time** - ~35s with full caching

## 📞 Support

### Getting Help

- **Security** - See [SECURITY.md](.github/SECURITY.md)
- **Documentation** - `/dgcb/docs` endpoint
- **Issues and JIRA Board** - [AI Data Science Project Board](https://degreedjira.atlassian.net/jira/software/projects/AIDATASCI/boards/85)
- **Status Page** - [DataDog Maestro Overview](https://app.datadoghq.com/dashboard/6z8-jwk-zm6/maestro-overview)

## 🔧 Troubleshooting

### Common Issues and Solutions

#### Redis Authentication Failures

**Symptom**: `ConnectionError: Authentication failed` or token expiration errors

**Solutions**:
```bash
# 1. Verify Azure Entra ID credentials
echo $MANAGED_REDIS_AZURE_CLIENT_ID

# 2. Check token refresh is enabled
REDIS_AUTO_REFRESH_ENABLED=True

# 3. Monitor Redis health
curl http://localhost:8000/dgcb/api/redis-monitoring/health

# 4. Review Redis recovery implementation
cat INTEGRATION_GUIDE.md
```

**Enhanced Features**: Automatic token refresh at 80% lifetime prevents most auth issues. See [INTEGRATION_GUIDE.md](INTEGRATION_GUIDE.md).

#### LiveKit Connection Issues

**Symptom**: Voice features not working, WebRTC connection failures

**Solutions**:
```bash
# 1. Verify LiveKit configuration
echo $LIVEKIT_URL
echo $LIVEKIT_API_KEY

# 2. Check LiveKit rooms
curl http://localhost:8000/dgcb/api/livekit/list-rooms

# 3. Test WebRTC connectivity
# Ensure STUNner gateway is running in Kubernetes

# 4. Check firewall/network policies
# LiveKit requires UDP ports for WebRTC
```

#### Azure OpenAI Rate Limiting

**Symptom**: `429 Too Many Requests` errors

**Solutions**:
- Check your Azure OpenAI quota and limits
- Implement exponential backoff (already built-in)
- Review DataDog metrics for request rates
- Consider upgrading Azure OpenAI tier

#### Environment Variable Issues

**Symptom**: Application starts but features don't work

**Solutions**:
```bash
# 1. Verify all required variables are set
cat .env | grep AZURE_OPENAI
cat .env | grep MANAGED_REDIS

# 2. Check for typos in variable names
# Variable names are case-sensitive

# 3. Restart application after .env changes
# Changes require app restart to take effect
```

#### Docker Build Failures

**Symptom**: Docker build fails or is very slow

**Solutions**:
```bash
# 1. Clear Docker cache
docker builder prune -a

# 2. Build with no cache
docker build --no-cache -t coach-builder -f backend/Dockerfile backend/

# 3. Check disk space
df -h

# 4. Verify all files are accessible
ls -la backend/requirements.txt
```

#### Test Failures

**Symptom**: Tests fail locally but should pass

**Solutions**:
```bash
# 1. Use test environment variables
cp .env .env.test
# Edit .env.test with test-specific config

# 2. Clear pytest cache
pytest --cache-clear

# 3. Run tests in isolation
pytest tests/test_specific.py -v

# 4. Check for test dependencies
pip install -r requirements.txt
```

#### Performance Issues

**Symptom**: Slow API responses, timeouts

**Solutions**:
- **Check DataDog APM**: Review traces for bottlenecks
- **Redis Connection**: Verify Redis is healthy and responding
- **Azure OpenAI**: Check API latency in Azure portal
- **Database Queries**: Look for slow vector searches
- **Memory Usage**: Check container memory limits

```bash
# Monitor application metrics
curl http://localhost:8000/dgcb/metrics

# Check service health
curl http://localhost:8000/dgcb/api/service-check-status
```

#### Pre-commit Hook Failures

**Symptom**: Commits are blocked by pre-commit hooks

**Solutions**:
```bash
# 1. Run hooks manually to see detailed errors
pre-commit run --all-files

# 2. Fix specific hook failures
pre-commit run detect-secrets --all-files
pre-commit run bandit --all-files

# 3. Update pre-commit hooks
pre-commit autoupdate

# 4. Bypass hooks temporarily (use with caution)
git commit --no-verify -m "message"
```

### Getting Additional Help

1. **Check Logs**:
   ```bash
   # Application logs
   kubectl logs -n coach-builder deployment/coach-builder-api

   # DataDog logs
   # Visit DataDog dashboard for centralized logs
   ```

2. **Health Checks**:
   ```bash
   # Comprehensive health check
   curl http://localhost:8000/dgcb/api/service-check-status

   # Individual service checks
   curl http://localhost:8000/dgcb/healthcheck
   curl http://localhost:8000/dgcb/redis_connection_check
   curl http://localhost:8000/dgcb/dogstatsd_connection_check
   ```

3. **Resources**:
   - [Redis Enhanced Features](INTEGRATION_GUIDE.md)
   - [Security Policy](.github/SECURITY.md)
   - [JIRA Board](https://degreedjira.atlassian.net/jira/software/projects/AIDATASCI/boards/85)
   - [DataDog Dashboard](https://app.datadoghq.com/dashboard/6z8-jwk-zm6/maestro-overview)

## 📄 License

This project is proprietary software owned by Degreed. All rights reserved.

## 🏆 Acknowledgments

- **Degreed Engineering Team** - Core development and architecture
- **DevOps Team** - Infrastructure and deployment automation
- **InfoSec Team** - Comprehensive security framework
- **QA Team** - Testing and quality assurance

---


For more information, visit [Degreed](https://degreed.com) or contact the development team.
