# KHY Minimal Large-Task Framework Blueprint

This document defines a practical, minimal framework blueprint for handling long-running, dependency-heavy, failure-prone tasks in KHY-OS.

## 0. Goal and Scope

Goal:
- Build a controllable, resumable, observable runtime for large tasks.
- Avoid full rewrites; evolve from existing KHY components.

Non-goals:
- Not pursuing distributed execution in Phase 1.
- Not solving exactly-once globally in Phase 1.

## 1. Execution Semantics (Must Decide First)

Use the following baseline:
- Delivery semantics: `at-least-once`
- Retry policy: transient errors are retryable with capped exponential backoff
- Timeout policy: activity-based idle timeout (not hard wall-clock kill)

Implication:
- Duplicate delivery is possible by design.
- All side-effect operations must be idempotent at business boundary.

## 2. Architecture (Control Plane + Data Plane)

```text
API/CLI -> Task Controller -> Queue/Scheduler -> Worker Pool
                     |                |
                     v                v
                Task DB <---- Checkpoint Store
                     |
                     v
             Metrics + Logs + Traces
```

Control Plane responsibilities:
- Task lifecycle management
- Lease/claim/heartbeat/cancel/pause/resume
- Retry scheduling and dead-letter routing

Data Plane responsibilities:
- Execute concrete task steps
- Emit progress events
- Persist checkpoints

## 3. Canonical State Machine

Allowed states:
- `queued`
- `claimed`
- `running`
- `retry_wait`
- `pausing`
- `paused`
- `cancelling`
- `succeeded`
- `failed`
- `cancelled`
- `dead_letter`

Critical transitions:
- `queued -> claimed -> running -> succeeded`
- `running -> retry_wait -> claimed`
- `running -> pausing -> paused -> running`
- `running|paused -> cancelling -> cancelled`
- `retry_wait -> dead_letter` (exceeded retry budget)

Rules:
- All transitions must be validated centrally.
- Terminal states are immutable.

## 4. Minimal Persistence Model

### 4.1 `tasks`

Core fields:
- `id` (pk)
- `type`
- `status`
- `payload_json`
- `priority`
- `attempt_count`
- `max_attempts`
- `next_run_at`
- `lease_owner`
- `lease_until`
- `heartbeat_at`
- `progress_pct`
- `idempotency_key` (nullable, unique when present)
- `trace_id`
- `created_at`
- `updated_at`

### 4.2 `task_attempts`

Core fields:
- `task_id`
- `attempt_no`
- `worker_id`
- `started_at`
- `ended_at`
- `result_status`
- `error_type`
- `error_message`
- `retry_delay_ms`

### 4.3 `task_checkpoints`

Core fields:
- `task_id`
- `step_no`
- `progress_pct`
- `state_blob_json`
- `schema_version`
- `created_at`

Checkpoint rules:
- Save at stable boundaries (step complete, batch complete, offset commit).
- Resume from latest compatible checkpoint.

## 5. Side-Effect Governance (Default Safe Mode)

Large-task runtime must enforce "default non-side-effect":

1. `plan()` phase:
- Read-only
- Compute and validate execution plan
- No external writes

2. `commit()` phase:
- Side effects allowed only with explicit commit intent
- All side effects pass through one executor boundary

Required pattern:
- `dry_run=true` by default
- `commit=true` required to execute side effects

## 6. Idempotency and Exactly-Once Boundary

`idempotency_key` does not remove side effects themselves.
It guarantees "same intent executes effect at most once" under retries.

Server-side requirements:
- Unique key constraint (`scope + idempotency_key`)
- Business write + key record in one transaction
- Repeat calls with same key return original result (response replay)

## 7. Resilience Guardrails

Must-have:
- Capped exponential backoff with jitter
- Circuit breaker for unstable downstreams
- Deadline + idle timeout separation
- Retry budget and dead-letter queue
- Cancellation token propagation

Nice-to-have (Phase 2):
- Adaptive concurrency slots
- Dynamic priority aging

## 8. Observability Baseline

Every task event should include:
- `trace_id`
- `task_id`
- `attempt_no`
- `state_from`, `state_to`
- `latency_ms`
- `error_type` (if any)

Metrics baseline:
- Queue depth
- Claim latency
- Success rate
- Retry rate
- Dead-letter rate
- P95/P99 end-to-end latency

## 9. KHY-OS Mapping (Current Codebase)

Existing useful components:
- `backend/src/services/retryWithBackoff.js`
- `backend/src/services/circuitBreaker.js`
- `backend/src/utils/spawnWithIdleTimeout.js`
- `backend/src/services/backgroundTaskManager.js`
- `backend/src/tasks/taskStore.js`
- `backend/src/tasks/diskOutput.js`

Current gap:
- Multiple task stores exist (`tasks/taskStore`, `backgroundTaskManager`, `tools/_taskStore`)
- Need one canonical source of truth backed by persistent storage

## 10. 7-Day Implementation Plan

Day 1-2:
- Introduce persistent task tables
- Implement centralized state transition validator

Day 3-4:
- Implement claim/lease/heartbeat/retry/dead-letter flow
- Add worker crash recovery by lease expiry

Day 5:
- Add checkpoint save/load/resume
- Add checkpoint schema version compatibility checks

Day 6:
- Add structured logs + trace_id propagation
- Add dashboard-level metrics aggregation

Day 7:
- Fault drills: kill worker, inject network failure, duplicate delivery, slow downstream
- Verify no data corruption and resumability from checkpoint

## 11. Phase-1 Acceptance Criteria

- Long task can resume from checkpoint after process crash.
- Repeated delivery does not duplicate critical side effects.
- Idle timeout never kills active-progress tasks.
- Task states are queryable and auditable end-to-end.
- Dead-letter tasks include actionable failure metadata.

## 12. Implementation Status (2026-05-16)

Implemented in codebase:
- Canonical persistent runtime store:
  - `backend/src/tasks/largeTaskRuntimeStore.js`
  - centralized state transitions, retries, dead-letter, checkpoints, idempotency records, event stream, metrics
  - retry classification policy (retryable vs non-retryable) with immediate terminal fail for permanent errors
  - configurable retry policy (`getRetryPolicy`/`setRetryPolicy`) for:
    - non-retryable error types/status codes/error kinds
    - retryable error kinds
    - default unknown-error behavior (`default_retryable`)
  - retry policy persistence + audit trail:
    - persisted `retry_policy` snapshot
    - persisted `retry_policy_events` stream with metadata (`trace_id`, `actor`, `source`, `reason`, `before_policy`, `after_policy`)
    - event pagination/filtering via `after_id` and `trace_id`
  - retry policy high-risk approval workflow:
    - persisted `retry_policy_approval_tickets`
    - patch-hash binding to prevent approval ticket replay on different patches
    - approval lifecycle: `pending -> approved/rejected -> consumed/expired`
    - retention compaction for terminal approval tickets/events (age + count bounded, pending tickets preserved)
- Orchestrator with plan/commit side-effect gate:
  - `backend/src/tasks/largeTaskOrchestrator.js`
  - side-effect commit protected by per-scope circuit breaker (fail-fast on open circuit)
  - control-plane pause/cancel awareness during execution (`task_paused` / `task_cancelled`)
  - activity-based sliding idle timeout (`idle_timeout_ms`) with progress-aware resets (`markActivity`, `reportProgress`, `saveCheckpoint`, `commit`)
- Continuous worker loop service:
  - `backend/src/tasks/largeTaskWorkerService.js`
  - start/stop/status/runTick
  - lease requeue + run budget per tick + queue depth/last run summary
  - propagates `idle_timeout_ms` to each task run
  - propagates `retry_policy` to each task run
  - Control-plane HTTP routes:
  - `backend/src/routes/largeTasks.js`
  - task APIs + metrics/events/SSE + worker APIs:
    - `GET /api/large-tasks/worker/status`
    - `POST /api/large-tasks/worker/start`
    - `POST /api/large-tasks/worker/stop`
    - `GET /api/large-tasks/circuit/commit?scope=...` (commit circuit breaker observability)
    - `POST /api/large-tasks/:taskId/run` accepts `idle_timeout_ms`
    - `POST /api/large-tasks/run-next` accepts `idle_timeout_ms`
    - task run options accept `retry_policy` overrides
    - `GET /api/large-tasks/retry-policy` (optionally include audit events)
    - `GET /api/large-tasks/retry-policy/events` (audit event stream query)
    - `POST /api/large-tasks/retry-policy` (policy update with audit record)
    - `GET /api/large-tasks/retry-policy/approvals/pending`
    - `GET /api/large-tasks/retry-policy/approvals/retention`
    - `POST /api/large-tasks/retry-policy/approvals/retention` (online tuning for approval ticket/event retention policy with audit event)
    - `GET /api/large-tasks/retry-policy/approvals/retention/events` (retention-policy audit event query)
    - `GET /api/large-tasks/retry-policy/approvals/retention/stream` (retention-policy audit SSE replay + watch + Last-Event-ID resume)
    - `GET /api/large-tasks/retry-policy/approvals/events` (historical approval-event query with `after_id`/filter pagination)
    - `POST /api/large-tasks/retry-policy/approvals/decision`
    - high-risk update path returns `202 approval_required` and requires `approval_ticket_id` for final apply
    - `GET /api/large-tasks/retry-policy/approvals/stream` (SSE replay + watch + Last-Event-ID resume)
    - `POST /api/large-tasks/:taskId/pause`
    - `POST /api/large-tasks/:taskId/resume`
    - `POST /api/large-tasks/:taskId/cancel`
  - retry policy update safety validation:
    - strict allowed keys
    - token format/length checks
    - status code range checks
    - array cardinality limits
    - guardrail floor rules block dangerous combinations (for example, `default_retryable=false` with no transient retry signal)
  - audit/event retry metadata contract:
    - attempts/events expose `retryable`, `retry_classification`, `error_kind`, `status_code`
    - retry policy outcomes are queryable from `GET /api/large-tasks/:taskId/audit` and `GET /api/large-tasks/events`
  - cross-device handover snapshot API:
    - `GET /api/large-tasks/handover/snapshot`
    - compact mode for mobile rendering: `GET /api/large-tasks/handover/snapshot?format=mobile`
    - returns recent key operations + retention-policy change summary + active task summary + pending approvals/todos

Verification:
- Route/service/runtime/orchestrator tests passing:
  - `backend/tests/services/largeTaskRuntimeStore.test.js`
  - `backend/tests/services/largeTaskOrchestrator.test.js`
  - `backend/tests/services/largeTaskWorkerService.test.js`
  - `backend/tests/routes/largeTasks.route.test.js`
