# oh-my-openagent Alignment Learning Log

This log is mandatory for every oh-my-openagent alignment study session.

## 2026-05-15 Session 1

- Focus:
  - Primary: D2 Skills System
  - Secondary: D5 Governance & Quality
- New Differences:
  1. [Capability][High] Baseline has explicit skill-eval assets (`.agents/skills/work-with-pr-workspace/evals/`, iteration benchmark artifacts), while KHY currently lacks a standardized per-skill eval loop.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/evals/evals.json`
     - KHY Current: `backend/src/skills/` has loading/execution, but no mandatory eval artifact convention.
     - Decision: Adopt (introduce skill eval convention + minimal benchmark schema).
  2. [Process][Medium] Baseline uses multi-level AGENTS docs (root + `src/AGENTS.md` + `web/AGENTS.md`) with generated inventory; KHY currently has a strong root guide but no enforced hierarchical refresh mechanism.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/AGENTS.md`, `src/AGENTS.md`, `web/AGENTS.md`
     - KHY Current: repository root `AGENTS.md` only.
     - Decision: Adapt (add selective hierarchical AGENTS for high-change subtrees, not full auto-generation yet).
  3. [Quality][Medium] Baseline AGENTS includes quantified architecture inventory (file counts, hook counts, tool matrix); KHY docs are clear but less quantified for drift tracking.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/AGENTS.md` OVERVIEW/HOOK COMPOSITION sections.
     - KHY Current: architecture and rules are present, but no periodic numeric inventory baseline.
     - Decision: Adapt (add periodic inventory snapshot script/report for key subsystems).
- Actions Committed:
  - Code: Updated `AGENTS.md` with mandatory alignment loop, score model, exit criteria, and log contract.
  - Verification: Manual baseline inspection from zip + path evidence capture; next step requires scripted inventory/check automation.
- Score Update:
  - D2: 1 -> 1
  - D5: 1 -> 2
- Next Highest-Impact Gap:
  - Build a repeatable skill-eval workflow (schema + runner + pass/fail thresholds) and connect it to CI.

## 2026-05-15 Session 2

- Focus:
  - Primary: D2 Skills System
  - Secondary: D5 Governance & Quality
- New Differences:
  1. [Process][High] Baseline eval assets define machine-readable evaluation cases (prompt + assertions) as a stable contract, while KHY had no formalized skill-eval config schema.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/evals/evals.json`
     - KHY Current: added `backend/src/skills/evals/skill-eval-config.schema.json` + `backend/src/skills/evals/skill-eval-baseline.json`
     - Decision: Adopt (introduce explicit eval config contract with threshold gating).
  2. [Quality][High] Baseline keeps iteration benchmark artifacts for auditability; KHY previously had no persisted run report for skill quality checks.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/iteration-1/benchmark.json`
     - KHY Current: added runner report output `docs/reports/skill-eval-latest.json` + report schema `backend/src/skills/evals/skill-eval-report.schema.json`
     - Decision: Adopt (persist score/failure details for trend tracking).
  3. [Capability][Medium] Baseline process assumes consistent skill metadata usage; KHY manifests currently mix `trigger/command` and `userInvocable/user_invocable`, creating compatibility drift risk without explicit normalization.
     - Baseline Evidence: baseline eval assertions rely on stable, predictable skill behavior per artifact contract
     - KHY Current: `backend/src/skills/index.js` now normalizes trigger and invocable flags across both key styles
     - Decision: Adapt (preserve backward compatibility while converging to one canonical format later).
- Actions Committed:
  - Code: Added `scripts/ci/check-skill-evals.js`, eval schema/baseline files under `backend/src/skills/evals/`, CI hook in `.github/workflows/ci.yml`, npm script in `package.json`, loader compatibility hardening in `backend/src/skills/index.js`, and regression test `backend/tests/services/skillsManifestCompat.test.js`.
  - Verification:
    - `node --check scripts/ci/check-skill-evals.js`
    - `node --check backend/src/skills/index.js`
    - `node scripts/ci/check-skill-evals.js --report docs/reports/skill-eval-latest.json`
    - `npm run check:quality-gates`
    - `npm run --workspace backend test -- --runInBand backend/tests/services/skillsManifestCompat.test.js`
    - `node scripts/check-agent-rules.js --changed` (warnings exist in pre-existing modified files outside this alignment slice)
- Score Update:
  - D2: 1 -> 2
  - D5: 2 -> 2
- Next Highest-Impact Gap:
  - Promote static skill metadata eval to scenario eval (prompt-level assertions + pass-rate deltas), and reduce warning baseline by standardizing missing `category`/`platforms` fields in built-in manifests.

## 2026-05-15 Session 3

- Focus:
  - Primary: D2 Skills System
  - Secondary: D5 Governance & Quality
- New Differences:
  1. [Quality][High] Baseline eval artifacts assume consistent, complete skill metadata; KHY still had metadata drift (missing `category`/`platforms`) causing noisy quality signals.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/iteration-1/benchmark.json` (stable assertion accounting)
     - KHY Current: 12 built-in manifests were missing `category/platforms` before this session; now normalized.
     - Decision: Adopt (enforce metadata completeness to improve eval signal quality).
  2. [Capability][High] Baseline uses scenario-oriented eval cases with explicit assertion lists; KHY only had metadata-level checks and lacked scenario assertions for skill behavior guidance.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/evals/evals.json`
     - KHY Current: added `backend/src/skills/evals/skill-scenario-suite.json` + `scripts/ci/check-skill-scenarios.js` with prompt-level assertions and thresholds.
     - Decision: Adopt (add scenario-style eval contract and automated runner).
  3. [Process][Medium] Baseline stores iteration artifacts for continuous comparison; KHY had static report output but no second report channel for scenario-level tracking.
     - Baseline Evidence: `/tmp/oh-my-openagent-dev/oh-my-openagent-dev/.agents/skills/work-with-pr-workspace/iteration-1/eval-1/eval_metadata.json`
     - KHY Current: now persists `docs/reports/skill-scenario-eval-latest.json` and defines `backend/src/skills/evals/skill-scenario-report.schema.json`.
     - Decision: Adapt (start with latest-report contract; later extend to historical iteration snapshots).
- Actions Committed:
  - Code:
    - Normalized missing metadata in built-in manifests:
      - `backend/src/skills/built-in/batch/manifest.json`
      - `backend/src/skills/built-in/claude-api/manifest.json`
      - `backend/src/skills/built-in/debug/manifest.json`
      - `backend/src/skills/built-in/dream/manifest.json`
      - `backend/src/skills/built-in/hunter/manifest.json`
      - `backend/src/skills/built-in/keybindings/manifest.json`
      - `backend/src/skills/built-in/loop/manifest.json`
      - `backend/src/skills/built-in/remember/manifest.json`
      - `backend/src/skills/built-in/skillify/manifest.json`
      - `backend/src/skills/built-in/stuck/manifest.json`
      - `backend/src/skills/built-in/update-config/manifest.json`
      - `backend/src/skills/built-in/verify/manifest.json`
    - Added scenario eval suite and runner:
      - `backend/src/skills/evals/skill-scenario-suite.json`
      - `scripts/ci/check-skill-scenarios.js`
      - `backend/src/skills/evals/skill-scenario-report.schema.json`
    - CI/quality integration:
      - `package.json` (`check:skill-scenario-eval`, expanded `check:quality-gates`)
      - `.github/workflows/ci.yml` (new scenario eval step)
    - Documentation/report updates:
      - `backend/src/skills/evals/README.md`
      - `docs/reports/skill-eval-latest.json`
      - `docs/reports/skill-scenario-eval-latest.json`
  - Verification:
    - `node scripts/ci/check-skill-evals.js --report docs/reports/skill-eval-latest.json` -> PASS (`overall=1.000`, `warning-failures=0`)
    - `node scripts/ci/check-skill-scenarios.js --report docs/reports/skill-scenario-eval-latest.json` -> PASS (`15/15 assertions`)
    - `npm run --workspace backend test -- --runInBand backend/tests/services/skillsManifestCompat.test.js` -> PASS
    - `node scripts/check-agent-rules.js --changed` -> warnings remain in pre-existing unrelated files
    - `npm run check:quality-gates` -> PASS (version sync + syntax checks + skill eval + skill scenario eval)
- Score Update:
  - D2: 2 -> 3
  - D5: 2 -> 3
- Next Highest-Impact Gap:
  - Add historical iteration storage and trend comparison for scenario eval reports (latest vs previous N runs), then expand scenario suite coverage beyond 5 skills.

## 2026-05-15 Session 4

- Focus:
  - Primary: D1 Agent Harness
  - Secondary: D3 MCP & Tooling
- New Differences:
  1. [Capability][High] Hashline LINE#ID edit tool — content-hash validation prevents stale edits. Baseline uses xxh32 hash of line content → 2-char ID from `ZPMQVRWSNKTXJBYH` alphabet; validated before apply to reject edits on changed lines.
     - Baseline Evidence: `src/tools/hashline-edit/hash-computation.ts`, `src/tools/hashline-edit/validation.ts` (HashlineMismatchError with context display + suggestLineForHash recovery)
     - KHY Current: No content-hash edit protection. `toolCalling.js` applies edits without staleness validation.
     - Decision: Adapt — implement a simpler content-fingerprint guard in toolCalling.js or toolSandbox.js that rejects edits when the target line content has changed since last read.
  2. [Capability][High] IntentGate keyword detector — mode switching via magic words (ultrawork, search, analyze, team, hyperplan). Strips code blocks first, then regex-matches keywords to inject system prompt overrides and activate specialized modes.
     - Baseline Evidence: `src/hooks/keyword-detector/detector.ts`, `src/hooks/keyword-detector/constants.ts` (6 keyword types)
     - KHY Current: `capabilityAssessment.js` does basic capability pre-assessment, but there is no intent-triggered mode injection that dynamically alters agent behavior based on keywords.
     - Decision: Adopt — add an IntentGate layer in the agentic harness that detects power keywords and injects corresponding system prompt sections.
  3. [Capability][High] Ralph Loop — persistent continuation loop with session state tracking. Keeps agent going across tool failures, context overflow, and session boundaries. Cross-session work state persisted as "Boulder State" (plan progress, session origins, task sessions).
     - Baseline Evidence: `src/hooks/ralph-loop/ralph-loop-hook.ts`, `src/features/boulder-state/types.ts`
     - KHY Current: `agenticHarnessService.js` has retry-with-backoff for transient errors, but no persistent cross-session continuation loop. No boulder/work state tracking beyond session traces.
     - Decision: Adopt — extend agenticHarnessService with a session-persistent continuation mechanism and work-state snapshot.
  4. [Capability][High] Model-specific prompt variants per agent. Sisyphus agent has separate prompt builds for Claude Opus 4.7, Gemini, GPT-5.4, GPT-5.5, Kimi K2.6, each tuned for that model's behavior quirks (e.g. Gemini aggressive tendencies, GPT block-structured guidance).
     - Baseline Evidence: `src/agents/sisyphus/index.ts` — exports model-specific builders, `src/agents/sisyphus/gemini.ts` has corrective overlays
     - KHY Current: `compact/prompt.js` references model-specific prompting but KHY has no per-model prompt variant system for its agent loops.
     - Decision: Adapt — introduce a prompt-variant registry keyed by model family, starting with Claude/Gemini/GPT differentiation in the gateway adapter layer.
  5. [Quality][Medium] Edit error recovery hook — auto-injects "READ the file first" reminder when Edit tool fails (oldString not found). Reduces wasted turns from stale edit attempts.
     - Baseline Evidence: `src/hooks/edit-error-recovery/hook.ts`
     - KHY Current: Tool call errors bubble up raw to the agent with no recovery guidance injection.
     - Decision: Adopt — add error recovery hints in toolCalling.js that inject actionable guidance on common failures (file not found, edit mismatch, permission denied).
  6. [Quality][Medium] 5-tier hook composition — Session hooks, ToolGuard hooks (16 types), Transform hooks, Continuation hooks, Skill hooks. Each hook is config-gated via `isHookEnabled()` and wrapped in `safeCreateHook` for fault isolation.
     - Baseline Evidence: `src/plugin/hooks/create-tool-guard-hooks.ts` (16 guard types), `AGENTS.md` documents 54-61 hooks total
     - KHY Current: No formal hook composition system. Tool behavior is modified inline in toolCalling.js / toolSandbox.js. No fault-isolated hook wrappers.
     - Decision: Adapt — design a hook registry with at least 3 tiers (pre-tool, post-tool, error-recovery) and config-gating, without copying the full 5-tier complexity.
- Actions Committed:
  - Code: `agenticHarnessService.js` already provides the harness entrypoint. Next concrete code action: add an `intentGate` module in `backend/src/services/` that detects power keywords and returns prompt injection payloads (Difference #2). Stub file: `backend/src/services/intentGate.js`.
  - Verification: Run `node -e "require('./backend/src/services/agenticHarnessService')"` to confirm harness loads, then add unit test for intentGate keyword detection covering at least the 6 keyword types.
- Score Update:
  - D1: 1 -> 1 (agenticHarnessService exists with retry/context routing, but lacks persistent loop/boulder state — needs Ralph Loop equivalent)
  - D3: 1 -> 1 (toolSandbox + toolCalling functional but no hashline validation, no hook composition, no edit error recovery)
  - D4: 1 -> 1 (coordinatorMode + agentCommunicationService exist with message queue, but no mailbox protocol or backpressure)
- Next Highest-Impact Gap:
  - D1: Implement IntentGate keyword detector + persistent continuation loop (Ralph Loop equivalent) in agenticHarnessService. These two items unlock the most behavioral improvement for the least code change.
