Metadata-Version: 2.4
Name: gamma-sigma
Version: 0.8.0
Summary: GAMMA-Σ: governed online dictionary-learning substrate — streaming novelty discovery with calibrated detection channels
Author: Weston Coogan / EfferentHQ
License: MIT
Project-URL: Homepage, https://github.com/weston-cmyk/optc-project
Project-URL: Issues, https://github.com/weston-cmyk/optc-project/issues
Project-URL: Changelog, https://github.com/weston-cmyk/optc-project/blob/main/CHANGELOG.md
Keywords: anomaly-detection,online-learning,dictionary-learning,sparse-coding,streaming,security-analytics,drift
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pandas>=2; extra == "dev"
Requires-Dist: pyarrow>=14; extra == "dev"
Requires-Dist: fastapi>=0.110; extra == "dev"
Requires-Dist: uvicorn>=0.29; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Provides-Extra: serve
Requires-Dist: fastapi>=0.110; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Provides-Extra: ingest
Requires-Dist: pandas>=2; extra == "ingest"
Requires-Dist: pyarrow>=14; extra == "ingest"
Dynamic: license-file

# GAMMA-Σ

[![CI](https://github.com/weston-cmyk/optc-project/actions/workflows/ci.yml/badge.svg)](https://github.com/weston-cmyk/optc-project/actions/workflows/ci.yml)

**A governed online-learning substrate that discovers and names structure it
was never trained on — with receipts.** No external models, no cloud, fully
deterministic per seed. Built for streams where the interesting thing is the
thing nobody labeled.

## Install

From a clone of this repo (not yet published to PyPI):

```bash
pip install -e ".[serve,ingest]"   # core + service + universal ingestion
pip install -e .                   # core only (numpy + scikit-learn)
```

Ships with starter pattern packs (`cyber_v5`: 68 MITRE-mapped endpoint
membranes; `unsw_flow_v1`: 9 network-attack families + Normal), so a fresh
install recognizes known-bad structure on day one — and the discovery loop
handles what nobody packed.

## The 60-second demo

```bash
gamma-sigma serve --pack cyber_v5 --port 8420 &
curl -s -X POST localhost:8420/score -H 'content-type: application/json' \
  -d '{"entity_id":"host-17","signals":[0.1,0.1, ... 38 values ...]}'
# → calibrated channels (conformal / decide / z) + a sha256-chained receipt
```

For the flagship hidden-archetype discovery demo (a pattern injected
mid-stream, discovered and named with zero labels):

```bash
git clone https://github.com/weston-cmyk/optc-project && cd optc-project
pip install -e ".[dev]"
python3 examples/synthetic_quickstart.py
```

## The headline evidence (CI-recomputed)

- **Real-data drift arena (UNSW-NB15):** with an attack family held out of all
  training, GAMMA-Σ out-recalls a frozen logistic classifier in **15/18**
  configurations across all 9 attack families — and beats a *retrained*
  oracle on Shellcode (0.97–0.99 vs 0.80). `SWEEP_UNSW_2026-08-12.md`.
- **Calibrated channels:** the conformal-margin channel empirically holds
  FPR 0.003–0.021 at a 5% target, on synthetic streams *and* real flows.
- **Every number regenerates:** `python3 claims.py --check` recomputes the
  advertised results from source. The register is CI-enforced; stale claims
  fail the build.

## Honest limits (also measured)

- It is **not** a drop-in replacement for a tuned classifier on static data —
  logreg wins that fight and we publish that table.
- High recall on novel structure currently comes with high FPR on the hot
  channel; the conformal channel is the actuation-grade gate. See the ROC
  frontier in `FIXES_V45_2026-08-12.md`.
- The mechanism needs *structured* novelty (attack patterns yes; diffuse
  behavioral drift no — measured in `NEW_DOMAINS_2026-08-12.md`).

## Platform surface (2026-08-12)

Installable package (`pip install -e .` from this repo, CLI entry point `gamma-sigma` 0.8.0):

- `gamma_cli.py` — the CLI: `train` (synthetic-corpus trainer → checkpoint),
  `score` (dataset → channel scores CSV), `ingest` (universal adapter run),
  `serve` (the scorer service).
- `gamma_serve.py` — FastAPI scorer with the calibrated cascade
  (conformal authorize / decide respond / z triage), `/membranes` registry,
  a zero-dependency web console at `/`, and sha256 hash-chained decision
  receipts (`/receipts`, `/receipts/verify`).
- `gamma_checkpoint.py` — substrate state save/restore (weights, nulls,
  probation state) for stream resume.
- `gamma_ingest.py` — universal ingestion: any CSV/parquet → auto-featurize →
  auto-seeded membranes → governed loop → calibrated channels. Accepts a
  semantic-map JSON for type-aware featurization (`--semantic-map`).
- `gamma_config.py` — typed config profiles (`research()` = the published
  campaign numbers, `production()` = conservative deployment).
- `GAMMA_MATH.md` — the mathematical frame (governed online dictionary
  learning; EVT/conformal decision layer; coherence control).
- Evidence + arena reports: WIN_BENCHMARK / DRIFT_ARENA / FIXES_V41..V46 /
  CAMPAIGN_V46_V47 / NEW_DOMAINS / SWEEP_UNSW — every claim regenerates.

## Get it / support

- **Core — free, MIT.** Python package (install from this repo; PyPI publication planned), all four pattern packs, the scorer
  service, the CLI, and every evidence report. This repo.
- **Pro deployment kit — paid.** Docker/compose stack, systemd + launchd
  service units, alert-webhook wiring, production runbooks, and priority
  support. Checkout link ships with the first release (see RELEASE.md).

## Repository map

- `gamma_sigma_v3.py` — the substrate engine; `GAMMA_MATH.md` — the frame
- `gamma_packs.py` + `packs/` — versioned starter pattern packs
- `gamma_reflex_bridge.py` — dry-run-first remediation bridge (see INTEGRATIONS.md)
- `optc_adapter.py`, `signal_extractor_v4.py`, `baseline_builder.py`,
  `canary_handler.py` — the OpTC eCAR lane (validated against DARPA GT)
- `trained_weights_v4.json`, `optc_baselines.json` — frozen v4 artifacts
- `OpTCRedTeamGroundTruth.pdf` — DARPA red team ground truth reference
- Evidence: `FINAL_VALIDATION_2026-08-12.md`, `E2E_REPORT.md`,
  `WIN_BENCHMARK_2026-08-12.md`, `DRIFT_ARENA_2026-08-12.md`,
  `FIXES_V41..V46`, `CAMPAIGN_V46_V47`, `NEW_DOMAINS`, `SWEEP_UNSW` —
  every advertised number regenerates via `python3 claims.py --check`.

## Data file (NOT in repo)

The benign eCAR data file is 1.59 GB compressed / 1.25 GB uncompressed,
which exceeds GitHub 100 MB per-file limit. It is not committed; the OpTC
data is publicly available (see the audit report's Addendum 4 for the
Hugging Face mirror used for recovery). Stats from a full scan:
- 25,224,989 events
- 25 hosts (SysClient0051 through 0075, systemia.com)
- Date range: 2019-09-19 to 2019-09-20

## Verified working

End-to-end platform validation (pipeline + 40k-row scale test with mid-stream
service restart and receipt-chain verification across the restart) is in
`E2E_REPORT.md`; the final multi-scenario security validation is in
`FINAL_VALIDATION_2026-08-12.md`. Both are reproducible:
`python3 e2e_pipeline_test.py` / `python3 e2e_scale_test.py`.

## Contributing, security, and release

The repository is still private. These files are staged so a public cut
does not go out without a security policy or a changelog.

- [CONTRIBUTING.md](CONTRIBUTING.md)
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
- [RELEASE.md](RELEASE.md) — do not tag until the package-name gate in
  issue #2 is closed. Tagging `v*` publishes to PyPI.

Install today from this clone (`pip install -e ".[serve,ingest]"`).
`pip install gamma-sigma` only works after the first public tag.
