Metadata-Version: 2.4
Name: parsimony-sdmx
Version: 0.2.0
Summary: SDMX protocol connector (ECB, Eurostat, IMF, OECD, BIS, World Bank, ILO) for the parsimony framework
Project-URL: Homepage, https://parsimony.dev
Project-URL: Repository, https://github.com/ockham-sh/parsimony-connectors
Project-URL: Issues, https://github.com/ockham-sh/parsimony-connectors/issues
Author-email: "Ockham.sh" <team@ockham.sh>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: connectors,data,finance,parsimony,sdmx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=5.0
Requires-Dist: pandas<3,>=2.3.0
Requires-Dist: parsimony-core<0.3,>=0.1.0a0
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow>=16.0
Requires-Dist: pydantic<3,>=2.11.1
Requires-Dist: requests>=2.32
Requires-Dist: sdmx1>=2.16
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.3.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=9.0.3; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Requires-Dist: respx>=0.22.0; extra == 'dev'
Requires-Dist: ruff>=0.15.10; extra == 'dev'
Description-Content-Type: text/markdown

# parsimony-sdmx

Flat SDMX catalog builder. Harvests dataflows from statistical agencies (ECB, Eurostat, IMF, World Bank) and writes two parquet files per agency — a dataset-level catalog and one series-level catalog per dataset — ready to feed a FAISS or SQL index.

## Layout

```
outputs/{AGENCY}/datasets.parquet        # columns: dataset_id, agency_id, title
outputs/{AGENCY}/series/{DATASET}.parquet # columns: id, dataset_id, title
```

Titles are composed as `"CODE1: label1 - CODE2: label2 - …"` over the DSD's non-`TIME_PERIOD` dimensions in DSD order. ECB series additionally append `TITLE` / `TITLE_COMPL` fetched from the per-series XML endpoint. HTML-embedded descriptions (common on Eurostat) are stripped before they reach parquet.

## Install

Requires Python ≥ 3.12 and [uv](https://docs.astral.sh/uv/).

```bash
make install              # uv sync --all-extras
```

## Usage

```bash
# Print every dataset the agency exposes
parsimony-sdmx -a ESTAT --list-datasets

# Write only outputs/{AGENCY}/datasets.parquet (no series fetched)
parsimony-sdmx -a ECB --catalog

# Fetch one dataset
parsimony-sdmx -a ECB -d YC

# Fetch every dataset the agency exposes
parsimony-sdmx -a ESTAT --all

# Preview what an --all run would write, without fetching
parsimony-sdmx -a ESTAT --all --dry-run

# Rebuild datasets whose parquet already exists (resume contract: file present → skip)
parsimony-sdmx -a ECB -d YC --force
```

Or via Make shortcuts:

```bash
make catalog AGENCY=ECB            # datasets.parquet only
make fetch AGENCY=ECB DATASET=YC   # single dataset
make fetch-all AGENCY=ESTAT        # every dataset for the agency
make list AGENCY=IMF_DATA          # enumerate datasets
```

Supported agencies: `ECB`, `ESTAT`, `IMF_DATA`, `WB_WDI`. Exit codes: `0` every dataset ok/empty, `1` at least one failed.

## Architecture

Four packages under `parsimony_sdmx/`:

- **`core/`** — pure, I/O-free: record dataclasses, title composition, codelist resolution, outcome types, domain exceptions.
- **`io/`** — boundary effects: atomic parquet writers, hardened lxml iterparse, HTTPS-only bounded HTTP session, path safety helpers, exception classification.
- **`providers/`** — per-agency adapters behind a narrow `CatalogProvider` protocol; ECB/ESTAT/IMF share a common sdmx1 flow helper, WB diverges with a path × decade sweep because its SDMX endpoint doesn't expose `series_keys`.
- **`cli/`** — argparse front-end, orchestrator that forks one subprocess per dataset (`mp.spawn`) for memory isolation, a psutil-backed memory monitor that kills the largest child above a threshold and writes OOM markers, atomic `.tmp/` cleanup, and an operator-readable end-of-run summary.

### Why subprocess-per-dataset

`sdmx1` caches structure messages at module level with no public invalidation hook. Running every dataset in its own subprocess is the only reliable way to start each fetch with a clean cache. Large catalogs (8 k+ Eurostat dataflows) have hit real OOMs in production; the memory monitor catches runaway workers before the kernel OOM killer does, preserving a classifiable failure instead of an opaque `exit 137`.

### Resume

Filesystem-backed: a dataset is skipped if `outputs/{AGENCY}/series/{DATASET}.parquet` already exists. Writes land in `.tmp/` first and are `os.replace`-d atomically, so the canonical path exists iff the previous run completed. `--force` overrides (and logs a count of overwrites).

## Development

```bash
make check         # ruff + mypy strict + fast tests
make test          # fast tests only (skip subprocess-path tests)
make test-slow     # subprocess tests (fork real mp.Process children)
make test-all      # everything
make format        # ruff format + --fix
make clean         # wipe caches + build artifacts
```

Hardening enforced by default:

- `mypy --strict` across source and tests
- `ruff` with `E, F, W, I, B, UP, S` (security rules on)
- Hardened `lxml.iterparse` (no entity resolution, no DTD load, no network)
- HTTPS-only `bounded_get` with a configurable byte cap
- Path traversal guards on every on-disk write

## Project layout

```
parsimony_sdmx/
├── core/      # pure domain logic
├── io/        # boundary-layer effects
├── providers/ # per-agency adapters
└── cli/       # argparse → orchestrator → worker → parquet

tests/        # flat: test_<module>.py per source module
```

Test suite: 312 tests. 4 of them (`@pytest.mark.slow`) fork real subprocesses to exercise the orchestrator's timeout / OOM classification / clean-exit paths; the other 308 run in < 2 s.
