Metadata-Version: 2.4
Name: medwer
Version: 0.1.0
Summary: Reproducible, model-free medical-ASR evaluation: protocol-v1 normalizer + phrase/acronym medical-WER + curated term list, parity-locked C++/Python.
License: MIT
Project-URL: Repository, https://github.com/Nordis-Tech/medwer
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: whisper-normalizer==0.1.12
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# MedWER

MedWER is a **reproducible, model-free evaluation protocol for medical ASR**: a
deterministic text normalizer, a phrase/acronym-aware **medical-WER**, and a curated,
license-clean medical term list — implemented twice (Python and C++) and **parity-locked
byte-for-byte** by golden fixtures that both implementations must reproduce in CI.

It is a protocol and a tool, not a new metric concept. Entity-weighted WER variants are
well studied; what they typically depend on is an eval-time NER model or cloud API,
which makes the denominator irreproducible. MedWER's denominator is a **fixed, typed,
curated term list** — the same strings for everyone, forever, no model in the loop.

## Why

- **Deterministic.** Same inputs, same number, on any machine. No NER model, no API,
  no versioned black box in the metric.
- **Parity-locked.** The Python package and the C++ library produce byte-identical
  normalization and identical scores, enforced by shared golden fixtures (pytest +
  ctest). You can benchmark in Python and deploy the identical scorer in C++.
- **License-clean.** Every term-list source permits open commercial redistribution
  (see [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE)); the list ships in the wheel.
- **Phrase/acronym-honest.** Multi-word terms ("serratia marcescens") and spelled
  acronyms ("a s t") are matched greedily longest-first and scored as single atomic
  units, so a missed phrase counts as one miss, not free partial credit.

## Install

```
pip install medwer
```

## CLI

```
medwer score --refs refs.jsonl --hyps hyps.jsonl [--terms path/to/terms.txt]
```

Both files: one JSON object per line, `{"id": ..., "text": ...}`. Hypotheses join to
references by `id`; a reference id missing from `--hyps` is an error. Output is a
single JSON object (rates are fractions):

```
{"utts": 2620, "ref_words": 52576, "med_ref_words": 4310, "wer": 0.042, "medical_wer": 0.31}
```

## Python API

```python
from medwer import Normalizer, Scorer

scorer = Scorer()                      # bundled 20,653-entry medical term list
result = scorer.corpus(refs, hyps)     # {"wer": ..., "medical_wer": ..., ...}

norm = Normalizer()                    # protocol v1 (no number folding)
norm("The patient won't stop coughing.")   # -> "the patient will not stop coughing"
```

## C++ library

`cpp/` contains the mirror implementation (`medwer::Normalizer`,
`medwer::compute_wer`, `medwer::compute_medical_wer`) with utf8proc and
nlohmann/json as its only dependencies:

```
cmake -S cpp -B build && cmake --build build -j && ctest --test-dir build
```

## The protocol (v1)

1. **Normalize** both sides with Whisper's `EnglishTextNormalizer` **minus** the
   number-word folding step (references are expected in spoken form; digit/word
   mismatches are scored, not forgiven). `Normalizer(fold_numbers=True)` gives the
   stock behavior for literature-comparable WER on standard benchmarks.
2. **WER**: token-level Levenshtein over whitespace tokens.
3. **Medical-WER**: filter both sides to term-list content (greedy longest-match,
   phrases atomic), then WER on what remains.
4. **Corpus aggregation**: micro-average (total edits / total reference words);
   samples whose normalized reference is empty are skipped.

Parity scope: byte-parity is guaranteed for Latin-script input (ASCII, Latin
accents, NFKD-compatibility forms). Full-Unicode `str.lower()` corner cases (CJK,
Greek final sigma) are outside the medical-Latin domain and not guaranteed.

## The term list

`medwer/data/medical_terms.txt` — 20,653 entries (12,604 multi-word) across drugs,
diagnoses, symptoms, mechanisms, organisms, and labs. Sources: Health Canada Drug
Product Database + Canadian Clinical Drug Data Set (Open Government Licence –
Canada), ICD-10-CM (US public domain), and curated organism/lab lists (MIT). Bare
strings only — no confusability relationships or third-party selection data. Full
provenance in [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE).

## Regenerating fixtures

Only needed when the pinned `whisper-normalizer` version is bumped or the fixture
corpora change:

```
python tools/gen_fixtures.py
```

## License

MIT (code). Term-list data: see [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE) for per-source terms.
