Metadata-Version: 2.2
Name: tokmor
Version: 1.4.0
Summary: Dependency-free deterministic tokenizer and morphology hints for 300+ configured languages
Author-email: Zeus Kim <zeus@zeus.kim>
License: MIT
Project-URL: Homepage, https://github.com/zeus-kim/tokmor
Project-URL: Documentation, https://github.com/zeus-kim/tokmor#readme
Project-URL: Source, https://github.com/zeus-kim/tokmor
Project-URL: Issues, https://github.com/zeus-kim/tokmor/issues
Keywords: nlp,preprocessing,multilingual,tokenizer,tokenization,segmentation,morphology,lemmatization,ner,rag,information-extraction,offline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: api
Requires-Dist: fastapi>=0.100; extra == "api"
Requires-Dist: uvicorn>=0.23; extra == "api"

# TokMor

**Dependency-free deterministic tokenizer and morphology hints for multilingual text.**

[![PyPI](https://img.shields.io/pypi/v/tokmor.svg)](https://pypi.org/project/tokmor/)
[![Python](https://img.shields.io/pypi/pyversions/tokmor.svg)](https://pypi.org/project/tokmor/)
[![License: MIT](https://img.shields.io/pypi/l/tokmor.svg)](https://pypi.org/project/tokmor/)

TokMor is a compact preprocessing library for tokenization, offsets, language-aware segmentation, and rule-based morphology hints. It is built for offline product use: deterministic output, no model downloads, and no runtime dependencies in the core package.

TokMor is not a Transformer model, POS tagger, NER classifier, crawler, dashboard, or news-analysis pipeline. Downstream systems can use TokMor spans and morphology hints before entity recognition, search, RAG, or analytics.

## Current Scope

- 300+ configured language codes with deterministic fallback tokenization.
- 121-language product contract gate for representative real-world text patterns.
- Locked morphology regression suite: 11,930 / 11,930 cases at the latest local gate.
- Focused no-space language support for Korean, Japanese, Chinese, Thai, and related CJK/SEA scripts.
- Stable handling for offsets, punctuation, dates, URLs, versions, units, abbreviations, and common clitic/MWT patterns.
- CLI, Python API, optional FastAPI server, and stdio MCP server.

Language quality is tiered. Some languages have tuned segmentation and morphology packs; others intentionally use conservative deterministic fallback rules. See [docs/LANGUAGE_TIERS.md](https://github.com/zeus-kim/tokmor/blob/main/docs/LANGUAGE_TIERS.md) and [docs/COMMERCIAL_MVP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/COMMERCIAL_MVP.md).

## Install

```bash
pip install tokmor
```

The core wheel includes compact starter dictionaries for CJK/SEA segmentation,
frequent-word POS hints, MWT/clitic splits, lemma manifests, and small domain
hint assets. No separate dictionary download is required for default local use.
Large lemma packs remain optional offline data packs.

Run `tokmor assets --pretty` after installation to inspect the bundled dictionary inventory.

From a checkout:

```bash
python3 -m pip install -e ".[dev]"
```

## Python

```python
import tokmor

out = tokmor.unified_tokenize("우리는 2025-01-10에 서울을 방문했다.", lang="ko", sns=False)
print([t["text"] for t in out["tokens"]])

prep = tokmor.ner_preprocess("東京の新製品を25℃でテストした。", lang="ja")
print(prep[:3])
```

Token dictionaries include `text`, `start`, and `end`; offsets slice back to the original string.

## CLI

```bash
tokmor tokenize --lang ko --text "로마로부터 정명석의" --text-output
tokmor analyze --lang ja --text "東京の新製品" --pretty
tokmor ner-preprocess --lang zh --text "我们在2025-01-10访问了首尔。" --pretty
tokmor explain --lang auto --text "TokMor checks bundled assets." --pretty
tokmor explain-document --lang auto --text "TokMor checks bundled assets. It keeps offsets stable." --summary-only --pretty
tokmor assets --pretty
```

If the console script is not on `PATH`, use the equivalent module form: `python -m tokmor tokenize ...`.

See [docs/CLI.md](https://github.com/zeus-kim/tokmor/blob/main/docs/CLI.md).

## MCP and API

Run the MCP server for AI hosts:

```bash
tokmor-mcp
```

Run the optional HTTP API after installing the `api` extra:

```bash
python3 -m pip install "tokmor[api]"
tokmor-api --host 127.0.0.1 --port 8000
```

Contract details are in [docs/MCP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/MCP.md) and [docs/TOKMOR_API_CONTRACT.md](https://github.com/zeus-kim/tokmor/blob/main/docs/TOKMOR_API_CONTRACT.md).

## Verify a Checkout

Fast product checks:

```bash
PYTHONPATH=TokMor_v1 python3 examples/quickstart_tier1.py
PYTHONPATH=TokMor_v1 python3 scripts/run_token_invariants_smoke.py
PYTHONPATH=TokMor_v1 python3 scripts/run_multilang_product_contract.py
PYTHONPATH=TokMor_v1 python3 scripts/run_morph_regress.py --langs en,ko,ja,zh,th,ar,hi
PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history
```

Full release gate:

```bash
python3 scripts/run_productization_checks.py \
  --python python3 \
  --with-pos-eval \
  --with-bench \
  --with-package \
  --bench-iters 300
```

## Repository Hygiene

This repository is kept to TokMor core runtime, product contracts, regression fixtures, packaging, and product documentation. Non-core dashboards, news-analysis apps, generated build outputs, local data, and session artifacts are intentionally excluded. Run the scope audit before release:

```bash
PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history --json
```

The expected Git author/committer identity for this repository is `Zeus Kim <zeus@zeus.kim>`.

## Docs

- [docs/COMMERCIAL_MVP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/COMMERCIAL_MVP.md): conservative product positioning and limits.
- [docs/AUTODEV_AUTOMATION.md](https://github.com/zeus-kim/tokmor/blob/main/docs/AUTODEV_AUTOMATION.md): automated d2 development loop.
- [docs/NEXT_DEVELOPMENT_PLAN.md](https://github.com/zeus-kim/tokmor/blob/main/docs/NEXT_DEVELOPMENT_PLAN.md): parallel development tracks and gates.
- [docs/LANGUAGE_TIERS.md](https://github.com/zeus-kim/tokmor/blob/main/docs/LANGUAGE_TIERS.md): language support model.
- [docs/TOKMOR_API_CONTRACT.md](https://github.com/zeus-kim/tokmor/blob/main/docs/TOKMOR_API_CONTRACT.md): Python/API/MCP contract expectations.
- [docs/AUTONER_TOKMOR_HANDOFF.md](https://github.com/zeus-kim/tokmor/blob/main/docs/AUTONER_TOKMOR_HANDOFF.md): downstream AutoNER integration notes.
- [docs/DATA_SOURCES_AND_LICENSES.md](https://github.com/zeus-kim/tokmor/blob/main/docs/DATA_SOURCES_AND_LICENSES.md): data source and license notes.
- [docs/FAQ.md](https://github.com/zeus-kim/tokmor/blob/main/docs/FAQ.md): what TokMor is and is not.

## License

MIT. See [LICENSE](LICENSE).
