Metadata-Version: 2.4
Name: nomeda
Version: 0.1.2
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Summary: High-performance Arabic tokenizer with dialect-aware routing
Keywords: arabic,tokenizer,nlp,dialect,rust
License: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/nomeda-lab/nomeda-arabic-tokenizer
Project-URL: Repository, https://github.com/nomeda-lab/nomeda-arabic-tokenizer

# Nomeda Arabic(MSA) Tokenizer

Nomeda is a Rust-powered Arabic tokenizer with Python bindings.

It currently provides a production-ready MSA tokenizer flow based on TSV vocab loading, with convenient pretrained-style loading APIs for Python.

## Install

From PyPI:

```bash
pip install nomeda
```

## Quick Start (Python)

```python
from nomeda import NomedaMSATokenizer

# Option 1: load by pretrained name
tok = NomedaMSATokenizer.from_pretrained("nomeda-msa-64k")

# Option 2: load default name (NOMEDA_DEFAULT_MODEL or nomeda-msa-64k)
# tok = NomedaMSATokenizer.load_default()

result = tok.encode("العلم نور")
print(result.ids)
print(result.tokens)
print(tok.decode(result.ids))
```

If you are using local tokenizer files, set:

```bash
export NOMEDA_MODEL_DIR=/path/to/vocab
```

Direct file loading is also available:

```python
from nomeda import NomedaMSATokenizer

tok = NomedaMSATokenizer.from_tsv("/path/to/Nomeda-MSA-64Kv1.tsv")
```

## Quick Start (Rust)

```rust
use nomeda_core::Tokenizer;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tok = Tokenizer::from_tsv("testdata/test_vocab.tsv")?;
    let result = tok.encode("العلم نور")?;

    println!("ids: {:?}", result.ids);
    println!("tokens: {:?}", result.tokens);
    println!("decoded: {}", tok.decode(&result.ids));
    Ok(())
}
```

## What Works Today

- High-performance MSA tokenization from TSV vocabulary
- Python package `nomeda` (PyO3 bindings)
- `from_pretrained(...)`, `load_default()`, `from_tsv(...)` in Python
- Encode/decode APIs with token IDs and token strings

## Planned / Partial Features

- Dialect router model loading is currently stubbed
- Plugin binary model loading is currently stubbed
- Primary production path today is MSA TSV loading

## Benchmarks

Benchmark snapshots and dataset-level comparisons are published in the Hugging Face model card:

- https://huggingface.co/nomeda-lab/Nomeda-MSA-64Kv1

## Repository Layout

- `nomeda-core/` - Rust tokenizer core
- `nomeda-python/` - Python bindings via PyO3
- `scripts/` - data prep/training utility scripts
- `docs/` - architecture and API notes
- `testdata/` - sample vocabulary for tests and demos

## Development

Run tests:

```bash
cargo test --workspace
```

Build Python extension locally:

```bash
python3 -m pip install -U maturin
python3 -m maturin develop --release
```

Build distribution artifacts:

```bash
python3 -m maturin build --release
```

## Release Automation

GitHub Actions workflow for trusted PyPI publishing is included in:

- `.github/workflows/pypi-publish.yml`

Tag-based publish flow:

```bash
git tag v0.1.2
git push origin v0.1.2
```

## License

MIT

