Metadata-Version: 2.4
Name: nomeda
Version: 0.1.1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Summary: High-performance Arabic tokenizer with dialect-aware routing
Keywords: arabic,tokenizer,nlp,dialect,rust
License: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/nomeda-lab/nomeda-arabic-tokenizer
Project-URL: Repository, https://github.com/nomeda-lab/nomeda-arabic-tokenizer

# Nomeda Arabic Tokenizer

> A high-performance, modular Arabic tokenizer with dialect-aware routing.

[![Rust](https://img.shields.io/badge/rust-%23000000.svg?style=for-the-badge&logo=rust&logoColor=white)](https://www.rust-lang.org)
[![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://www.python.org)

## Vision

Modern Arabic NLP is bottlenecked by tokenizers trained primarily on MSA (Modern Standard Arabic). Dialectal Arabic — which comprises 80%+ of social media, speech, and informal text — is treated as out-of-vocabulary noise. Existing "Arabic" tokenizers collapse Egyptian, Levantine, Gulf, and Maghrebi dialects into a single vocabulary, destroying the very linguistic features that make each dialect distinct.

**Nomeda** solves this with a **router + plugin** architecture:

1. **Base MSA Tokenizer** — rock-solid foundation for formal Arabic, trained on curated literary corpora
2. **Fast Dialect Router** — Rust-based classifier that detects dialect at the sentence level
3. **Dialect Plugins** — swappable tokenizers for Egyptian, Shami, Gulf, Maghrebi, and beyond
4. **Unified API** — one interface, multiple dialects, seamless code-switching handling

## Key Features

- **Blazing Fast**: Rust core with `O(1)` token lookups via trie-based vocabulary
- **Modular by Design**: Add dialects like Lego blocks — train a plugin, drop it in, done
- **Code-Switching Aware**: Sentence-level routing handles MSA + dialect mixtures
- **MSA Preservation**: Base tokenizer trained exclusively on high-quality MSA (Hindawi corpus + Wikipedia + Quran)
- **Python Native**: First-class Python bindings via PyO3 for seamless ML integration
- **Streaming Ready**: Memory-mapped plugin files, zero-copy batch encoding

## Architecture at a Glance

```
Input Text
    │
    ▼
┌─────────────────────────────────────────┐
│  Sentence Segmenter                     │
│  (splits on punctuation / newlines)     │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│  Dialect Router (Rust, <1ms/sentence)   │
│  • MSA confidence score                 │
│  • Dialect probability distribution       │
└─────────────────────────────────────────┘
    │
    ├── confidence < 0.7 ──► Base MSA Tokenizer
    │
    ├── Egyptian ──► Egyptian Plugin
    ├── Shami ─────► Levantine Plugin
    ├── Gulf ──────► Gulf Plugin
    ├── Maghrebi ──► Maghrebi Plugin
    └── [user plugin] ──► Custom Plugin
    │
    ▼
┌─────────────────────────────────────────┐
│  Token IDs (unified space across all    │
│  plugins via shared base vocabulary)    │
└─────────────────────────────────────────┘
```

## Quick Start

### Installation

```bash
pip install nomeda-tokenizer
```

### Basic Usage

```python
from nomeda_tokenizer import Tokenizer

# Load base MSA tokenizer
tok = Tokenizer.load("msa-base-v1")

# Encode text
tokens = tok.encode("العلم نور والجهل ظلام")
print(tokens.ids)        # [101, 2345, 892, 102]
print(tokens.tokens)     # ["[BOS]", "العلم", "نور", "[EOS]"]

# Decode back
text = tok.decode(tokens.ids)
print(text)              # "العلم نور"
```

### With Dialect Routing

```python
from nomeda_tokenizer import Tokenizer, Router

# Load router + all available plugins
tok = Tokenizer.load("msa-base-v1", plugins=["egy", "shami", "gulf", "magh"])

# Automatic routing
tokens = tok.encode("يا عم إحنا رايحين فين؟")  # Egyptian dialect
print(tokens.dialect)    # "egy"
print(tokens.tokens)     # ["[BOS]", "يا", "عم", "إحنا", "رايحين", "فين", "؟", "[EOS]"]

# Force a specific dialect
tokens = tok.encode("شو بدك تعمل؟", dialect="shami")
print(tokens.dialect)    # "shami"
```

### Adding a Custom Plugin

```python
# Train your own dialect tokenizer on your corpus
from nomeda_tokenizer.training import train_plugin

train_plugin(
    corpus_path="my_dialect_corpus.txt",
    base_vocab="msa-base-v1",
    output_path="my_dialect.plugin",
    vocab_size=16000,
)

# Load it
tok.load_plugin("my_dialect.plugin", name="mydia")
tokens = tok.encode("...", dialect="mydia")
```

## Project Structure

```
.
├── docs/                    # Architecture docs and RFCs
│   ├── ARCHITECTURE.md
│   ├── PLUGIN_SYSTEM.md
│   ├── ROADMAP.md
│   └── API.md
├── rust/                    # Rust core library
│   ├── src/
│   │   ├── lib.rs          # Main crate
│   │   ├── tokenizer.rs    # Core encode/decode engine
│   │   ├── router.rs       # Dialect classification
│   │   ├── vocab.rs        # Vocabulary trie + plugin loader
│   │   └── normalize.rs    # Arabic text normalization
│   ├── benches/            # Criterion benchmarks
│   └── examples/
├── python/                  # Python bindings (PyO3)
│   └── nomeda_tokenizer/
├── plugins/                 # Pre-trained plugin releases
│   ├── egy.plugin
│   ├── shami.plugin
│   ├── gulf.plugin
│   └── magh.plugin
├── training/                # Training scripts
│   ├── train_base.py        # Train MSA base tokenizer
│   ├── train_plugin.py      # Train dialect plugin
│   └── prepare_corpus.py    # Corpus preprocessing
├── corpus/                  # Training corpora
│   └── hindawi/             # Extracted & cleaned Hindawi books
└── scripts/                 # Utility scripts
```

## Performance

| Metric | Value |
|---|---|
| Encode throughput | ~2M tokens/sec (single thread, Rust) |
| Plugin switch latency | ~50 ns |
| Router latency | <1 ms per sentence |
| Memory per plugin | ~8-16 MB (vocab + scores) |
| Base vocab size | 50,000 tokens |
| Plugin vocab size | 16,000 tokens each |

## Supported Dialects

| Plugin | Region | Cities | Status |
|---|---|---|---|
| `egy` | Egyptian / Nile Basin | Cairo, Alexandria, Sudan | Planned |
| `shami` | Levantine | Damascus, Beirut, Amman, Jerusalem | Planned |
| `gulf` | Gulf | Riyadh, Jeddah, Doha, Muscat | Planned |
| `magh` | Maghrebi | Rabat, Algiers, Tunis, Fes | Planned |
| `iraq` | Iraqi | Baghdad, Basra, Mosul | Planned |

## Training Corpora

### Base MSA Tokenizer
- **Hindawi Arabic Books**: 3,280 books, 26 categories, ~800M chars
- **Wikipedia Arabic**: dump processed
- **Quran**: classical Arabic baseline

### Dialect Plugins (Planned)
- **MADAR Corpus**: 25 city dialects, parallel sentences
- **NADI Dataset**: tweet-level dialect labels
- **AOC (Arabic Online Commentary)**: dialectal text
- **Common Crawl filtered**: by dialect identification

## Contributing

See [ROADMAP.md](docs/ROADMAP.md) for current development priorities and [ARCHITECTURE.md](docs/ARCHITECTURE.md) for technical deep-dives.

## License

MIT License — see LICENSE file.

## Acknowledgments

- Hindawi Foundation for the incredible Arabic digital library
- CAMeL Lab for MADAR corpus and dialect identification research
- HuggingFace `tokenizers` crate for design inspiration

