Metadata-Version: 2.4
Name: zopen
Version: 0.3.5
Summary: Portable, autodetecting byte-level BPE tokenizer with architecture-specialized native acceleration
Author: ZOpen contributors
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: C
Classifier: Operating System :: POSIX
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"

# ZOpen 0.3.3

## 0.3.3 performance pass

- Exact SIMD byte-pair identity probe using AVX-512/AVX2 gather.
- AVX-512 byte-token materialization on decoder.
- Portable fallback retained for ARM/older x86.
- Large-input identity detection is now exact across the full buffer.

# ZOpen 0.3.2 — Faster BPE core, scheduling, SIMD and training

> **English + Español**

ZOpen is a native, byte-level BPE tokenizer designed around a simple idea: keep the algorithm deterministic and portable, then specialize the hot loops at build and runtime.

It supports Python 3.9+, a C11 native core, runtime SIMD dispatch, adaptive microkernel thresholds, packed `uint32` APIs, batch encoding, file streaming, and tokenizer training/loading.

---

### What is new in 0.3.2

- Byte-weighted batch scheduling for heterogeneous document sizes.
- Byte-weighted parallel training initialization.
- AVX-512 lookup for the training pair hash table when available.
- Shared SIMD byte→`uint32` microkernel for encoder identity paths.
- Existing AVX2/NEON decoder materialization remains enabled.
- Deterministic rank/FIFO merge queues are preserved.
- New focused benchmark: `scripts/benchmark_0_3_2.py`.

## English

### What is new in 0.3.2

ZOpen 0.3.3 focuses on **portable installation + aggressive specialization without CPU lock-in**.

- Runtime CPU dispatch for **AVX-512F, AVX2, SSE2, ARM NEON, and optional SVE2** where the compiler/CPU exposes them.
- Install-time microbenchmarks choose useful size ranges instead of blindly enabling one SIMD path for everything.
- `setup.py` no longer selects `-march=native`, so a build is not silently tied to the installation machine.
- Architecture normalization covers `x86`, `x86_64`, `arm`, `aarch64/arm64`, and common aliases.
- LTO is tested before being enabled and can be forced with `ZOPEN_LTO=0`.
- `ZOPEN_SKIP_PROBE=1` is available for constrained/cross-build environments.
- A diagnostic `zopen.capabilities()` API exposes the compiled architecture and detected SIMD backends.
- Strong API and round-trip tests cover empty input, binary bytes, Unicode, boundary lengths, and batches.
- `scripts/verify_install.py` provides a post-install smoke/strong test.
- `scripts/build_matrix.py` compares GCC/Clang and LTO choices on the current host.
- The project now ships a much more explicit bilingual README and packaging metadata.

### Design principle: portable baseline + specialized paths

ZOpen intentionally does **not** put `-march=native` on the extension. Instead, the source contains target-specific kernels and the running CPU selects them dynamically.

That matters for wheels and reproducible source builds:

```text
Python API
   │
   ▼
Native C core
   │
   ├── scalar baseline
   ├── SSE2 (x86 fallback)
   ├── AVX2
   ├── AVX-512F
   └── NEON (ARM/AArch64)
          │
          ▼
install-time thresholds + runtime CPU dispatch
```

The benchmark performed during installation is a **local tuner**. It measures the host that is actually compiling the package. It is not an emulator for another architecture.

### Dense microkernel tuner in 0.3.3

The new `scripts/benchmark_microkernels_dense.py` is the high-resolution tuner for development and CPU-specific releases. Instead of probing only a handful of representative points, it tests thousands of concrete buffer sizes and directly compares the individual kernels:

```text
scalar
SSE2
AVX2
AVX2-mask
AVX-512F
AVX-512F-mask
```

It records `MB/s`/time-per-element equivalents, selects only victories that remain stable across a local neighborhood, and emits up to eight non-overlapping size intervals per kernel family. The generated dispatcher therefore applies the measured winner only where it was actually demonstrated to win.

For compiler/build comparisons, `scripts/benchmark_microkernels_matrix.py` repeats the same experiment for `portable`, `native`, `LTO`, and `native+LTO`. The dense tuner is intentionally opt-in for package builds because a complete thousands-of-sizes sweep is much more expensive than the normal installation probe:

```bash
ZOPEN_DENSE_AUTOTUNE=1 python -m pip install .

python scripts/benchmark_microkernels_dense.py --max-bytes 8388608 --write-config
python scripts/benchmark_microkernels_matrix.py --max-bytes 8388608
```

The important output is `build/dense-microkernel-benchmark/results.json`, plus `results.csv` for analysis.

### Why the install probe exists

Vector instructions do not win at every input length. A tiny buffer can be faster with scalar code because dispatch and setup cost dominate. A large buffer can favor a wider vector path. ZOpen therefore probes representative sizes and generates a small `build/zopen_mk_config.h` with selected intervals.

The generated file is build output, not source API.

### Supported architecture strategy

| Architecture | Baseline | Specialized paths | Notes |
|---|---|---|---|
| x86_64 | scalar | SSE2, AVX2, AVX-512F | runtime detection via compiler CPU feature API |
| x86 / i386 | scalar | SSE2 where supported | avoids assuming AVX on 32-bit x86 |
| AArch64 / arm64 | scalar | NEON, optional SVE2 | Covers Apple Silicon, Qualcomm/Snapdragon, Ampere, AWS Graviton, and other ARM64 CPUs |
| ARM 32-bit | scalar | NEON when compiler exposes it | depends on target/toolchain |
| POWER64 | scalar | architecture-safe fallback | ready for future VSX kernel selection |
| RISC-V | scalar | architecture-safe fallback | ready for future RVV kernel selection |
| Other | scalar | none | remains functional rather than failing at build time |

### CPU brands and families

ZOpen treats **instruction-set support** and **CPU brand** as separate concerns. Intel and AMD CPUs share the x86 SIMD family, so the runtime selects AVX2/AVX-512/SSE2 paths according to actual CPU features and the local benchmark rather than assuming that every Intel or AMD chip behaves the same.

On ARM64, the same design covers Apple Silicon, Qualcomm Snapdragon, Ampere, AWS Graviton and other AArch64 systems through NEON, with an optional SVE2 path when the compiler headers and runtime HWCAP support are available. The diagnostics API reports both the architecture and detected CPU family when the compiler can identify it.

This means a Snapdragon does **not** need a Snapdragon-specific binary: it gets an ARM64/NEON baseline plus a local native build when you call `zopen.activate(march_native=True, cpu=True)`. Likewise, AMD Zen and Intel Core/Xeon families share the x86 dispatcher while still getting CPU-family diagnostics and per-size tuning.

### Installation

Standard source install:

```bash
python -m pip install .
```

Build a wheel:

```bash
python -m pip wheel . --no-deps -w dist
```

For a constrained build environment where executing the probe is not possible:

```bash
ZOPEN_SKIP_PROBE=1 python -m pip install .
```

For a cross-build where the target CPU cannot execute on the build host, set the target explicitly; the probe is skipped unless `ZOPEN_FORCE_PROBE=1` is also set:

```bash
ZOPEN_TARGET_ARCH=aarch64 python -m pip wheel . --no-deps -w dist
```

Disable LTO:

```bash
ZOPEN_LTO=0 python -m pip install .
```

Select a compiler explicitly:

```bash
CC=clang python -m pip install .
```

The build never adds `-march=native` automatically.

### Optional local `-march=native` build

For source checkouts, ZOpen can deliberately rebuild the native extension for the CPU that is actually compiling it. This is **off by default**. It is not a universal wheel setting: it produces a machine-specific local extension.

```python
import zopen

result = zopen.activate(march_native=True, cpu=True)
print(result)
```

Equivalent helper through the convenience class:

```python
from zopen import ZOpen

ZOpen.activate(march_native=True, cpu=True)
```

The activation sets `ZOPEN_MARCH_NATIVE=1` and recompiles with:

```text
-march=native -mtune=native
```

`cpu=False` disables that tuning and rebuilds the portable baseline:

```python
zopen.activate(march_native=False, cpu=False)
# or
zopen.deactivate()
```

The current Python process should be restarted after recompilation so the new `.so` is loaded. With `pip install -e .`, this workflow is convenient for iterative CPU-specific tuning.

### Verify the installed backend

```bash
python scripts/verify_install.py
python scripts/verify_install.py --strong
```

Or directly from Python:

```python
import zopen

print(zopen.__version__)
print(zopen.capabilities())
```

A typical x86_64 host may report something like:

```python
{
    "compiled_arch": "x86_64",
    "cpu_count": 3,
    "microkernels": {
        "avx2": True,
        "avx512f": True,
        "sse2": True,
        "neon": False,
    },
}
```

The exact result depends on the machine where ZOpen is installed.

### Python API

Create a tokenizer:

```python
import zopen

tok = zopen.Tokenizer(
    vocab_size=32000,
    workers=0,        # automatic worker count
)
```

Encode/decode:

```python
ids = tok.encode("Hello from ZOpen")
text = tok.decode_text(ids)
assert text == "Hello from ZOpen"
```

Batch:

```python
batch = tok.encode_batch([
    "first document",
    "second document",
    "第三篇文档",
])
```

Packed native `uint32` output:

```python
packed = tok.encode_u32("high throughput")
restored = tok.decode_u32(packed)
```

File helpers:

```python
ids = tok.encode_file("input.txt")
tok.encode_file_u32_to("input.txt", "tokens.u32")
tok.encode_file_varint("input.txt", "tokens.varint")
```

Train and load:

```python
model = zopen.train([
    "one training document",
    "another training document",
], vocab_size=4096)

model.save("model.zopen")
loaded = zopen.load("model.zopen")
```

Diagnostic information:

```python
info = zopen.capabilities()
```

### Microkernel philosophy

ZOpen currently has specialized kernels for the checks that are cheap to vectorize and frequent enough to matter:

- token-ID range validation;
- byte-token validation (`< 256`);
- `uint32` zeroing for large workspaces;
- SIMD alternatives for x86 and ARM where practical;
- size-dependent interval selection generated by the build probe.

The important part is not maximizing the number of kernels. It is selecting **the right kernel for the input size and CPU**.

### Strong testing

Run the package tests:

```bash
python -m pytest -q
```

Run the post-install verifier:

```bash
python scripts/verify_install.py --strong
```

Run the host-local build matrix:

```bash
python scripts/build_matrix.py
python scripts/build_matrix.py --with-probe
```

The matrix intentionally tests compiler/build combinations rather than pretending to emulate ARM64 from x86_64.

### Performance work that remains possible

The architecture is ready for additional specialized kernels such as faster UTF-8 classification, ASCII-run detection, merge-table prefetch helpers, and architecture-specific hash probes. These should only be added when a benchmark demonstrates a real win and correctness tests cover the new path.

A fast tokenizer is not just about SIMD width. It is also about allocation pressure, branch behavior, cache locality, merge-table density, worker scheduling, and avoiding work entirely when an identity proof is available.

### Packaging policy

ZOpen uses `pyproject.toml` + setuptools. The build backend remains conventional so that `pip install .` and wheel builds work without a custom installer.

The installer may benchmark the local host, but the extension remains portable within its architecture family because CPU-specific instructions are isolated behind target-specific functions and runtime dispatch.

### License

MIT.

---

## Español

### Compilación local opcional con `-march=native`

ZOpen también permite pedir explícitamente una recompilación optimizada para el CPU de la máquina donde se ejecuta. No se activa automáticamente.

```python
import zopen
zopen.activate(march_native=True, cpu=True)
```

Esto compila localmente con `-march=native -mtune=native`. Para volver al binario portable:

```python
zopen.deactivate()
```

Después de recompilar hay que reiniciar el proceso de Python para cargar la nueva extensión nativa.


### ¿Qué es ZOpen?

ZOpen es un tokenizador BPE por bytes con núcleo nativo en C. La idea central es mantener el algoritmo determinista y portable, y después especializar los bucles calientes según la arquitectura y el CPU reales.

Está pensado para Python 3.9+, C11, dispatch SIMD en tiempo de ejecución, tuning de microkernels durante la instalación, APIs compactas con `uint32`, procesamiento por lotes, archivos y entrenamiento/carga de tokenizadores.

### ¿Qué mejora en 0.3.3?

Esta versión se enfoca en **instalación portable + rutas especializadas sin amarrar el binario a una sola máquina**.

- Detección y dispatch de **AVX-512F, AVX2, SSE2 y ARM NEON** cuando están disponibles.
- El instalador mide diferentes tamaños de entrada y selecciona intervalos útiles en lugar de usar SIMD indiscriminadamente.
- `setup.py` ya no añade `-march=native`, así que una instalación no queda ligada accidentalmente al CPU donde fue compilada.
- Se normalizan `x86`, `x86_64`, `arm`, `aarch64/arm64` y alias comunes.
- LTO se prueba antes de activarse y se puede desactivar con `ZOPEN_LTO=0`.
- `ZOPEN_SKIP_PROBE=1` permite instalar en entornos donde no se puede ejecutar el probe durante el build.
- `zopen.capabilities()` permite consultar arquitectura compilada y microkernels disponibles.
- Hay pruebas fuertes de API y round-trip para entradas vacías, bytes binarios, Unicode, longitudes límite y batches.
- `scripts/verify_install.py` comprueba una instalación real.
- `scripts/build_matrix.py` compara GCC/Clang y LTO en el host actual.
- README bilingüe y metadata de empaquetado mejorados.

### Filosofía: baseline portable + rutas especializadas

No queremos resolver portabilidad metiendo `-march=native` en todo el proyecto. En su lugar, ZOpen contiene kernels especializados y el CPU que ejecuta el programa decide cuál usar.

```text
API Python
   │
   ▼
Núcleo C nativo
   │
   ├── scalar baseline
   ├── SSE2 (fallback x86)
   ├── AVX2
   ├── AVX-512F
   └── NEON (ARM/AArch64)
          │
          ▼
tunable durante instalación + dispatch en runtime
```

El benchmark de instalación es un **tuner local**: mide la máquina que está compilando. No pretende simular ARM64 desde x86_64.

### Arquitecturas

| Arquitectura | Baseline | Rutas especializadas | Comentario |
|---|---|---|---|
| x86_64 | scalar | SSE2, AVX2, AVX-512F | detección en runtime |
| x86 / i386 | scalar | SSE2 cuando existe | evita asumir AVX |
| AArch64 / arm64 | scalar | NEON | NEON es parte normal de AArch64 |
| ARM 32-bit | scalar | NEON si el compilador la expone | depende del target/toolchain |
| Otra | scalar | ninguna | sigue funcionando con ruta segura |

### Instalación

```bash
python -m pip install .
```

Wheel:

```bash
python -m pip wheel . --no-deps -w dist
```

Sin probe durante el build:

```bash
ZOPEN_SKIP_PROBE=1 python -m pip install .
```

Para cross-builds donde el CPU objetivo no puede ejecutar en el host de compilación, se puede indicar el target; el probe se omite salvo que se fuerce:

```bash
ZOPEN_TARGET_ARCH=aarch64 python -m pip wheel . --no-deps -w dist
```

Sin LTO:

```bash
ZOPEN_LTO=0 python -m pip install .
```

Con Clang:

```bash
CC=clang python -m pip install .
```

El build no utiliza `-march=native` automáticamente.

### Ver qué ruta quedó activa

```python
import zopen

print(zopen.__version__)
print(zopen.capabilities())
```

En este entorno concreto, por ejemplo, el host reportó un **Intel Xeon x86_64** con **AVX2 y AVX-512F**. El resultado exacto cambia según la máquina donde se haga `pip install`.

### Uso básico

```python
import zopen

tok = zopen.Tokenizer(vocab_size=32000, workers=0)

ids = tok.encode("Hola desde ZOpen")
text = tok.decode_text(ids)
assert text == "Hola desde ZOpen"
```

Batch:

```python
batch = tok.encode_batch([
    "primer documento",
    "segundo documento",
    "第三篇文档",
])
```

Entrenamiento:

```python
model = zopen.train(
    ["primer texto", "otro texto"],
    vocab_size=4096,
)
```

### ¿Qué microkernels queremos seguir agregando?

No conviene agregar kernels por cantidad. Conviene agregarlos cuando exista un hotspot real y una prueba que demuestre la mejora.

Caminos razonables para siguientes versiones:

- clasificación rápida de UTF-8;
- detección de runs ASCII;
- probing de tablas de merges con mejores patrones de cache;
- prefetch específico de hash tables;
- kernels adicionales para ARM SVE cuando haya soporte de build/runtime bien definido;
- kernels más especializados para tamaños diminutos, medios y enormes.

El objetivo es que el tokenizador sea rápido no solo por tener SIMD, sino por **evitar trabajo**, reducir allocaciones, mejorar localidad de caché, minimizar branches y elegir correctamente cada algoritmo.

### Pruebas

```bash
python -m pytest -q
python scripts/verify_install.py --strong
python scripts/build_matrix.py
```

La matriz compara combinaciones reales del host. Con `--with-probe` además ejecuta el tuning local; no falsifica benchmarks de una arquitectura que la máquina no puede ejecutar.

### Empaquetado

ZOpen usa `pyproject.toml` y setuptools. El objetivo es que `pip install .` sea normal, reproducible y fácil de diagnosticar, manteniendo el núcleo nativo especializado.

### Licencia

MIT.

## Install-time self-calibration / Auto-calibración durante la instalación

### English

ZOpen 0.3.3 can automatically calibrate the local native build. During a source build, ZOpen compares a small set of safe compiler configurations and the generated microkernel dispatch configuration. Each candidate is exercised with deterministic random and repeated data at multiple sizes across **encode, decode, and training**. A candidate is eligible to win only when its correctness and stability checks pass. The winner is the fastest stable configuration under the composite score and is baked into the extension being installed.

The calibration result is cached per machine/compiler identity, so subsequent rebuilds can reuse the winner without rerunning the full tournament. Use `ZOPEN_RECALIBRATE=1` to force a fresh calibration. Use `ZOPEN_AUTOTUNE=0` for a conventional build without the tournament.

Useful controls:

```bash
ZOPEN_AUTOTUNE=1 pip install .
ZOPEN_RECALIBRATE=1 pip install .
ZOPEN_AUTOTUNE=0 pip install .
```

The installed build records the winner in `build/zopen_calibration.json` and `build/zopen_build_manifest.json` when building from source. The composite score weights encode/decode more heavily while still including training. A failed or unstable candidate is automatically rejected.

### Español

ZOpen 0.3.3 puede calibrar automáticamente la compilación nativa de la máquina. Durante una compilación desde código fuente, ZOpen compara varias configuraciones seguras del compilador junto con la configuración generada de microkernels. Cada candidata se prueba con datos deterministas, aleatorios y repetidos, en varios tamaños y en **encode, decode y training**. Una candidata solo puede ganar si supera las pruebas de corrección y estabilidad. La ganadora es la configuración estable más rápida según la métrica compuesta y queda integrada directamente en la extensión instalada.

El resultado se guarda en caché según la identidad de la máquina y del compilador, de modo que las siguientes compilaciones pueden reutilizar la ganadora sin repetir todo el torneo. Usa `ZOPEN_RECALIBRATE=1` para forzar una nueva calibración. Usa `ZOPEN_AUTOTUNE=0` para desactivar el torneo y hacer una compilación convencional.

Controles útiles:

```bash
ZOPEN_AUTOTUNE=1 pip install .
ZOPEN_RECALIBRATE=1 pip install .
ZOPEN_AUTOTUNE=0 pip install .
```

Cuando se compila desde código fuente, la selección queda registrada en `build/zopen_calibration.json` y `build/zopen_build_manifest.json`. La métrica compuesta da más peso a encode/decode, pero también incluye training. Cualquier candidata inestable o incorrecta queda descartada automáticamente.

LTO is retained in 0.3.3 and is validated against the real extension sources and a Python import before activation.
### LTO validation in 0.3.3

LTO remains enabled when selected by the build calibration, but it is validated against the real ZOpen extension: all four native sources are compiled, the extension is linked, and `zopen._zopen` is imported. GCC uses its normal LTO linker path; Clang may use `lld`. This avoids producing a `.so` that contains unprocessed LLVM bitcode or lacks `PyInit__zopen`.


### 0.3.3 performance calibration improvements

The 0.3.3 source keeps the public version unchanged while adding a stricter end-to-end calibration path. `scripts/benchmark_candidate.py` now benchmarks a trained tokenizer with the encode cache disabled, uses unique inputs per sample, records MB/s plus token/s and latency, and includes a real batch workload. The batch pool adaptively activates 1, 2, or all workers based on total input bytes; tiny batches stay on the persistent serial workspace.

Decode now computes the exact output capacity in its validation pass and can enter the existing exact materializer on the public packed-ID path. This removes the former sampled-estimate-only bottleneck for large packed decodes while keeping the conservative fallback for generic iterables.

For a deeper compiler tournament, use:

```bash
ZOPEN_CALIBRATION_DEEP=1 ZOPEN_RECALIBRATE=1 python scripts/calibrate_build.py
```

The build still reports and selects the winner without changing the public package version from `0.3.3`.


## Evidence-driven microkernel dispatch (0.3.3)

Microkernel implementations are treated as candidates. A kernel is dispatched only when the local probe shows a stable >=5% speedup over the scalar/`memset` baseline across neighboring sizes. CPU/compiler availability alone never enables a kernel. `-march=native` is an explicit benchmark/build candidate and is not a default winner.

## Direct API examples

See [`docs/API.md`](docs/API.md) for copy-paste examples covering train, encode, decode, packed `u32` paths, batch/file APIs, save/load, cache control, and runtime capabilities.

## Performance and optimization

See [`OPTIMIZATION_0.3.2.md`](OPTIMIZATION_0.3.2.md) for the 0.3.2 optimization pass, scheduler changes, SIMD paths, and validation notes.
