Metadata-Version: 2.4
Name: toxpol-nlp
Version: 0.2.0
Summary: NLP toolkit for toxicity and polarization research: synthetic datasets and detection algorithms
Author-email: SwkratisCS <swkratisgiannoutsos@gmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/SwkratisCS/polarizedtrees
Keywords: toxicity,polarization,annotation,synthetic data,nlp,disagreement,crowdsourcing,demographics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: pandas
Provides-Extra: ndfu
Requires-Dist: ndfu; extra == "ndfu"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# toxpol-nlp

NLP toolkit for **toxicity and polarization research**. Provides tools for synthetic dataset generation and polarization detection in human annotation studies.

## Install

```bash
pip install toxpol-nlp

# with nDFU support
pip install "toxpol-nlp[ndfu]"
```

## Tools

| Module | Description | Status |
|---|---|---|
| `toxpol.datagen` | Synthetic annotator pool with injected, ground-truth polarization | Stable |
| `toxpol.trees` | Polarized Trees detection algorithm | Coming soon |

## `toxpol.datagen`

Builds a pool of annotators with explicit demographic identities and generates annotation datasets where each text is independently assigned a severity tier:

- **High** — strong, non-overlapping toxic/civil split
- **Moderate** — same mechanism, softer/overlapping split
- **Low** — little to no polarization (mostly a true negative control)

The bias configuration is returned alongside the dataset as ground truth, enabling direct validation of detection algorithms.

```python
from toxpol.datagen import AnnotatorPool

dimensions = {
    "gender": ["male", "female", "non-binary"],
    "politics": ["left", "center", "right"],
    "age": ["<25", "25-50", ">50"],
    "education": ["low", "medium", "high"],
    "orientation": ["heterosexual", "lgbtq+"],
}

pool = AnnotatorPool(
    dimensions=dimensions,
    scale=5,
    toxic_range=(4, 5),
    civil_range=(1, 2),
    neutral_range=(3, 3),
    exclude=None,
    annotators_per_identity=10,
)

dataset, bias_configs = pool.generate_dataset(
    n_texts=100,
    n_annotators_per_text=150,
    noise=0.05,
    high_ratio=0.60,
    moderate_ratio=0.20,
    low_ratio=0.20,
    low_unimodal_share=0.40,
)
# dataset columns: text_id, annotator_id, <dimensions>, rating
# bias_configs: ground-truth tier, sub-case, bias config, and threshold per text
```

Full API documentation available on [GitHub](https://github.com/Swkratis210204/toxpol-nlp/tree/main).
