Metadata-Version: 2.4
Name: lexicont
Version: 0.1.1
Summary: Policy-driven agent for real-time text moderation
Author: corefrg
License-Expression: MIT
Project-URL: Homepage, https://github.com/corefrg/lexicont
Project-URL: Bug Tracker, https://github.com/corefrg/lexicont/issues
Keywords: moderation,nlp,toxicity,content-moderation,llm
Classifier: Programming Language :: Python :: 3.11
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Filters
Requires-Python: <3.12,>=3.11
Description-Content-Type: text/markdown
Requires-Dist: glin-profanity
Requires-Dist: rapidfuzz>=3.14.3
Requires-Dist: detoxify>=0.5.2
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.32
Requires-Dist: qdrant-client>=1.12.0
Requires-Dist: sentence-transformers>=3.3.0
Requires-Dist: fastapi<0.136.0,>=0.135.2
Requires-Dist: uvicorn<0.43.0,>=0.42.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-mock>=3.0; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"

# Lexicont: lightweight policy driven agent for text moderation

Lexicont is a deterministic moderation system built as a **policy-driven agent** that combines fast rule-based filters, machine learning, and LLM reasoning. It processes the majority of inputs in milliseconds and only invokes the LLM when confidence is low, making it suitable for production environments where low latency is required.

---

## Overview

- **Fast layers** - profanity detection, fuzzy matching, toxicity ML - run quickly.
- **Early stop** - high-confidence blocks skip further processing.
- **LLM layers** - intermediate triage and final judgment with Qwen via llamacpp or Ollama.
- **RAG support** - Qdrant vector database for retrieving similar examples.
- **Policy driven agent** - explicit control loop ensures deterministic behaviour.

---

## Architecture

The moderation pipeline follows a fixed order of stages:

1. `profanity_filter` - dictionary + leetspeak detection
2. `fuzzy_trigger` - partial ratio matching
3. `toxicity_ml` - multilingual toxicity classifier
4. `llm_entry_judge` - LLM for text normalisation and decision on stage 4
5. `llm_judge` - final LLM with RAG

**Control logic**
- After stage 1 or stage 2, if confidence >= 0.85 and decision is `block`, the pipeline stops.
- Stage 4 is invoked only if either:
  - the triage layer (stage 3.5) explicitly allows it, or
  - triage is disabled and max confidence < 0.80.

---

## Quick start

### Local (Poetry)

```bash
git clone https://github.com/corefrg/lexicont.git
cd lexicont
poetry install
poetry run lexicont "offensive text"
```

### Docker Compose

```bash
docker-compose up -d
curl -X POST http://localhost:8000/moderate \
  -H "Content-Type: application/json" \
  -d '{"text":"buy fake documents"}'
```

---

## Usage

### Command line

```bash
# legacy syntax - works as before
poetry run lexicont "text to moderate"
poetry run lexicont "text" --log-level DEBUG --verbose

# explicit subcommand
poetry run lexicont check "text to moderate"
poetry run lexicont check "text" --log-level DEBUG --verbose

# interactive mode
poetry run lexicont
```

### HTTP API

Start the server:

```bash
poetry run uvicorn lexicont.api:app --host 0.0.0.0 --port 8000
```

Endpoints:
- `POST /moderate` - returns decision, confidence, and stage details
- `GET /health` - status check

### From Python

```python
from lexicont.pipeline import run

result = run("text")
print(result.final_decision)   # block, review, or pass
print(result.max_confidence)
```

---

## Configuration

### Config files

| File | Purpose | Env var |
|------|---------|---------|
| `moderation_config.yaml` | Main settings: thresholds, stage toggles, LLM, RAG | `LEXICONT_CONFIG` |
| `moderation_rules.v1.yaml` | Phrase lists for profanity and fuzzy stages | `LEXICONT_RULES` |
| `patterns.jsonl` | Semantic patterns for the RAG layer | `LEXICONT_PATTERNS` |

Each file has a built-in default bundled with the package. To override, either set the env var or place the file in the working directory - it will be picked up automatically.

### Copy defaults and edit

```bash
poetry run lexicont init --dir my_configs

set LEXICONT_CONFIG=my_configs\moderation_config.yaml
set LEXICONT_RULES=my_configs\moderation_rules.v1.yaml
set LEXICONT_PATTERNS=my_configs\patterns.jsonl

poetry run lexicont check "buy a fake license"
```

On Linux/macOS use `export` instead of `set`.

### Key options in moderation_config.yaml

- `general.early_stop_confidence` - threshold for early termination (default 0.85)
- `general.stage4_trigger_confidence` - threshold to invoke LLM when triage disabled (default 0.80)
- `general.enable_stage1/2/3` - toggle individual stages on or off
- `llm_judge.backend` - `llamacpp` or `ollama`
- `llm_judge.rag.store_backend` - `inmemory` or `qdrant_server`

For Docker, use `host.docker.internal` to reach LLM services running on the host.

### moderation_rules.v1.yaml structure

```yaml
categories:
  profanity:
    - phrase one
  illegal:
    - buy fake license
  my_category:
    - custom phrase
```

### patterns.jsonl structure

One JSON object per line:

```
{"text": "buy a license through traffic police", "label": "offer to buy forged documents", "category": "illegal"}
```

---

## Development

```bash
poetry install
pre-commit install
ruff format src tests
ruff check --fix src tests
```

To add a new filter, implement a function in `filters/` and register it in `agent.py`.

---

## License

MIT
