Metadata-Version: 2.4
Name: inferml
Version: 1.0.3
Summary: Any HuggingFace model. Local. Multi-modal. Served over an OpenAI-compatible API.
Author: InferML
License: MIT License
        
        Copyright (c) 2026 InferML, Gitesh Chawda
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
        THE SOFTWARE.
        
Project-URL: Homepage, https://github.com/IMvision12/InferML
Keywords: huggingface,transformers,inference,openai,local,llm,diffusion
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.29
Requires-Dist: huggingface_hub
Requires-Dist: platformdirs>=4
Requires-Dist: psutil>=5.9
Provides-Extra: inference
Requires-Dist: transformers>=5.7.0; extra == "inference"
Requires-Dist: torch>=2.6; extra == "inference"
Requires-Dist: torchvision; extra == "inference"
Requires-Dist: torchaudio>=2.6; extra == "inference"
Requires-Dist: diffusers; extra == "inference"
Requires-Dist: accelerate; extra == "inference"
Requires-Dist: timm; extra == "inference"
Requires-Dist: pillow; extra == "inference"
Requires-Dist: soundfile; extra == "inference"
Requires-Dist: librosa; extra == "inference"
Requires-Dist: numpy; extra == "inference"
Requires-Dist: scipy; extra == "inference"
Requires-Dist: sentencepiece; extra == "inference"
Requires-Dist: protobuf; extra == "inference"
Dynamic: license-file

<p align="center">
  <img src="assets/logo.png" alt="InferML logo" width="140" />
</p>

# InferML

Any Hugging Face model. Local. Multi-modal. Now a **local web server** with an
**OpenAI-compatible API** - no Electron, no native binary.

Run 143+ model families fully on-device (LLMs, VLMs, diffusion, ASR, TTS,
segmentation, detection) behind a browser UI, and point agent frameworks
(LangChain, LangGraph, the OpenAI SDK) at it the way you point them at Ollama.

## Install

Requires **Python 3.10+** - the installer checks for it but won't install Python
for you. One line in your terminal:

```bash
# macOS / Linux
curl -fsSL https://inferml.vercel.app/install.sh | sh
# Windows (PowerShell)
irm https://inferml.vercel.app/install.ps1 | iex
```

The script bootstraps pipx and installs the InferML server. On first launch the
app walks you through installing the inference stack (PyTorch + transformers) for
your hardware - pick **CPU** or **GPU** and it fetches the matching build.

Prefer to do it by hand?

```bash
pipx install inferml                 # server only; the app installs torch on first run
pipx install "inferml[inference]"    # or grab the whole stack up front (generic torch wheel)
```

## Run

```bash
inferml                 # starts the server and opens http://localhost:11500
inferml --port 8080     # custom port
inferml --host 0.0.0.0 --no-browser   # expose on the LAN, headless
```

Open the printed URL, download a model from the Hub tab, and run it.

## OpenAI-compatible API

Point any OpenAI client at `http://localhost:11500/v1` (any api key). It routes
to whichever LLM is currently loaded in InferML.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11500/v1", api_key="not-needed")
client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
```

Supports streaming (`stream=True`), `GET /v1/models`, and tool/function calling
for the Qwen/Hermes, Llama, and Mistral families.

## Docker

```bash
docker build -t inferml .
docker run --rm -p 11500:11500 inferml            # CPU
docker run --rm --gpus all -p 11500:11500 inferml # GPU
```

## Development

The React UI lives in `src/renderer/` (built with esbuild) and talks to the
server via `window.inferml` (see `src/renderer/web-bridge.js`). The Python
server + inference engine live in `python/`.

```bash
npm install          # build deps (esbuild + the vendored UMD libs)
npm run build        # compile the renderer and bundle it into the package
pip install -e ".[inference]"
inferml
```
