Metadata-Version: 2.4
Name: ghostcrawl
Version: 2.1.1
Summary: Official Python SDK for the Ghostcrawl local orchestration API.
Author: Ghostcrawl
License: Apache-2.0
Project-URL: Homepage, https://github.com/ghostcrawl/ghostcrawl
Keywords: ghostcrawl,scraping,browser,automation,agent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.13
Requires-Dist: typer>=0.12
Requires-Dist: mcp>=1.27.1
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: respx>=0.23.1; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Dynamic: license-file

# ghostcrawl — Python SDK + CLI

Typed Python client and `ghostcrawl` CLI for the GhostCrawl REST + MCP surface.
Full coverage of scrape, crawl, sessions, datasets, recordings, profiles,
webhooks, and the MCP tools.

Public engine names are `chrome`, `firefox`, and `webkit`. There are no other engine
names — `engine="auto"` lets the orchestrator pick.

## Install

```bash
pip install ghostcrawl
```

Requires Python 3.10+. Runtime dependencies: `httpx>=0.28.1`, `pydantic>=2.13`,
`typer>=0.12`, `mcp>=1.27.1`.

## Quickstart — Identity (v1.6 SaaS)

The v1.6 surface adds the `ghostcrawl init / identity` workflow for generating
authenticated, opaque identity envelopes consumed by the engine binary.

### CLI

```bash
# One-time setup: stores API key to ~/.config/ghostcrawl/config.toml
ghostcrawl init

# Request an identity envelope
ghostcrawl identity --os ios --browser safari --device iphone-15-pro
```

Output is an opaque JSON envelope with an AES-256-GCM encrypted payload.
The payload is not user-decodable — it is consumed verbatim by the browser binary.

### Python SDK

```python
from ghostcrawl import Ghostcrawl

client = Ghostcrawl(api_key="gc_...")
envelope = client.identity(
    claim_os="ios",
    claim_browser="safari",
    device_model="iphone-15-pro",
)

# Pass to the engine binary; do not attempt to decode ciphertext
print(envelope.payload.alg)        # 'AES-256-GCM'
print(envelope.payload.key_id)     # '16-char-hex-id'
```

> **Note:** The `payload.ciphertext` field is an opaque base64 string. It is
> designed to be passed directly to the engine binary, which handles decryption
> using a server-derived key. The SDK never decodes or inspects the ciphertext.

## Quickstart — SDK (sync)

```python
from ghostcrawl import GhostcrawlClient

client = GhostcrawlClient(api_key="YOUR_KEY", base_url="https://api.ghostcrawl.io")

# Scrape
result = client.page.scrape(url="https://example.com", engine="chrome")
print(result)

# Start a crawl run
run = client.crawl_runs.create(
    seed_url="https://example.com",
    max_depth=2,
    max_pages=50,
)
print(run.run_id)

# List sessions
sessions = client.sessions.list()
for s in sessions.items:
    print(s.session_id, s.status)
```

## Quickstart — SDK (async)

```python
import asyncio
from ghostcrawl import AsyncGhostcrawlClient

async def main():
    async with AsyncGhostcrawlClient(api_key="YOUR_KEY") as client:
        result = await client.page.scrape(url="https://example.com")
        print(result)

asyncio.run(main())
```

## Quickstart — CLI

```bash
export GHOSTCRAWL_API_KEY=your-token

# Scrape a URL (JSON to stdout)
ghostcrawl scrape https://example.com

# Start a crawl run
ghostcrawl crawl https://example.com --max-depth 2 --out crawl.json

# List sessions
ghostcrawl session list

# Auth helpers
ghostcrawl auth whoami
ghostcrawl auth login
```

## MCP Wrapper

Typed wrapper over the MCP streamable-HTTP transport.

```python
import asyncio
from ghostcrawl.mcp import GhostcrawlMCPClient

async def main():
    mcp = GhostcrawlMCPClient(
        mcp_url="https://api.ghostcrawl.io/mcp",
        api_key="YOUR_KEY",
    )
    await mcp.connect()

    # Navigate
    result = await mcp.navigate(url="https://example.com")

    # Act on page
    result = await mcp.act(goal="Find the contact email")

    # Extract structured data
    result = await mcp.extract(schema={"type": "object", "properties": {"title": {"type": "string"}}})

    # Screenshot
    shot = await mcp.screenshot()

    # Health check
    pong = await mcp.ping()

    await mcp.close()

asyncio.run(main())
```

## Resource Coverage

| Resource | Client attribute | Endpoints covered |
|----------|-----------------|-------------------|
| Page (scrape/crawl/agent) | `client.page` | scrape, map, screenshot, pdf, search |
| Sessions | `client.sessions` | list, get, create, end, navigate, pause, resume |
| Crawl runs | `client.crawl_runs` | create, list, get, cancel |
| Datasets | `client.datasets` | list, get, create, delete, append |
| Recordings | `client.recordings` | list, get, create, delete |
| Profiles | `client.profiles` | list, get, create, delete |
| Webhooks | `client.webhooks` | list, get, create, delete, test, queue |
| Storage states | `client.storage_states` | list, get, import, export |
| Script | `client.script` | run |
| Observability | `client.observability` | metrics, health |
| Auth / tokens | `client.auth`, `client.auth_tokens` | login, refresh, token CRUD |
| Budgets | `client.budgets` | get, update |
| Audit log | `client.audit` | list events |

## Authentication

```python
client = GhostcrawlClient(
    api_key="gc_live_...",
    base_url="https://api.ghostcrawl.io",  # default — override for self-hosted
)
```

Every request sends `Authorization: Bearer <api_key>` and `X-API-Key: <api_key>`.

## License

Apache-2.0
