Metadata-Version: 2.4
Name: optimzed-loop
Version: 0.1.0
Summary: High-performance, memory-safe scraping with curl_cffi and lxml
Author-email: Your Name <you@example.com>
License: MIT
Project-URL: Homepage, https://github.com/yourusername/optimzed-loop
Project-URL: Repository, https://github.com/yourusername/optimzed-loop.git
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: curl_cffi>=0.7
Requires-Dist: lxml>=4.9
Provides-Extra: diagnostics
Requires-Dist: psutil>=5.9; extra == "diagnostics"
Dynamic: license-file

# Optimzed-Loop

A high-performance, memory-safe Python package for web scraping using `curl_cffi` and `lxml`, addressing common optimization issues:

- **Memory leaks** from `lxml` HTML parsing (via subprocess isolation).
- **Memory leaks** from `curl_cffi` sessions (via session reuse and reset).
- **Timeout and SSL handshake hangs** (via hard timeout enforcement and retries).
- **Connection pooling** (via session reuse).

## Features

- `SessionManager` (sync) and `AsyncSessionManager` (async) with:
  - Connection pooling (reuse one session).
  - Automatic retries with exponential backoff.
  - Hard timeout enforcement using `asyncio.wait_for`.
  - Reset to clear stale connections (fixes `CurlOpt.RESOLVE` leak).
- `safe_parse_html()` and `safe_parse_html_async()`:
  - Runs `lxml` parsing in a separate subprocess to isolate memory leaks.
  - Timeout protection.
- `MemoryMonitor`: track memory usage and raise warnings.
- Debug logging integration.

## Installation

```bash
pip install optimzed-loop
```

Requires Python 3.10+.

## Quick Start

```
from optimzed_loop import SessionManager, safe_parse_html

# Sync usage
with SessionManager(impersonate="chrome") as session:
    response = session.get("https://example.com")
    html = response.text
    # Parse in subprocess to avoid memory leak
    links = safe_parse_html(html, xpath="//a/@href")
    print(links)
```

Async usage:

```
import asyncio
from optimzed_loop import AsyncSessionManager, safe_parse_html_async

async def main():
    async with AsyncSessionManager(impersonate="chrome") as session:
        response = await session.get("https://example.com")
        html = response.text
        links = await safe_parse_html_async(html, xpath="//a/@href")
        print(links)

asyncio.run(main())
```

For more details, see the documentation.

