Metadata-Version: 2.4
Name: openviking
Version: 0.1.1
Summary: An Agent-native context database: Data in, Context out
Author-email: ByteDance <noreply@bytedance.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/volcengine/openviking
Project-URL: Documentation, https://openviking.readthedocs.io
Project-URL: Repository, https://github.com/volcengine/openviking
Project-URL: Issues, https://github.com/volcengine/openviking/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0.0
Requires-Dist: typing-extensions>=4.5.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: pdfplumber>=0.10.0
Requires-Dist: readabilipy>=0.2.0
Requires-Dist: markdownify>=0.11.0
Requires-Dist: openai>=1.0.0
Requires-Dist: requests>=2.28.0
Requires-Dist: json-repair>=0.25.0
Requires-Dist: pytest>=7.0.0
Requires-Dist: pytest-asyncio>=0.21.0
Requires-Dist: pytest-cov>=4.0.0
Requires-Dist: black>=23.0.0
Requires-Dist: isort>=5.12.0
Requires-Dist: mypy>=1.0.0
Requires-Dist: ruff>=0.1.0
Requires-Dist: sphinx>=7.0.0
Requires-Dist: sphinx-rtd-theme>=1.3.0
Requires-Dist: myst-parser>=2.0.0
Requires-Dist: apscheduler>=3.11.0
Requires-Dist: orjson>=3.11.4
Requires-Dist: volcengine>=1.0.212
Requires-Dist: volcengine-python-sdk[ark]>=5.0.3
Requires-Dist: pyagfs>=1.4.0
Requires-Dist: psutil>=7.2.1
Requires-Dist: fastapi>=0.128.0
Requires-Dist: uvicorn>=0.39.0
Requires-Dist: werkzeug>=3.1.4
Requires-Dist: pandas>=2.3.3
Requires-Dist: xxhash>=3.0.0
Dynamic: license-file

# OpenViking

OpenViking is an agent-oriented **Context Database**, and it **Organize Context for AI Agents!**

OpenViking is a context database designed for AI Agents. It provides a complete solution from data to context. Whether documents, web pages, or multimedia, OpenViking can parse, store, and retrieve them intelligently to supply precise context for your AI Agent.

---

## Introduction

### Core Ideas

OpenViking embraces Context Engineering as the core design principle:

- Data in: Accepts multiple input formats (PDF, Word, web pages, images, etc.)
- Context out: Outputs intelligently processed context directly usable by Agents

### Highlights

- Treat everything as a File: Unified management via AGFS filesystem abstraction
- Agent Friendly: Native support for MCP and OpenSkills, easy Agent integration
- High-performance Retrieval: int8 Quantization, Dense & Sparse hybrid search, Scalar Filtering in OpenViking's VectorDB
- Embedded or Service Mode: Embedded for local dev (single process), Service mode for production (multi-process)
- Fast Production Transition: Easy integration with VikingDB, TOS, JuiceFS

---

## Quick Start

### 1. Installation

Install with `uv` (recommended):

```bash
uv pip install openviking
```

Set configuration file:

```bash
export OPENVIKING_CONFIG_FILE=ov.conf
```

`ov.conf` example, please read [models-configuration-guide/volcengine-models-china.md](./docs/models-configuration-guide/volcengine-models-china.md) to get your api-key from [Volcengine](https://console.volcengine.com/). OpenAI models are supported for Non-China users.
```json
{
  "embedding": {
    "dense": {
        "model": "doubao-embedding-vision-250615",
        "api_key": "{ your-volcengine-api-key }",
        "api_base": "https://ark.cn-beijing.volces.com/api/v3",
        "dimension": 1024,
        "input": "multimodal",
        "backend": "volcengine"
    }
  },
  "vlm": {
    "model": "doubao-seed-1-8-251228",
    "api_key": "{ your-volcengine-api-key }",
    "api_base": "https://ark.cn-beijing.volces.com/api/v3",
    "temperature": 0.0,
    "max_retries": 2,
    "backend": "volcengine"
  },
  "log_level": "INFO",
  "log_format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
  "log_output": "stdout"
}
```

### 2. Embedded Mode

The simplest way to use OpenViking, no extra services required:

```python
from openviking import SyncOpenViking

# Create a local instance (VikingVectorIndex uses local storage, AGFS starts automatically)
client = SyncOpenViking(path="./data")
client.initialize()

# Add resource — supports URLs, file paths, etc.
result = client.add_resource(
    "./documents/guide.md",
    reason="User guide documentation"
)
print(f"Added: {result['root_uri']}")

# View directory structure (available immediately)
entries = client.ls(result['root_uri'])
for entry in entries:
    print(f"  {entry['name']} - {'dir' if entry['isDir'] else 'file'}")

# Read raw file content (available immediately)
content = client.read("viking://resources/documents/guide.md")
print(f"Content preview: {content[:200]}")

# Wait for semantic processing to complete
client.wait_processed()

# Now semantic features are available
# L0: Abstract (~100 tokens)
abstract = client.abstract(result['root_uri'])
print(f"Abstract: {abstract}")

# L1: Overview (~2000 tokens)
overview = client.overview(result['root_uri'])
print(f"Overview: {overview[:300]}")

# Semantic search
results = client.find("how to use")
print(f"Found {results.total} results")

for ctx in results.resources:
    print(f"  {ctx.uri} (score: {ctx.score:.2f})")

client.close()
```

### 3. Server Mode

For production or multi-process deployment, first start VikingVectorIndex and AGFS services, then connect:

```python
from openviking import SyncOpenViking

# Connect to remote services
client = SyncOpenViking(
    vectordb_url="http://localhost:5000",
    agfs_url="http://localhost:8080"
)
client.initialize()

# Same usage as Embedded mode
results = client.find("your query")
print(f"Found {results.total} results")

client.close()
```

See [Storage Documentation](./docs/OpenViking存储.md) for service deployment guide.

---

## Key Features

### 1. Everything as a File — Filesystem Abstraction

Built on [AGFS](./third_party/agfs) to provide a unified filesystem abstraction:

- Viking URI: Unified resource identifier `viking://{scope}/{path}/{name}`
- Three-layer information model: L0 (abstract) / L1 (overview) / L2 (full content)
- Soft links: Semantic associations between resources

```
viking://
├── resources/           # Independent resources (organized by project)
├── user/
│   └── memories/       # user memory
├── agent/
│   ├── memories/       # Agent learning memory
│   ├── instructions/   # Agent instructions
│   └── skills/         # Skill registry
└── session/{id}/       # Session data
```

### 2. Agent Friendly — Agent Workflow Ready

Native support for mainstream Agent protocols: MCP and Skills.

### 3. Context-Engineering-Friendly Vector Database Protocol

High performance and low cost vector search:

| Feature | Description |
|---------|-------------|
| int8 Quantization | High compression ratio vector quantization with int8 precision |
| Dense & Sparse | Hybrid search combining semantics and keywords |
| Scalar Filtering | Efficient scalar filtering |
| Directory Index | Directory-level index acceleration |
| Rerank Support | Native integration of Doubao Rerank model |

### 4. Smooth Transition to Production

From development to production without friction:

| Component | Development | Production |
|-----------|-------------|------------|
| Storage | Local files | TOS / JuiceFS |
| Vector DB | Embedded | VikingDB |
| LLM | OpenAI Compatible | Doubao series models |

Resources to prepare:

- MinerU (document parsing)
- API keys for LLM / VLM / Embedding / Rerank
- Storage backend (optional)

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                           OpenViking                                 │
├─────────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │
│  │   Parser    │────▶│ TreeBuilder │────▶│   Storage   │           │
│  │             │     │             │     │             │           │
│  └─────────────┘     └─────────────┘     └──────┬──────┘           │
│         │                                        │                  │
│         │                                        │                  │
│         ▼                                        ▼                  │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │
│  │     VLM     │     │    AGFS     │◀────│  VikingVectorIndex   │           │
│  │  Processor  │     │ File System │     │             │           │
│  └─────────────┘     └─────────────┘     └──────┬──────┘           │
│                                                  │                  │
│                                                  ▼                  │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │
│  │   Intent    │────▶│  Retriever  │────▶│   Session   │           │
│  │  Analyzer   │     │             │     │             │           │
│  └─────────────┘     └─────────────┘     └─────────────┘           │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Deployment

### Local Deployment

#### Build AGFS (First time only)

```bash
cd third_party/agfs/agfs-server && make build
cp build/agfs-server ../bin/
```

AGFS will start automatically when you create an OpenViking instance.

#### Using AGFS Shell (Optional)

```bash
uv run agfs-shell --agfs-api-url http://127.0.0.1:8080
```

### Container Deployment

### Cloud Deployment

---

## Best Practices

### Knowledge Management

### Writing Assistant

### DeerFlow Deep Research

---

## Project Structure

```
openviking/
├── build.sh              # Build scripts
├── requirements.txt      # Python dependencies
├── README.md
├── third_party/
│   ├── agfs/             # AGFS filesystem
│   ├── croaring/         # Bitmap library
│   └── rapidjson/        # JSON parsing
├── src/                  # C++ core implementation
├── openviking/           # Python SDK
│   ├── core/             # Core data models
│   ├── parse/            # Resource parsers
│   ├── retrieve/         # Retrieval system
│   ├── session/          # Session management
│   ├── storage/          # Storage interfaces
│   └── utils/            # Utilities
└── examples/             # Example code
```

---

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](./CONTRIBUTING.md) for details.

---

## Acknowledgements

OpenViking is built upon these excellent open-source projects:

| Project | Description | License |
|---------|-------------|---------|
| [AGFS](https://github.com/bytedance/agfs) | Agent-native filesystem abstraction | Apache-2.0 |
| [LevelDB](https://github.com/google/leveldb) | Fast key-value storage library | BSD-3-Clause |
| [CRoaring](https://github.com/RoaringBitmap/CRoaring) | Roaring bitmap implementation in C | Apache-2.0 |
| [RapidJSON](https://github.com/Tencent/rapidjson) | Fast JSON parser/generator | MIT |
| [spdlog](https://github.com/gabime/spdlog) | Fast C++ logging library | MIT |

---

## License

Apache License 2.0 — see [LICENSE](./LICENSE) for details.

---

## Links

- GitHub: https://github.com/bytedance/openviking
- Issues: https://github.com/bytedance/openviking/issues
