Metadata-Version: 2.4
Name: mentat-gulp
Version: 1.6.82
Summary: gULP - (generic) Unified Log Processor.
Author-email: Mentat <info@mentat.is>
Requires-Python: <3.14,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE.AGPL-3.0.md
License-File: LICENSE.GULP.md
License-File: LICENSE.md
Requires-Dist: aiobotocore==3.2.1
Requires-Dist: aiofile==3.9.0
Requires-Dist: aiofiles==24.1.0
Requires-Dist: aiohappyeyeballs==2.6.1
Requires-Dist: aiohttp==3.13.3
Requires-Dist: aioitertools==0.13.0
Requires-Dist: aiomultiprocess==0.9.1
Requires-Dist: aioshutil==1.5
Requires-Dist: aiosignal==1.4.0
Requires-Dist: aiosmtplib==3.0.2
Requires-Dist: annotated-types==0.7.0
Requires-Dist: anyio==4.6.2.post1
Requires-Dist: art==6.2
Requires-Dist: async-unzip==0.3.6
Requires-Dist: asyncio-atexit==1.0.1
Requires-Dist: asyncio-pool==0.6.0
Requires-Dist: attrs==24.2.0
Requires-Dist: Authlib==1.4.0
Requires-Dist: botocore==1.42.61
Requires-Dist: caio==0.9.17
Requires-Dist: certifi==2024.8.30
Requires-Dist: cffi==1.17.1
Requires-Dist: charset-normalizer==3.3.2
Requires-Dist: click==8.1.7
Requires-Dist: colorama==0.4.6
Requires-Dist: colorclass==2.2.2
Requires-Dist: coloredlogs==15.0.1
Requires-Dist: construct==2.10.70
Requires-Dist: cryptography==44.0.0
Requires-Dist: defusedxml==0.7.1
Requires-Dist: deprecation==2.1.0
Requires-Dist: distro==1.9.0
Requires-Dist: dnspython==2.7.0
Requires-Dist: docx2txt==0.9
Requires-Dist: dotwiz==0.4.0
Requires-Dist: easygui==0.98.3
Requires-Dist: elastic-transport==8.15.1
Requires-Dist: elasticsearch==8.15.1
Requires-Dist: elementpath==5.0.4
Requires-Dist: Events==0.5
Requires-Dist: evtx==0.8.8
Requires-Dist: fastapi==0.115.5
Requires-Dist: frozenlist==1.5.0
Requires-Dist: gitdb==4.0.11
Requires-Dist: gulp-sdk
Requires-Dist: GitPython==3.1.41
Requires-Dist: greenlet==3.1.1
Requires-Dist: h11==0.14.0
Requires-Dist: httpcore==1.0.7
Requires-Dist: httpx==0.28.1
Requires-Dist: humanfriendly==10.0
Requires-Dist: idna==3.10
Requires-Dist: inflection==0.5.1
Requires-Dist: iniconfig==2.0.0
Requires-Dist: itsdangerous==2.2.0
Requires-Dist: Jinja2==3.1.6
Requires-Dist: jiter==0.9.0
Requires-Dist: jmespath==1.1.0
Requires-Dist: json5==0.9.25
Requires-Dist: libpff-python==20231205
Requires-Dist: llvmlite==0.44.0
Requires-Dist: lxml==6.0.2
Requires-Dist: MarkupSafe==2.1.5
Requires-Dist: msoffcrypto-tool==5.4.2
Requires-Dist: multidict==6.1.0
Requires-Dist: muty-python
Requires-Dist: ntplib==0.4.0
Requires-Dist: numba==0.61.0
Requires-Dist: numpy==2.1.3
Requires-Dist: odfpy==1.4.1
Requires-Dist: olefile==0.47
Requires-Dist: oletools==0.60.2
Requires-Dist: opensearch-py==2.7.1
Requires-Dist: orjson==3.10.18
Requires-Dist: packaging==24.1
Requires-Dist: pandas==2.3.3
Requires-Dist: pcodedmp==1.2.6
Requires-Dist: pillow==12.0.0
Requires-Dist: pluggy==1.5.0
Requires-Dist: prettytable==3.12.0
Requires-Dist: prometheus-client==0.21.1
Requires-Dist: prometheus-fastapi-instrumentator==7.0.2
Requires-Dist: propcache==0.2.0
Requires-Dist: psutil==6.0.0
Requires-Dist: psycopg==3.2.2
Requires-Dist: psycopg-binary==3.2.2
Requires-Dist: psycopg-pool==3.2.3
Requires-Dist: pycparser==2.22
Requires-Dist: pycryptodome==3.20.0
Requires-Dist: pydantic==2.9.2
Requires-Dist: pydantic_core==2.23.4
Requires-Dist: pyheck==0.1.5
Requires-Dist: pyparsing==3.1.0
Requires-Dist: pypdf==6.1.1
Requires-Dist: pySigma==0.11.23
Requires-Dist: pySigma-backend-opensearch==1.0.4
Requires-Dist: pytest==8.3.4
Requires-Dist: pytest-asyncio==0.24.0
Requires-Dist: pytest-ordering==0.6
Requires-Dist: python-dateutil==2.9.0.post0
Requires-Dist: python-multipart==0.0.9
Requires-Dist: python-pip==1.1.1
Requires-Dist: python-pptx==1.0.2
Requires-Dist: python-whois==0.9.5
Requires-Dist: pytz==2024.2
Requires-Dist: PyYAML==6.0.2
Requires-Dist: redis==7.0.1
Requires-Dist: regex==2024.11.6
Requires-Dist: requests==2.32.3
Requires-Dist: requests-toolbelt==1.0.0
Requires-Dist: scipy==1.15.1
Requires-Dist: setuptools==75.1.0
Requires-Dist: sigma-cli==1.0.4
Requires-Dist: six==1.16.0
Requires-Dist: smmap==5.0.1
Requires-Dist: sniffio==1.3.1
Requires-Dist: SQLAlchemy==2.0.34
Requires-Dist: sqlalchemy-dlock==0.6.1.post1
Requires-Dist: sqlalchemy-mixins==2.0.5
Requires-Dist: SQLAlchemy-Utils==0.41.2
Requires-Dist: starlette==0.41.2
Requires-Dist: stumpy==1.13.0
Requires-Dist: tenacity==9.0.0
Requires-Dist: tiktoken==0.12.0
Requires-Dist: tqdm==4.67.1
Requires-Dist: types-aiobotocore==3.2.1
Requires-Dist: types-aiobotocore-s3==3.2.1
Requires-Dist: typing-inspection==0.4.2
Requires-Dist: typing_extensions==4.12.2
Requires-Dist: tzdata==2025.2
Requires-Dist: urllib3==2.2.3
Requires-Dist: uvicorn==0.30.6
Requires-Dist: wcwidth==0.2.13
Requires-Dist: websockets==13.0.1
Requires-Dist: wrapt==2.1.1
Requires-Dist: xlsxwriter==3.2.9
Requires-Dist: xmltodict==0.13.0
Requires-Dist: xxhash==3.5.0
Requires-Dist: yarl==1.18.0
Dynamic: license-file

<div align="center">

<picture>
 <source media="(prefers-color-scheme: dark)" srcset="./logo.svg">
 <source media="(prefers-color-scheme: light)" srcset="./logo.svg">
 <img alt="gULP" src="./logo.svg" width="30%" height="30%">
</picture>

the graphical, universal Log processor for incident response!

_made with :heart: by Mentat._

</div>

<div align="center">

![GitHub followers](https://img.shields.io/github/followers/mentat-is?style=social)
![GitHub stars](https://img.shields.io/github/stars/mentat-is/gulp?style=social)
![GitHub languages](https://img.shields.io/github/languages/top/mentat-is/gulp)
![GitHub issues](https://img.shields.io/github/issues/mentat-is/gulp)
![GitHub sponsors](https://img.shields.io/github/sponsors/mentat-is)
</div>

<div align="center">

[Description](#description) - [Architecture](#architecture) - [Installation](#installation) - [Run & examples](#commandline-examples) - [GUI](#clients) - [Troubleshooting](./docs/troubleshooting.md)

</div>

## description

Gulp is a powerful software tool designed to streamline incident response and analysis. Its core features includes:

- **Data Ingestion Plugins**: Gulp can ingest data from a variety of sources, thanks to its versatile plugin system.
- **OpenSearch and ECS**: Gulp is built on OpenSearch and uses the _Elasticsearch Common Scheme (ECS)_ as its ingestion format, ensuring compatibility and ease of use.
- **High-Speed Multiprocessing Engine**: Gulp's engine is designed for speed, offering fast ingestion and querying capabilities through multiprocessing.
- **Query using SIGMA rules**: Gulp supports querying using Sigma Rules, allowing for easy, one-click queries with thousands of rules in parallel.
- **Collaboration Platform**: Gulp includes a collaboration platform, enabling teams to work together on the same incident. Features include note-taking, highlighting, and link adding.
- [**An innovative UI**](https://github.com/mentat-is/gulpui-web): Gulp's user interface includes multiple on-screen per-context(i.e. a log source) zoomable timelines for visualizing events, making it easier to understand and analyze incidents.

  <div class="slides">
    <div class="slide" id="slide-1"><img src="screenshot-1.png" alt="screenshot 1"></div>
    <div class="slide" id="slide-2"><img src="screenshot-2.png" alt="screenshot 2"></div>
    <div class="slide" id="slide-3"><img src="screenshot-3.png" alt="screenshot 3"></div>
  </div>


- **Scalable**: Gulp is designed with scalability in mind. As your data and team grow, you can simply add more gulp nodes, more cores to increase parallel ingestion and query capabilities, more OpenSearch and PostgreSQL nodes. This makes Gulp a flexible solution that can adapt to your evolving needs!
- **Python based**: Gulp is written in Python, leveraging open-source libraries whenever possible. This maximizes ease of adoption from the community, as Python is widely used and understood.

[here is a detailed datasheet](./DATASHEET.md)

## architecture

- [GULP architecture](./docs/architecture.md)

### plugins development

- [plugins and mapping](./docs/plugins_and_mapping.md)
- [testing guidelines](./docs/testing.md)

### integration with other applications

gulp can be of course [integrated with other applications](./docs/integration.md) !

> both websocket and REST API is available!

## installation

install our pypi package and run the necessary services via the example [docker-compose](./docker-compose.yml) and you should be good to go with the backend:

> **WARNING**: pip installation works only with python3.13 (tested) and 3.12, 3.14 is currently not supported because some dependencies have not yet released compatible versions.

~~~bash
set -e
curl -o .env https://raw.githubusercontent.com/mentat-is/gulp/develop/.env \
  && curl -o docker-compose.yml https://raw.githubusercontent.com/mentat-is/gulp/develop/docker-compose.yml \
  && python3 -m pip install --user mentat-gulp
curl https://github.com/mentat-is/gulp/blob/b90a2f1510c0bf382101f29f628fbda7e56c5a00/docker-compose.yml
pip3 install mentat-gulp
# then start the docker compose i.e. docker compose up -d to start the necessary services and then run gulp
# it will create a default configuration in ~/.config/gulp/gulp_cfg.json if not already present and start listening on http://localhost:8080
~~~

for advanced deployment, always check the following:

- [docker](./docs/install_docker.md)
- [install from sources/dev setup](./docs/install_dev.md)
- [installing extra plugins](./docs/install_dev.md#7-optional-installing-extra-plugins)

> pypi/docker registry versions may be outdated... to use the bleeding edge version of gulp, install from sources and use the 'develop' branches.
 
### clients

[gulp web ui](https://github.com/mentat-is/gulpui-web)

### environment variables

the following environment variables may be set to override configuration options.

- `GULP_BIND_TO_ADDR`, `GULP_BIND_TO_PORT` : if set, gulp will listen to this interface and port (either, the default `0.0.0.0`, `8080` is used).
  - for the override to work, both `GULP_BIND_TO_ADDR` and `GULP_BIND_TO_PORT` must be specified, either the value of one alone is ignored.

- `GULP_WORKING_DIR`: this is the **working directory** for gulp (defaults to `~/.config/gulp`), which contains:
  - `gulp_cfg.json`: the configuration, initialized with [template](./src/gulp/gulp_cfg_template.json) if not present
  - `plugins`: optional extra plugins (have precedence over `$INSTALLDIR/plugins`)
  - `mapping_files`: optional extra mapping files (have precedence over `$INSTALLDIR/mapping_files`)
  - `certs`: optional [SSL](#ssl) certificates for HTTPS
  - `ingest_local` directory to store big files for quick ingestion (`ingest_local` API)
  - `tmp_upload` folder to cache partial uploads during ingestion

- `GULP_OPENSEARCH_URL`: if set, overrides `opensearch_url` in the configuration to.
- `GULP_POSTGRES_URL`: if set, overrides `postgres_url` in the configuration.
- `GULP_REDIS_URL`: if set, overrides `redis_url` in the configuration.
- `GULP_S3_URL`: if set, overrides `s3_url` in the configuration.
- `GULP_INTEGRATION_TEST`: this may be set during tests to disable debug features if they're enabled

### exposed services

> using the default [docker-compose.yml](./docker-compose.yml) with no profile set.
> further profiles (i.e. `dev`, `metrics`) may expose additional services, check the compose file for details.

#### gulp

- [gulp swagger page on http://localhost:8080/docs](http://localhost:8080/docs)
- [gulp web UI on http://localhost:3000](http://localhost:3000)
  - **user/pwd: `admin/admin`** (default gulp admin user)

#### postgreSQL

- postgreSQL on **localhost:5432**
  - **user/pwd: `postgres/Gulp1234!`**

- [adminer on http://localhost:8001](http://localhost:8081) to manage postgreSQL (use `--profile dev`)
  - **server/user/pwd: `postgres/postgres/Gulp1234!`**

#### OpenSearch

- [opensearch on http://localhost:9200](http://localhost:9200)
  - **user/pwd: `admin/Gulp1234!`**

- [elasticvue on http://localhost:8082](http://localhost:8082) to visualize OpensSearch indexes  (use `--profile dev`)

- [opensearch dashboards on http://localhost:5001](http://localhost:5601) for a more comprehensive OpenSearch management.

#### redis

- redis on **localhost:6379**
  - **user/pwd: `default/Gulp1234!`**

- [redis insight on http://localhost:5540](http://localhost:8002) to manage redis  (use `--profile dev`)

#### MinIO

- [minio on http://localhost:9000](http://localhost:9000)

  - **user/pwd: `admin/Gulp1234!`**
  - this is used as S3-compatible filestore for plugins which may need it.
  
### SSL

to use SSL, the following configuration options and files variables may be provided:

#### OpenSearch

- Gulp configuration
  - `opensearch_verify_certs`: set to `false` to skip server verification
- certificate files
  - `$GULP_WORKING_DIR/certs/opensearch-ca.pem`: CA certificate for Gulp to connect to the Opensearch server
  - `$GULP_WORKING_DIR/certs/opensearch.pem`: client certificate for Gulp to connect to the Opensearch server
  - `$GULP_WORKING_DIR/certs/opensearch.key`: ***passwordless*** client certificate key

#### PostgreSQL

- Gulp configuration
  - `postgres_ssl`: use SSL for postgres connection, set to `false` to not use.
  - `postgres_verify_certs`: set to `false` to skip server verification
- certificate files
  - `$GULP_WORKING_DIR/certs/postgres-ca.pem`: CA certificate for Gulp to connect to the PostgreSQL server
  - `$GULP_WORKING_DIR/certs/postgres.pem`: client certificate for Gulp to connect to PostgreSQL server
  - `$GULP_WORKING_DIR/certs/postgres.key`: client certificate key

#### gulp

- Gulp configuration
  - `https_enforce`: set to `true` to enforce connection to Gulp only through HTTPS
  - `https_enforce_client_certs`: set to `true` to enforce check of client certificates signed by `gulp-ca.pem` CA
- certificate files
  - `$GULP_WORKING_DIR/certs/gulp-ca.pem`: Gulp CA
  - `$GULP_WORKING_DIR/certs/gulp.pem`: Gulp server certificate
  - `$GULP_WORKING_DIR/certs/gulp.key`: Gulp server certificate key

#### redis & minIO

check their specific documentation: basically you have to put certificates in `$GULP_WORKING_DIR/certs`, whcih is [mounted in the containers](./docker-compose.yml), and tweak their configuration.

## commandline examples

default startup, creates collab database with an operation names `test_operation` on the very first run.

~~~bash
gulp
~~~

> to detect if gulp has already run once check for `~/.config/gulp/.first_run_done` and delete it to revert to first run on the next run.

deletes data related to `ALL` existing operations, both on collaboration database and OpenSearch.

~~~bash
gulp --reset-collab
~~~

deletes data related to `ALL` existing operations, both on collaboration database and OpenSearch, in the end creates/recreates `my_operation`.

~~~bash
gulp --reset-collab --create my_operation
~~~

acts only on `my_operation`: creates/recreates operation, deletes all related data both on collaboration database and OpenSearch.

~~~bash
gulp --create my_operation
~~~
