Metadata-Version: 2.4
Name: airflow-providers-dna
Version: 0.1.4
Summary: DAG-run lineage for DataHub — completes the acryl plugin's pipeline 'Runs' tab on Apache Airflow 3.
Keywords: airflow-provider,datahub,lineage,openlineage,airflow
Author: ESO-DBA-ODIN
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-Expression: Apache-2.0
Classifier: Framework :: Apache Airflow
Classifier: Framework :: Apache Airflow :: Provider
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
License-File: LICENSE
License-File: NOTICE
Requires-Dist: apache-airflow>=3.0.0,<4.0.0
Requires-Dist: acryl-datahub-airflow-plugin>=0.14,<2.0
Requires-Dist: acryl-datahub>=0.14,<2.0
Requires-Dist: pytest>=8.0 ; extra == "dev"
Requires-Dist: ruff>=0.5 ; extra == "dev"
Provides-Extra: dev

# airflow-providers-dna

DAG-run-level lineage for **DataHub** on **Apache Airflow 3**.

## What problem it solves

The acryl `datahub-airflow-plugin` emits run instances only at the **task** level
(`DataProcessInstance` → `DataJob`). Its `on_dag_run_success` / `on_dag_run_failed`
listener hooks are an unimplemented `TODO` upstream, so no **DAG-run-level**
`DataProcessInstance` is created. Effect: the per-task "Runs" tab is populated, but
the **pipeline (dataFlow) "Runs" tab stays empty** — that view reads `dataFlow.runs`,
i.e. `DataProcessInstance`s parented to the dataFlow.

This provider registers an Airflow listener that, on DAG-run completion, emits a
`DataProcessInstance` parented to the dataFlow (with start + end run events). That
populates the pipeline "Runs" tab.

## How it works (reuse, not reinvention)

It **piggy-backs on the acryl plugin** rather than duplicating it:

- Reuses the acryl listener singleton (`get_airflow_plugin_listener()`) → its
  `config` and emitter, so it talks to the **same GMS over the same connection**.
- Reuses `AirflowGenerator.generate_dataflow(config, dag)` and ids the DPI by
  `dag_run.run_id`, so URNs line up exactly with the dataFlow/dataJobs the acryl
  plugin already emits.

It emits the DPI events via the DataHub SDK directly (not `AirflowGenerator.run_dataflow`),
which avoids that helper's Airflow-2-shaped property bag that is fragile on Airflow 3.

## Install

```bash
pip install airflow-providers-dna
```

It depends on `acryl-datahub-airflow-plugin` and `acryl-datahub`; configure the
acryl plugin as usual (e.g. the `datahub_rest_default` connection / `[datahub]`
section). No extra configuration is required — this provider reuses that setup.

## Configuration

| Env var | Default | Purpose |
|---|---|---|
| `DNA_LINEAGE_DISABLED` | `false` | Set to `true` to disable DAG-run emission without uninstalling. |

Everything else (GMS URL, token, cluster, DAG filters) is inherited from the acryl
plugin's configuration.

## Develop, build & publish

A `Makefile` wraps the whole flow in an isolated `.venv` (so it never touches your
system Python). Run `make help` to list targets.

```bash
make test           # unit tests (needs Python 3.10+ / Apache Airflow 3)
make build          # clean + build wheel/sdist into dist/
make check          # validate distribution metadata (twine check)
make publish-test   # build + check + upload to TestPyPI
make publish        # build + check + upload to PyPI
```

Override the interpreter if needed, e.g. `make build PYTHON=python3.11`.

### Publishing

Releases go to [PyPI](https://pypi.org/project/airflow-providers-dna/) as
`airflow-providers-dna`. To cut a release:

1. Bump `version` in `pyproject.toml` (PyPI rejects re-uploading an existing
   version) and update `CHANGELOG.md`.
2. `make publish-test` and verify it installs from TestPyPI.
3. `make publish`.

At the upload prompt use username `__token__` and a PyPI API token as the
password (or configure `~/.pypirc`).

Once published, install it anywhere with:

```bash
pip install airflow-providers-dna
```

## Upstreaming

This implements the upstream `TODO` (`on_dag_run_success` / `on_dag_run_failed` →
`AirflowGenerator.complete_dataflow`). If/when an equivalent lands in
`acryl-datahub-airflow-plugin`, this provider can be retired.

