Add providers E2E tests framework and OpenLineage tests

[kacpermuda]

Add framework for providers E2E tests and implement OpenLineage provider e2e tests to Airflow CI

The OpenLineage provider emits lineage events when Dags run, but until now there was no automated way in Airflow CI to verify those events are actually correct against a real, deployed Airflow stack. This PR introduces that capability: a self-contained e2e test harness that spins up Airflow via docker-compose, runs the provider system-test Dags, and asserts the OpenLineage events emitted by the transport match expected payloads. The tests run on demand (via workflow_dispatch) and are triggered automatically by selective checks when the openlineage or common providers, the providers-e2e-tests harness, or related files change.

The approach follows the same pattern already established by the Task SDK integration tests and the Airflow e2e tests: a standalone directory under providers-e2e-tests/openlineage/ with its own pyproject.toml, docker-compose.yaml, and pytest suite, invoked through a new breeze testing providers-e2e-tests <provider> command.

More providers can be added in the future if needed. These OL test do not rely on any external service, so they can easily be run in CI (no db, external api, google/amazon/azure service needed).

What's included:

• providers-e2e-tests/ — new top-level directory; each provider gets its own subdirectory with a
  docker-compose stack, tests, and pyproject.toml.
• breeze testing providers-e2e-tests <provider> — spins up the stack and runs pytest.
• Two modes: default PROD image, or --airflow-version <ver> to test older Airflow + current providers
  from main (builds apache/airflow:<ver> + provider wheels). Required providers are declared in each
  provider's pyproject.toml under [tool.e2e-tests] required-providers — nothing hardcoded in breeze.
• OpenLineage implementation: postgres + API server + scheduler + dag-processor + triggerer + worker.
  DAGs come from the existing providers/openlineage/tests/system/openlineage/ — prepare_dags.py
  copies and strips their pytest-only footers at runtime. Events are captured via VariableTransport
  (stores OL events in Airflow Variables); each DAG's OpenLineageTestOperator validates them — no
  external backend needed. A run that ends success means lineage matched.
• Selective checks trigger the suite on openlineage / common-providers changes. Core and task-sdk changes are not wired in - a regression there wouldn't trigger these tests on the PR, only on the next canary run. Adding core/task-sdk to the file group would work technically (PROD image is already built when those change via other test paths), but it would widen the trigger surface significantly. Leaving it for discussion.
• CI: PROD and compat matrix suite on every relevant PR and canary/main; the compat matrix (3.0.6 / 3.1.8 / 3.2.2) is reused directly from PROVIDERS_COMPATIBILITY_TESTS_MATRIX in breeze's global_constants.py - the same source as the provider unit compat tests - so adding a new Airflow release there automatically picks up a new e2e compat run. The 2.x entry is skipped via a startsWith('3.') condition since the e2e stack requires the Airflow 3 API server (we will bump min AF version for providers soon anyway, so did not waste time making it work for AF2).

I tried to add a new breeze command for this, but it's first time ever for me contributing in this area, so it may not be ideal, tried to follow an example of current code.

When tests are failing, there is a nice log summary for what exactly failed so we don't have to get zipped logs every time. Here is how it looks like (it's how the TestOperator in system tests logs failures, we can improve that as we go to be better, the idea here is that we include up to 100 lines of warning, error and critical logs from failed tasks for easier debugging)

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Sonnet 4.6 following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[github-actions]

uv.lock on main just moved via #69303 ("[main] Upgrade important CI environment"), commit 1fa6b4f and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

[shahar1]

Overall LGTM, minor comments and nits.

[potiuk]

Thanks @kacpermuda — this is a well-built, well-documented framework. Reviewed the whole diff: it's test/CI-only (no production code), the providers/openlineage/tests/system/** edits are uniform and keep those DAGs as the single source of truth, and the design is empirically validated by this PR's own green CI — the PROD run plus every compat variant (3.0.6 → 3.3.0, and the 2.x skip path) all pass.

Particularly liked: the selective-checks gating + forced PROD image build, the "adding a new provider" runbook in the README, and the honest handling of first-run flakiness (retry-once with a surfaced ⚠ notice rather than hiding it).

A couple of purely-cosmetic nits, non-blocking: the fixed time.sleep(15) in the auth_headers fixture is a touch fragile given compose.up(wait=True) already gates readiness, and wait_for_airflow_api() uses a bare requests.get while its siblings use the retry-mounted self.session. Neither needs to hold up merge.

shahar1's comments look addressed. LGTM 👍

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Airflow maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Airflow handles maintainer review: contributing-docs/05_pull_requests.rst.

[potiuk]

Let's see if that will be "bearable" in terms of stablity and build time.

[kacpermuda]

the fixed time.sleep(15) in the auth_headers fixture is a touch fragile given compose.up(wait=True) already gates readiness, and wait_for_airflow_api() uses a bare requests.get while its siblings use the retry-mounted self.session. Neither needs to hold up merge.

Agreed, this is probably to many safety checks for that. I've ported some of the code from the actual e2e tests I've been running on 3rd party infra for some time, where there is much more instability of infra causing flakiness, so I might have been too cautious here. Let's keep it for now if it's okay, will be happy to adjust soon if we see these tests taking up to much time or just during my next update to this framework (I'll leave a note to myself about cleaning this)

[kacpermuda]

Let's see if that will be "bearable" in terms of stablity and build time.

Sure, I'll try to keep an eye for scheduled runs failures or any flakiness, but feel free to ping me if it goes unnoticed

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker e756975 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

[jason810496]

Sorry to come here late. May I ask what is the purpose of introducing another providers-e2e-tests directory? Since the existing airflow-e2e-tests already includes the providers, for examples: The ES and OS provider for remote logging feature, the Kafka provider for external event driven feature.

https://github.com/apache/airflow/blob/dc5ddd26202f972f634a6ee9ce25f04f17669ad7/airflow-e2e-tests/tests/airflow_e2e_tests/

Would it be better to consolidate there to reuse all the setup instead of re-introducing a new test suite?

[shahar1]

Sorry to come here late. May I ask what is the purpose of introducing another providers-e2e-tests directory? Since the existing airflow-e2e-tests already includes the providers, for examples: The ES and OS provider for remote logging feature, the Kafka provider for external event driven feature.

https://github.com/apache/airflow/blob/dc5ddd26202f972f634a6ee9ce25f04f17669ad7/airflow-e2e-tests/tests/airflow_e2e_tests/

Would it be better to consolidate there to reuse all the setup instead of re-introducing a new test suite?

Good catch - didn't notice that all e2e tests are already there.
As a first step it might be indeed better to consolidate them.
However, long term-wise - maybe they should reside in providers/<provider>/tests/e2e/?

[kacpermuda]

However, long term-wise - maybe they should reside in providers//tests/e2e/?

Yup, that would probably be the cleanest solution