Skip to content

Add IBM Db2 provider with Hook #67532

Open
Amitkumar293 wants to merge 12 commits into
apache:mainfrom
Amitkumar293:airflow-db2-adapter
Open

Add IBM Db2 provider with Hook #67532
Amitkumar293 wants to merge 12 commits into
apache:mainfrom
Amitkumar293:airflow-db2-adapter

Conversation

@Amitkumar293

@Amitkumar293 Amitkumar293 commented May 26, 2026

Copy link
Copy Markdown

Description

This PR adds a new provider for IBM Db2 database support in Apache Airflow.

What: New Db2 provider package with Hook and Operator
Why: Enable Airflow users to connect to and execute queries against IBM Db2 databases

Changes

  • Db2Hook: Extends DbApiHook for IBM Db2 connectivity with SSL support
  • Added Db2Dialect: - Mentioned the SQLAlchemy dialect component
  • Comprehensive unit tests (Tests covering all functionality)
  • Documentation and example DAGs
  • Provider metadata and configuration

Testing

  • ✅ All unit tests passing
  • ✅ Ruff linting and formatting checks passed
  • ✅ Follows Airflow coding standards

@boring-cyborg

boring-cyborg Bot commented May 26, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@Amitkumar293 Amitkumar293 marked this pull request as ready for review May 27, 2026 05:59
Comment thread .github/boring-cyborg.yml Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Note, that some fomality with announcement/DISCUSS is neede to add a new provider, following https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst

@Amitkumar293

Copy link
Copy Markdown
Author

Note, that some fomality with announcement/DISCUSS is neede to add a new provider, following https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst

Thanks for pointing this out.

I’ve created a DISCUSS thread on the dev@airflow.apache.org mailing list for the new IBM Db2 provider, following the ACCEPTING_PROVIDERS.rst process.

Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/hooks/__init__.py
@github-actions

github-actions Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #68322 ("[main] Upgrade important CI environment"), commit 227506a and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from 5d470ba to 7b1633e Compare June 1, 2026 10:03
Comment thread scripts/ci/docker-compose/tests-sources.yml Outdated
Comment thread providers/ibm/db2/docs/commits.rst Outdated
Comment thread providers/ibm/db2/docs/example-dags.rst Outdated
Comment thread providers/ibm/db2/docs/installing-providers-from-sources.rst Outdated
Comment thread providers/ibm/db2/docs/security.rst Outdated
Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/operators/db2.py Outdated
Comment thread providers/ibm/db2/provider.yaml
@Amitkumar293

Copy link
Copy Markdown
Author

@jscheffl Thanks for your review, very helpful. I am working on the review comments.

@eladkal

eladkal commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from 0844797 to 23f0976 Compare June 8, 2026 12:06
@Amitkumar293

Copy link
Copy Markdown
Author

Was there a mailing list thread about this new provider? https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst#examples-of-current-process

Yes, I opened a mailing list thread for the new provider last week.

@Amitkumar293

Copy link
Copy Markdown
Author

@jscheffl Fixed the provider structure per your feedback: added dialect support, restructured the docs, removed operators, and added UI connection fields in 'provider.yaml'.

Thank you for the detailed review. Could you please take another look when you have a chance?

@eladkal

eladkal commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Was there a mailing list thread about this new provider? https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst#examples-of-current-process

Yes, I opened a mailing list thread for the new provider last week.

Can you share a link to the mailing list thread?

@Amitkumar293

Amitkumar293 commented Jun 9, 2026

Copy link
Copy Markdown
Author

Was there a mailing list thread about this new provider? https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst#examples-of-current-process

Yes, I opened a mailing list thread for the new provider last week.

Can you share a link to the mailing list thread?

@eladkal
Here is the link to the mailing list thread: https://lists.apache.org/list?dev@airflow.apache.org:lte=1M:Db2

@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from 23f0976 to 951db05 Compare June 11, 2026 09:46
Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/hooks/db2.py Outdated
Comment thread providers/ibm/db2/src/airflow/providers/ibm/db2/hooks/db2.py Outdated
Comment thread providers/ibm/db2/README.rst Outdated
@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from 16837e2 to d42e710 Compare June 11, 2026 13:14
@jscheffl

Copy link
Copy Markdown
Contributor

Can you please resolve comments if your think they are completed?

@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from d42e710 to 6b9a3e7 Compare June 12, 2026 06:01
Amit Kumar and others added 11 commits June 12, 2026 12:05
Implements initial IBM Db2 provider package with:
- Db2Hook: Database connectivity with SSL support
- Db2Operator: SQL execution with Jinja templating
- Unit tests for both hook and operator
- Example DAG demonstrating usage
- Add Db2-specific connection test SQL using SYSIBM.SYSDUMMY1
- Add cleanup task to example DAG for idempotent runs
- Add autocommit=True to all DDL/DML operations in example DAG
- Ensure proper transaction handling for Db2 database operations
- Implement Db2Hook extending DbApiHook for IBM Db2 connectivity
- Add Db2Operator for SQL execution with parameter binding
- Include comprehensive unit tests for Hook and Operator
- Add documentation, example DAGs, and provider metadata
- Support SSL connections, autocommit, and batch execution
Move provider from `providers/db2/` to `providers/ibm/db2/` and update package name from `apache-airflow-providers-db2` to `apache-airflow-providers-ibm-db2`. Updates all import paths, documentation, tests, and workspace configuration.
- Extract get_conn_str() method for better separation
- Move metadata methods to Dialect with pure SQL
@Amitkumar293 Amitkumar293 force-pushed the airflow-db2-adapter branch from cc77413 to ef25c93 Compare June 12, 2026 06:37
@Amitkumar293 Amitkumar293 requested a review from dabla June 15, 2026 06:19
@Amitkumar293 Amitkumar293 changed the title Add IBM Db2 provider with Hook and Operator Add IBM Db2 provider with Hook Jun 15, 2026
@Amitkumar293

Copy link
Copy Markdown
Author

@dabla Could you please review it again? I've made the changes you suggested. I appreciate your help and thank you for your time.

@Amitkumar293

Copy link
Copy Markdown
Author

Can you please resolve comments if your think they are completed?

Hi @jscheffl
Could you please let me know what is still remaining on my side in order to get this PR approved?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants