Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,22 @@ You can install such cross-provider dependencies when installing from PyPI. For
{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
{%- endif %}

{%- if OPTIONAL_DEPENDENCIES %}

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install {{ PACKAGE_PIP_NAME }}[{{ (OPTIONAL_DEPENDENCIES | list)[0] }}]


{{ OPTIONAL_DEPENDENCIES_TABLE_RST | safe }}
{%- endif %}

Downloading official packages
-----------------------------

Expand Down
40 changes: 40 additions & 0 deletions dev/breeze/tests/test_packages.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
apply_version_suffix_to_non_provider_pyproject_tomls,
apply_version_suffix_to_provider_pyproject_toml,
convert_cross_package_dependencies_to_table,
convert_optional_dependencies_to_table,
convert_pip_requirements_to_table,
expand_all_provider_distributions,
find_matching_long_package_names,
Expand All @@ -39,11 +40,13 @@
get_pip_package_name,
get_provider_details,
get_provider_info_dict,
get_provider_jinja_context,
get_provider_requirements,
get_removed_provider_ids,
get_short_package_name,
get_suspended_provider_folders,
get_suspended_provider_ids,
render_template,
validate_provider_info_with_runtime_schema,
)
from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH
Expand Down Expand Up @@ -317,6 +320,43 @@ def test_convert_cross_package_dependencies_to_table():
)


@pytest.mark.parametrize("markdown", [True, False])
def test_convert_optional_dependencies_to_table(markdown: bool):
optional_dependencies = {
"aiobotocore": ["aiobotocore>=3.0.0"],
"s3fs": ["s3fs>=2023.10.0", "boto3>=1.0"],
}
table = convert_optional_dependencies_to_table(optional_dependencies, markdown=markdown)
quote = "`" if markdown else "``"
assert "Extra" in table
assert "Dependencies" in table
assert f"{quote}aiobotocore{quote}" in table
assert f"{quote}aiobotocore>=3.0.0{quote}" in table
# Multiple dependencies for a single extra are comma-joined in one cell.
assert f"{quote}s3fs>=2023.10.0{quote}, {quote}boto3>=1.0{quote}" in table


def test_provider_index_template_renders_optional_dependencies():
# Amazon carries plain-PyPI extras (e.g. ``aiobotocore``) that are not cross-provider
# dependencies, so their only rendering path is the "Optional dependencies" section.
context = get_provider_jinja_context("amazon", current_release_version="1.0.0", version_suffix="")
rendered = render_template(
template_name="PROVIDER_INDEX", context=context, extension=".rst", keep_trailing_newline=True
)
assert "Optional dependencies\n---------------------" in rendered
assert "``aiobotocore``" in rendered


def test_provider_index_template_omits_optional_dependencies_when_none():
context = get_provider_jinja_context("amazon", current_release_version="1.0.0", version_suffix="")
context["OPTIONAL_DEPENDENCIES"] = {}
context["OPTIONAL_DEPENDENCIES_TABLE_RST"] = ""
rendered = render_template(
template_name="PROVIDER_INDEX", context=context, extension=".rst", keep_trailing_newline=True
)
assert "Optional dependencies\n---------------------" not in rendered


def test_get_provider_info_dict():
provider_info_dict = get_provider_info_dict("amazon")
assert provider_info_dict["name"] == "Amazon"
Expand Down
17 changes: 17 additions & 0 deletions providers/akeyless/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,23 @@ PIP package Version required
``akeyless`` ``>=5.0.0``
========================================== ==================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash
pip install apache-airflow-providers-akeyless[cloud_id]
============ =====================
Extra Dependencies
============ =====================
``cloud_id`` ``akeyless_cloud_id``
============ =====================

Downloading official packages
-----------------------------

Expand Down
35 changes: 35 additions & 0 deletions providers/amazon/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,41 @@ Dependent package
`apache-airflow-providers-ssh <https://airflow.apache.org/docs/apache-airflow-providers-ssh>`_ ``ssh``
======================================================================================================================== ====================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-amazon[aiobotocore]


==================== ============================================================================================================================================================
Extra Dependencies
==================== ============================================================================================================================================================
``aiobotocore`` ``aiobotocore>=3.0.0``
``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.2.0``
``s3fs`` ``s3fs>=2023.10.0``
``python3-saml`` ``python3-saml>=1.16.0; python_version < '3.13'``, ``xmlsec>=1.3.14; python_version < '3.13'``, ``lxml>=6.0.0; python_version < '3.13'``
``apache.hive`` ``apache-airflow-providers-apache-hive``
``exasol`` ``apache-airflow-providers-exasol``
``fab`` ``apache-airflow-providers-fab>=2.2.0``
``ftp`` ``apache-airflow-providers-ftp``
``google`` ``apache-airflow-providers-google``
``imap`` ``apache-airflow-providers-imap``
``microsoft.azure`` ``apache-airflow-providers-microsoft-azure``
``mongo`` ``apache-airflow-providers-mongo``
``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
``openlineage`` ``apache-airflow-providers-openlineage>=2.3.0``
``salesforce`` ``apache-airflow-providers-salesforce``
``ssh`` ``apache-airflow-providers-ssh``
``standard`` ``apache-airflow-providers-standard``
``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0``
``sqlalchemy`` ``sqlalchemy>=1.4.54``
==================== ============================================================================================================================================================

Downloading official packages
-----------------------------

Expand Down
19 changes: 19 additions & 0 deletions providers/anthropic/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,25 @@ PIP package Version required
``anthropic`` ``>=0.101.0``
========================================== ==================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-anthropic[bedrock]


=========== ===============================
Extra Dependencies
=========== ===============================
``bedrock`` ``anthropic[bedrock]>=0.101.0``
``vertex`` ``anthropic[vertex]>=0.101.0``
``aws`` ``anthropic[aws]>=0.101.0``
=========== ===============================

Downloading official packages
-----------------------------

Expand Down
17 changes: 17 additions & 0 deletions providers/apache/druid/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,23 @@ Dependent package
`apache-airflow-providers-apache-hive <https://airflow.apache.org/docs/apache-airflow-providers-apache-hive>`_ ``apache.hive``
============================================================================================================== ===============

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-apache-druid[apache.hive]


=============== ========================================
Extra Dependencies
=============== ========================================
``apache.hive`` ``apache-airflow-providers-apache-hive``
=============== ========================================

Downloading official packages
-----------------------------

Expand Down
24 changes: 24 additions & 0 deletions providers/apache/hive/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,30 @@ Dependent package
`apache-airflow-providers-vertica <https://airflow.apache.org/docs/apache-airflow-providers-vertica>`_ ``vertica``
====================================================================================================================== ===================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-apache-hive[amazon]


=================== =============================================================================================
Extra Dependencies
=================== =============================================================================================
``amazon`` ``apache-airflow-providers-amazon``
``microsoft.mssql`` ``apache-airflow-providers-microsoft-mssql``
``mysql`` ``apache-airflow-providers-mysql``
``presto`` ``apache-airflow-providers-presto``
``samba`` ``apache-airflow-providers-samba``
``sqlalchemy`` ``sqlalchemy>=1.4.54``
``vertica`` ``apache-airflow-providers-vertica``
``GSSAPI`` ``winkerberos>=0.7.0; sys_platform == "win32"``, ``kerberos>=1.3.0; sys_platform != "win32"``
=================== =============================================================================================

Downloading official packages
-----------------------------

Expand Down
18 changes: 18 additions & 0 deletions providers/apache/impala/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,24 @@ PIP package Version required
``apache-airflow`` ``>=2.11.0``
========================================== ==================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-apache-impala[kerberos]


============== ======================
Extra Dependencies
============== ======================
``kerberos`` ``kerberos>=1.3.0``
``sqlalchemy`` ``sqlalchemy>=1.4.54``
============== ======================

Downloading official packages
-----------------------------

Expand Down
18 changes: 18 additions & 0 deletions providers/apache/kafka/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,24 @@ Dependent package
`apache-airflow-providers-google <https://airflow.apache.org/docs/apache-airflow-providers-google>`_ ``google``
======================================================================================================================== ====================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-apache-kafka[google]


==================== ====================================================
Extra Dependencies
==================== ====================================================
``google`` ``apache-airflow-providers-google``
``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0``
==================== ====================================================

Downloading official packages
-----------------------------

Expand Down
19 changes: 19 additions & 0 deletions providers/apache/spark/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,25 @@ Dependent package
`apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_ ``cncf.kubernetes``
====================================================================================================================== ===================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-apache-spark[cncf.kubernetes]


=================== ===================================================
Extra Dependencies
=================== ===================================================
``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.4.0``
``openlineage`` ``apache-airflow-providers-openlineage``
``pyspark`` ``pyspark>=4.0.0``
=================== ===================================================

Downloading official packages
-----------------------------

Expand Down
17 changes: 17 additions & 0 deletions providers/celery/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,23 @@ Dependent package
`apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_ ``cncf.kubernetes``
====================================================================================================================== ===================

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-celery[cncf.kubernetes]


=================== ===================================================
Extra Dependencies
=================== ===================================================
``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.4.0``
=================== ===================================================

Downloading official packages
-----------------------------

Expand Down
32 changes: 32 additions & 0 deletions providers/common/ai/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,38 @@ Dependent package
`apache-airflow-providers-git <https://airflow.apache.org/docs/apache-airflow-providers-git>`_ ``git``
============================================================================================================ ==============

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-common-ai[anthropic]


============== ==========================================================================================================
Extra Dependencies
============== ==========================================================================================================
``anthropic`` ``pydantic-ai-slim[anthropic]``
``bedrock`` ``pydantic-ai-slim[bedrock]``
``google`` ``pydantic-ai-slim[google]``
``openai`` ``pydantic-ai-slim[openai]``
``mcp`` ``pydantic-ai-slim[mcp]``
``code-mode`` ``pydantic-ai-harness[codemode]>=0.3.0``
``skills`` ``apache-airflow-providers-git>=0.4.0``, ``pydantic-ai-skills>=0.11.0``
``avro`` ``fastavro>=1.10.0; python_version < "3.14"``, ``fastavro>=1.12.1; python_version >= "3.14"``
``parquet`` ``pyarrow>=18.0.0; python_version < '3.14'``, ``pyarrow>=22.0.0; python_version >= '3.14'``
``sql`` ``apache-airflow-providers-common-sql``, ``sqlglot>=30.0.0``
``common.sql`` ``apache-airflow-providers-common-sql``
``langchain`` ``langchain>=1.0.0``
``llamaindex`` ``llama-index-core>=0.13.0``, ``llama-index-embeddings-openai>=0.6.0``, ``llama-index-llms-openai>=0.6.0``
``pdf`` ``pypdf>=4.0.0``
``docx`` ``python-docx>=1.0.0``
``git`` ``apache-airflow-providers-git``
============== ==========================================================================================================

Downloading official packages
-----------------------------

Expand Down
18 changes: 18 additions & 0 deletions providers/common/compat/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,24 @@ Dependent package
`apache-airflow-providers-openlineage <https://airflow.apache.org/docs/apache-airflow-providers-openlineage>`_ ``openlineage``
============================================================================================================== ===============

Optional dependencies
---------------------

These extras install optional third-party libraries that enable additional features of the provider.
Install them when installing from PyPI. For example:

.. code-block:: bash

pip install apache-airflow-providers-common-compat[openlineage]


=============== ========================================
Extra Dependencies
=============== ========================================
``openlineage`` ``apache-airflow-providers-openlineage``
``standard`` ``apache-airflow-providers-standard``
=============== ========================================

Downloading official packages
-----------------------------

Expand Down
Loading
Loading