From 0ebfaa330c061db1da30bcba6c0f79bb40674db8 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Tue, 8 Aug 2023 15:46:25 +0200 Subject: [PATCH] Better fix for add-back-references This is a better fix for add-back-references breeze command, as the previous fix had a bug and processed more than just providers when providers --gen-type was selected. See #33149. This version fixes it in a much more consistent way: * you have to specify what you want to work on as option * you can select few of the provider via auto-complete * you can also select apache-airflow and helm-chart * you can mix and match them if you need All the scripts and references where `--gen-type` was used previously have been updated and documentation was changed to reflect it --- BREEZE.rst | 26 +++-- dev/README_RELEASE_AIRFLOW.md | 2 +- dev/README_RELEASE_HELM_CHART.md | 2 +- dev/README_RELEASE_PROVIDER_PACKAGES.md | 2 +- .../commands/developer_commands.py | 4 +- .../commands/release_management_commands.py | 43 +++---- .../release_management_commands_config.py | 2 +- .../src/airflow_breeze/global_constants.py | 14 ++- .../utils/add_back_references.py | 37 +++--- .../airflow_breeze/utils/common_options.py | 10 +- dev/breeze/tests/test_global_constants.py | 6 +- .../publish_provider_documentation.sh | 2 + images/breeze/output-commands-hash.txt | 14 +-- images/breeze/output-commands.svg | 108 ++++++++--------- images/breeze/output_build-docs.svg | 84 ++++++------- images/breeze/output_release-management.svg | 52 ++++----- ...release-management_add-back-references.svg | 80 ++++++------- ...ement_generate-issue-content-providers.svg | 86 +++++++------- ...agement_prepare-provider-documentation.svg | 92 ++++++++------- ...e-management_prepare-provider-packages.svg | 78 +++++++------ ...output_release-management_publish-docs.svg | 110 +++++++++--------- 21 files changed, 424 insertions(+), 430 deletions(-) diff --git a/BREEZE.rst b/BREEZE.rst index f060b84ee2bec..45af0d7091073 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -2034,23 +2034,35 @@ To add back references to the documentation generated by ``build-docs`` in Breez use the ``release-management add-back-references`` command. This is important to support backward compatibility the airflow documentation. -.. code-block:: bash - - breeze release-management add-back-references +You have to specify which packages you run it on. For example you can run it for all providers: .. code-block:: bash - release-management add-back-references --airflow-site-directory DIRECTORY + release-management add-back-references --airflow-site-directory DIRECTORY all-providers The flag ``--airflow-site-directory`` takes the path of the cloned ``airflow-site``. The command will not proceed if this is an invalid path. +You can also run the command for apache-airflow (core documentation): + +.. code-block:: bash + + breeze release-management publish-docs --airflow-site-directory DIRECTORY apache-airflow + +Also for helm-chart package: + +.. code-block:: bash + + breeze release-management publish-docs --airflow-site-directory DIRECTORY helm-chart + + +You can also manually specify (it's auto-completable) list of packages to run the command for including individual +providers - you can mix apache-airflow, helm-chart and provider packages this way: + .. code-block:: bash - breeze release-management publish-docs --gen-type [airflow | providers | helm] + breeze release-management publish-docs --airflow-site-directory DIRECTORY apache.airflow apache.beam google -The flag ``--gen-type`` is a string of which type of back references to generate. It can take the values of -airflow, providers, or helm. The command will fail if this is incorrect. Those are all available flags of ``release-management add-back-references`` command: diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index 53e59c8a71fc3..778565edadf31 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -818,7 +818,7 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo ```shell script breeze release-management publish-docs --package-filter apache-airflow --package-filter docker-stack - breeze release-management add-back-references --gen-type airflow + breeze release-management add-back-references apache-airflow git add . git commit -m "Add documentation for Apache Airflow ${VERSION}" git push diff --git a/dev/README_RELEASE_HELM_CHART.md b/dev/README_RELEASE_HELM_CHART.md index 01994e6241a00..7e0f65f9bcb98 100644 --- a/dev/README_RELEASE_HELM_CHART.md +++ b/dev/README_RELEASE_HELM_CHART.md @@ -633,7 +633,7 @@ between the two repositories to be able to build the documentation. Regenerate `index.yaml` so it can be added to the Airflow website to allow: `helm repo add https://airflow.apache.org`. ```shell - breeze release-management add-back-references --gen-type helm + breeze release-management add-back-references helm-chart curl https://dist.apache.org/repos/dist/dev/airflow/helm-chart/$RC/index.yaml -o index.yaml cp ${AIRFLOW_SVN_RELEASE_HELM}/${VERSION}/airflow-${VERSION}.tgz . helm repo index --merge ./index.yaml . --url "https://downloads.apache.org/airflow/helm-chart/$VERSION" diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md index 5e4016746bc8b..9c6196f6e430e 100644 --- a/dev/README_RELEASE_PROVIDER_PACKAGES.md +++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md @@ -392,7 +392,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \ --override-versioned -breeze release-management add-back-references --gen-type providers +breeze release-management add-back-references all-providers ``` If you see `ModuleNotFoundError: No module named 'docs'`, set: diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py index f29584bb32667..71c6fc85e04b3 100644 --- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py @@ -34,7 +34,7 @@ DEFAULT_PYTHON_MAJOR_MINOR_VERSION, DOCKER_DEFAULT_PLATFORM, MOUNT_SELECTED, - get_available_documentation_provider_packages, + get_available_documentation_packages, ) from airflow_breeze.params.build_ci_params import BuildCiParams from airflow_breeze.params.doc_build_params import DocBuildParams @@ -362,7 +362,7 @@ def start_airflow( @click.option( "--package-filter", help="List of packages to consider.", - type=NotVerifiedBetterChoice(get_available_documentation_provider_packages()), + type=NotVerifiedBetterChoice(get_available_documentation_packages()), multiple=True, ) @click.option( diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py index aca5f66395bb1..7ba67c7943b3c 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py @@ -43,16 +43,16 @@ MOUNT_ALL, MOUNT_SELECTED, MULTI_PLATFORM, - get_available_documentation_provider_packages, + get_available_documentation_packages, ) from airflow_breeze.params.shell_params import ShellParams from airflow_breeze.utils.add_back_references import ( - GenerationType, start_generating_back_references, ) from airflow_breeze.utils.ci_group import ci_group from airflow_breeze.utils.common_options import ( argument_packages, + argument_packages_plus_all_providers, option_airflow_constraints_mode_ci, option_airflow_constraints_reference, option_airflow_extras, @@ -783,7 +783,7 @@ def alias_image(image_from: str, image_to: str): @click.option( "--package-filter", help="List of packages to consider.", - type=NotVerifiedBetterChoice(get_available_documentation_provider_packages()), + type=NotVerifiedBetterChoice(get_available_documentation_packages()), multiple=True, ) @option_verbose @@ -815,7 +815,7 @@ def publish_docs( @release_management.command( name="add-back-references", - help="Command to add back references for documentation to make it backward compatible", + help="Command to add back references for documentation to make it backward compatible.", ) @click.option( "-a", @@ -825,23 +825,12 @@ def publish_docs( help="Local directory path of cloned airflow-site repo.", required=True, ) -@click.option( - "-g", - "--gen-type", - show_default=True, - help="Type of back references to generate. Forced to providers if providers specified as arguments.", - type=BetterChoice( - [e.name for e in GenerationType], - ), - default=GenerationType.airflow.name, -) -@argument_packages +@argument_packages_plus_all_providers @option_verbose @option_dry_run def add_back_references( airflow_site_directory: str, - gen_type: str, - packages: list[str], + packages_plus_all_providers: tuple[str], ): """Adds back references for documentation generated by build-docs and publish-docs""" site_path = Path(airflow_site_directory) @@ -851,22 +840,16 @@ def add_back_references( "Provide the path of cloned airflow-site repo\n" ) sys.exit(1) - if len(packages) != 0 and gen_type != GenerationType.providers.name: + if len(packages_plus_all_providers) == 0: get_console().print( - [ - f"[warning]Forcing gen type to " - f"{GenerationType.providers} as some provider_packages are selected." - ] - ) - gen_type = GenerationType.providers.name - gen = GenerationType[gen_type] - if gen not in GenerationType: - get_console().print( - "\n[error]invalid type of doc generation required. Pass one of [airflow | providers | helm]\n" + "\n[error]You need to specify at least one package to generate back references for\n" ) sys.exit(1) - - start_generating_back_references(gen, site_path, packages) + packages = list(packages_plus_all_providers) + if "all-providers" in packages_plus_all_providers: + packages.remove("all-providers") + packages.extend(get_available_documentation_packages(only_providers=True, short_version=True)) + start_generating_back_references(site_path, packages) @release_management.command( diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py index adc67d7f1fc85..6ae328445fa74 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py @@ -174,7 +174,7 @@ "breeze release-management add-back-references": [ { "name": "Add Back References to Docs", - "options": ["--airflow-site-directory", "--gen-type"], + "options": ["--airflow-site-directory"], }, ], "breeze release-management generate-issue-content-providers": [ diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py index dadae7a2e4db5..04aca0df16022 100644 --- a/dev/breeze/src/airflow_breeze/global_constants.py +++ b/dev/breeze/src/airflow_breeze/global_constants.py @@ -145,18 +145,20 @@ def all_helm_test_packages() -> list[str]: ALL_HISTORICAL_PYTHON_VERSIONS = ["3.6", "3.7", "3.8", "3.9", "3.10", "3.11"] -def get_available_documentation_provider_packages(short_version=False) -> list[str]: +def get_available_documentation_packages(short_version=False, only_providers: bool = False) -> list[str]: provider_names: list[str] = list(json.loads(PROVIDER_DEPENDENCIES_JSON_FILE_PATH.read_text()).keys()) doc_provider_names = [provider_name.replace(".", "-") for provider_name in provider_names] - available_packages = [f"apache-airflow-providers-{doc_provider}" for doc_provider in doc_provider_names] - available_packages.extend(["apache-airflow", "docker-stack", "helm-chart"]) - available_packages.sort() + available_packages = [] + if not only_providers: + available_packages.extend(["apache-airflow", "docker-stack", "helm-chart"]) + all_providers = [f"apache-airflow-providers-{doc_provider}" for doc_provider in doc_provider_names] + all_providers.sort() + available_packages.extend(all_providers) if short_version: prefix_len = len("apache-airflow-providers-") available_packages = [ - package[prefix_len:].replace("-", ".") + package[prefix_len:].replace("-", ".") if len(package) > prefix_len else package for package in available_packages - if len(package) > prefix_len ] return available_packages diff --git a/dev/breeze/src/airflow_breeze/utils/add_back_references.py b/dev/breeze/src/airflow_breeze/utils/add_back_references.py index 0b7c2abfe8907..ebc0032acdd5e 100644 --- a/dev/breeze/src/airflow_breeze/utils/add_back_references.py +++ b/dev/breeze/src/airflow_breeze/utils/add_back_references.py @@ -16,7 +16,6 @@ # under the License. from __future__ import annotations -import enum import os import re import tempfile @@ -26,7 +25,7 @@ from rich import print -from airflow_breeze.global_constants import get_available_documentation_provider_packages +from airflow_breeze.utils.console import get_console airflow_redirects_link = ( "https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/redirects.txt" @@ -34,13 +33,6 @@ helm_redirects_link = "https://raw.githubusercontent.com/apache/airflow/main/docs/helm-chart/redirects.txt" -# types of generations supported -class GenerationType(enum.Enum): - airflow = 1 - helm = 2 - providers = 3 - - def download_file(url): try: temp_dir = Path(tempfile.mkdtemp(prefix="temp_dir", suffix="")) @@ -149,26 +141,23 @@ def generate_back_references(link: str, base_path: Path): create_back_reference_html(relative_path, dest_file_path) -def start_generating_back_references( - gen_type: GenerationType, airflow_site_directory: Path, short_provider_package_ids: list[str] -): - # Either packages or gen_type should be provided +def start_generating_back_references(airflow_site_directory: Path, short_provider_package_ids: list[str]): docs_archive_path = airflow_site_directory / "docs-archive" airflow_docs_path = docs_archive_path / "apache-airflow" helm_docs_path = docs_archive_path / "helm-chart" - - if gen_type == GenerationType.airflow: + if "apache-airflow" in short_provider_package_ids: generate_back_references(airflow_redirects_link, airflow_docs_path) - elif gen_type == GenerationType.helm: + short_provider_package_ids.remove("apache-airflow") + if "helm-chart" in short_provider_package_ids: generate_back_references(helm_redirects_link, helm_docs_path) - elif gen_type == GenerationType.providers: - if short_provider_package_ids: - all_providers = [ - f"apache-airflow-providers-{package.replace('.','-')}" - for package in short_provider_package_ids - ] - else: - all_providers = get_available_documentation_provider_packages() + short_provider_package_ids.remove("helm-chart") + if "docker-stack" in short_provider_package_ids: + get_console().print("[info]Skipping docker-stack package. No back-reference needed.") + short_provider_package_ids.remove("docker-stack") + if short_provider_package_ids: + all_providers = [ + f"apache-airflow-providers-{package.replace('.','-')}" for package in short_provider_package_ids + ] for p in all_providers: print(f"Processing airflow provider: {p}") generate_back_references(get_github_redirects_url(p), docs_archive_path / p) diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py b/dev/breeze/src/airflow_breeze/utils/common_options.py index bd039593a51b1..3e63ce0e70b8e 100644 --- a/dev/breeze/src/airflow_breeze/utils/common_options.py +++ b/dev/breeze/src/airflow_breeze/utils/common_options.py @@ -43,7 +43,7 @@ SINGLE_PLATFORMS, START_AIRFLOW_ALLOWED_EXECUTORS, START_AIRFLOW_DEFAULT_ALLOWED_EXECUTORS, - get_available_documentation_provider_packages, + get_available_documentation_packages, ) from airflow_breeze.utils.custom_param_types import ( AnswerChoice, @@ -448,7 +448,13 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option, "packages", nargs=-1, required=False, - type=BetterChoice(get_available_documentation_provider_packages(short_version=True)), + type=BetterChoice(get_available_documentation_packages(short_version=True)), +) +argument_packages_plus_all_providers = click.argument( + "packages_plus_all_providers", + nargs=-1, + required=False, + type=BetterChoice(["all-providers"] + get_available_documentation_packages(short_version=True)), ) option_airflow_constraints_reference = click.option( "--airflow-constraints-reference", diff --git a/dev/breeze/tests/test_global_constants.py b/dev/breeze/tests/test_global_constants.py index f9878f7914423..ee0f72888ef1f 100644 --- a/dev/breeze/tests/test_global_constants.py +++ b/dev/breeze/tests/test_global_constants.py @@ -16,12 +16,12 @@ # under the License. from __future__ import annotations -from airflow_breeze.global_constants import get_available_documentation_provider_packages +from airflow_breeze.global_constants import get_available_documentation_packages AVAILABLE_PACKAGES_STARTING_LIST = ("apache-airflow", "helm-chart", "docker-stack") def test_get_available_packages(): - assert len(get_available_documentation_provider_packages()) > 70 - for package in get_available_documentation_provider_packages(): + assert len(get_available_documentation_packages()) > 70 + for package in get_available_documentation_packages(): assert package.startswith(AVAILABLE_PACKAGES_STARTING_LIST) diff --git a/dev/provider_packages/publish_provider_documentation.sh b/dev/provider_packages/publish_provider_documentation.sh index bf8eaca1fb577..0616a824905a8 100755 --- a/dev/provider_packages/publish_provider_documentation.sh +++ b/dev/provider_packages/publish_provider_documentation.sh @@ -30,4 +30,6 @@ done breeze build-docs "${provider_filters[@]}" +breeze release-management add-back-references "${@}" + cd "${AIRFLOW_SITE_DIRECTORY}" diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt index df67b3627b434..173e2b1982e72 100644 --- a/images/breeze/output-commands-hash.txt +++ b/images/breeze/output-commands-hash.txt @@ -2,7 +2,7 @@ # Please do not solve it but run `breeze setup regenerate-command-images`. # This command should fix the conflict and regenerate help images that you have conflict with. main:344261ca3aa7ff31e098b1d88280566a -build-docs:45ccfec7bb1bd73eb635c3363d0353a6 +build-docs:bfc91db55c595516fe84d56c96a5a8e0 ci:fix-ownership:3e5a73533cc96045e72cb258783cfc96 ci:free-space:49af17b032039c05c41a7a8283f365cc ci:get-workflow-info:8246038093359b9c3c110043419473e2 @@ -35,21 +35,21 @@ prod-image:build:85bb45de92e86a55474238c3f04def92 prod-image:pull:76f1f27e6119928412abecf153fce4bb prod-image:verify:bd2b78738a7c388dbad6076c41a9f906 prod-image:7b3369e182724fc155b3399c95d0fd73 -release-management:add-back-references:0d4eb5ed82e5381bc630b343ba605a72 +release-management:add-back-references:919b3bd706acf794e0e5e129fc587f6d release-management:create-minor-branch:a3834afc4aa5d1e98002c9e9e7a9931d release-management:generate-constraints:b8fcaf8f0acd35ed5dbd48659bdb6485 -release-management:generate-issue-content-providers:6b0d954cb6dbdec0da0a7988feec58f0 +release-management:generate-issue-content-providers:dc7149a2c6fc43d323e7c83865c0ca0b release-management:generate-providers-metadata:d4e8e5cfaa024e3963af02d7a873048d release-management:install-provider-packages:a89493be1ae961c13469b5a25a605069 release-management:prepare-airflow-package:85d01c57e5b5ee0fb9e5f9d9706ed3b5 -release-management:prepare-provider-documentation:b8defe8b94bf790d88d055c3e4c0ffba -release-management:prepare-provider-packages:4599a06e636c6fe8fda41b5a751c1429 -release-management:publish-docs:00eae3f9acc2134c14c9a94d789423ce +release-management:prepare-provider-documentation:71dd952212206ca5c3b24bf8deaf13d6 +release-management:prepare-provider-packages:ed727969a3fa27e41c75c08381c9ebf4 +release-management:publish-docs:6d8ea0362b7a4394d8367e2105516cf8 release-management:release-prod-images:cfbfe8b19fee91fd90718f98ef2fd078 release-management:start-rc-process:b27bd524dd3c89f50a747b60a7e892c1 release-management:start-release:419f48f6a4ff4457cb9de7ff496aebbe release-management:verify-provider-packages:96dce5644aad6b37080acf77b3d8de3a -release-management:dc4897917210deefb4338f2038f8cb33 +release-management:376b788c9d4d2d431d391313482fe791 sbom:generate-provider-requirements:9abe53200ea5f40e0bf7c27f6087f27f sbom:update-sbom-information:0ce56884e5f842e3e80d6619df1ccc64 sbom:935d041028e847d3faf763a95b51063e diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg index 38d9a6c3665ee..65ed6cdbb0c21 100644 --- a/images/breeze/output-commands.svg +++ b/images/breeze/output-commands.svg @@ -35,8 +35,8 @@ .breeze-help-r1 { fill: #c5c8c6;font-weight: bold } .breeze-help-r2 { fill: #c5c8c6 } .breeze-help-r3 { fill: #d0b344;font-weight: bold } -.breeze-help-r4 { fill: #868887 } -.breeze-help-r5 { fill: #68a0b3;font-weight: bold } +.breeze-help-r4 { fill: #68a0b3;font-weight: bold } +.breeze-help-r5 { fill: #868887 } .breeze-help-r6 { fill: #98a84b;font-weight: bold } .breeze-help-r7 { fill: #8d7b39 } @@ -217,59 +217,59 @@ -Usage: breeze [OPTIONS] COMMAND [ARGS]... +Usage: breeze [OPTIONS] COMMAND [ARGS]... -╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--python-pPython major/minor version used in Airflow image for images.(>3.8< | 3.9 | 3.10 | 3.11)│ -│[default: 3.8]                                              │ -│--backend-bDatabase backend to use.(>sqlite< | mysql | postgres | mssql)[default: sqlite]│ -│--postgres-version-PVersion of Postgres used.(>11< | 12 | 13 | 14 | 15)[default: 11]│ -│--mysql-version-MVersion of MySQL used.(>5.7< | 8)[default: 5.7]│ -│--mssql-version-SVersion of MsSQL used.(>2017-latest< | 2019-latest)[default: 2017-latest]│ -│--integrationIntegration(s) to enable when running (can be more than one).                            │ -│(all | all-testable | cassandra | celery | kafka | kerberos | mongo | otel | pinot |     │ -│statsd | statsd | trino)                                                                 │ -│--forward-credentials-fForward local credentials to container when running.│ -│--db-reset-dReset DB when entering the container.│ -│--max-timeMaximum time that the command should take - if it takes longer, the command will fail.│ -│(INTEGER RANGE)                                                                       │ -│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ -│--builderBuildx builder used to perform `docker buildx build` commands.(TEXT)│ -│[default: autodetect]                                         │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--verbose-vPrint verbose information about performed steps.│ -│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ -│--answer-aForce answer to questions.(y | n | q | yes | no | quit)│ -│--help-hShow this message and exit.│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Developer commands ─────────────────────────────────────────────────────────────────────────────────────────────────╮ -│start-airflow     Enter breeze environment and starts all Airflow components in the tmux session. Compile assets  │ -│if contents of www directory changed.                                                           │ -│static-checks     Run static checks.                                                                              │ -│build-docs        Build documents.                                                                                │ -│down              Stop running breeze environment.                                                                │ -│shell             Enter breeze environment. this is the default command use when no other is selected.            │ -│exec              Joins the interactive shell of running airflow container.                                       │ -│compile-www-assetsCompiles www assets.                                                                            │ -│cleanup           Cleans the cache of parameters, docker cache and optionally built CI/PROD images.               │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Testing commands ───────────────────────────────────────────────────────────────────────────────────────────────────╮ -│testing        Tools that developers can use to run tests                                                         │ -│k8s            Tools that developers use to run Kubernetes tests                                                  │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Image commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ci-image         Tools that developers can use to manually manage CI images                                       │ -│prod-image       Tools that developers can use to manually manage PROD images                                     │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Release management commands ────────────────────────────────────────────────────────────────────────────────────────╮ -│release-management     Tools that release managers can use to prepare and manage Airflow releases                 │ -│sbom                   Tools that release managers can use to prepare sbom information                            │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Other commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│setup     Tools that developers can use to configure Breeze                                                       │ -│ci        Tools that CI workflows use to cleanup/manage CI environment                                            │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--python-pPython major/minor version used in Airflow image for images.(>3.8< | 3.9 | 3.10 | 3.11)│ +│[default: 3.8]                                              │ +│--backend-bDatabase backend to use.(>sqlite< | mysql | postgres | mssql)[default: sqlite]│ +│--postgres-version-PVersion of Postgres used.(>11< | 12 | 13 | 14 | 15)[default: 11]│ +│--mysql-version-MVersion of MySQL used.(>5.7< | 8)[default: 5.7]│ +│--mssql-version-SVersion of MsSQL used.(>2017-latest< | 2019-latest)[default: 2017-latest]│ +│--integrationIntegration(s) to enable when running (can be more than one).                            │ +│(all | all-testable | cassandra | celery | kafka | kerberos | mongo | otel | pinot |     │ +│statsd | statsd | trino)                                                                 │ +│--forward-credentials-fForward local credentials to container when running.│ +│--db-reset-dReset DB when entering the container.│ +│--max-timeMaximum time that the command should take - if it takes longer, the command will fail.│ +│(INTEGER RANGE)                                                                       │ +│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ +│--builderBuildx builder used to perform `docker buildx build` commands.(TEXT)│ +│[default: autodetect]                                         │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--verbose-vPrint verbose information about performed steps.│ +│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ +│--answer-aForce answer to questions.(y | n | q | yes | no | quit)│ +│--help-hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Developer commands ─────────────────────────────────────────────────────────────────────────────────────────────────╮ +│start-airflow     Enter breeze environment and starts all Airflow components in the tmux session. Compile assets  │ +│if contents of www directory changed.                                                           │ +│static-checks     Run static checks.                                                                              │ +│build-docs        Build documents.                                                                                │ +│down              Stop running breeze environment.                                                                │ +│shell             Enter breeze environment. this is the default command use when no other is selected.            │ +│exec              Joins the interactive shell of running airflow container.                                       │ +│compile-www-assetsCompiles www assets.                                                                            │ +│cleanup           Cleans the cache of parameters, docker cache and optionally built CI/PROD images.               │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Testing commands ───────────────────────────────────────────────────────────────────────────────────────────────────╮ +│testing        Tools that developers can use to run tests                                                         │ +│k8s            Tools that developers use to run Kubernetes tests                                                  │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Image commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ci-image         Tools that developers can use to manually manage CI images                                       │ +│prod-image       Tools that developers can use to manually manage PROD images                                     │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Release management commands ────────────────────────────────────────────────────────────────────────────────────────╮ +│release-management     Tools that release managers can use to prepare and manage Airflow releases                 │ +│sbom                   Tools that release managers can use to prepare sbom information                            │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Other commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│setup     Tools that developers can use to configure Breeze                                                       │ +│ci        Tools that CI workflows use to cleanup/manage CI environment                                            │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/images/breeze/output_build-docs.svg b/images/breeze/output_build-docs.svg index fdec4306b09e2..564a540f7b84e 100644 --- a/images/breeze/output_build-docs.svg +++ b/images/breeze/output_build-docs.svg @@ -252,48 +252,48 @@ │before the build - useful for a clean build.                                               │ │--one-pass-onlyBuilds documentation in one pass only. This is useful for debugging sphinx errors.│ │--package-filterList of packages to consider.                                                              │ -│(apache-airflow | apache-airflow-providers-airbyte | apache-airflow-providers-alibaba |    │ -│apache-airflow-providers-amazon | apache-airflow-providers-apache-beam |                   │ -│apache-airflow-providers-apache-cassandra | apache-airflow-providers-apache-drill |        │ -│apache-airflow-providers-apache-druid | apache-airflow-providers-apache-flink |            │ -│apache-airflow-providers-apache-hdfs | apache-airflow-providers-apache-hive |              │ -│apache-airflow-providers-apache-impala | apache-airflow-providers-apache-kafka |           │ -│apache-airflow-providers-apache-kylin | apache-airflow-providers-apache-livy |             │ -│apache-airflow-providers-apache-pig | apache-airflow-providers-apache-pinot |              │ -│apache-airflow-providers-apache-spark | apache-airflow-providers-apache-sqoop |            │ -│apache-airflow-providers-apprise | apache-airflow-providers-arangodb |                     │ -│apache-airflow-providers-asana | apache-airflow-providers-atlassian-jira |                 │ -│apache-airflow-providers-celery | apache-airflow-providers-cloudant |                      │ -│apache-airflow-providers-cncf-kubernetes | apache-airflow-providers-common-sql |           │ -│apache-airflow-providers-daskexecutor | apache-airflow-providers-databricks |              │ -│apache-airflow-providers-datadog | apache-airflow-providers-dbt-cloud |                    │ -│apache-airflow-providers-dingding | apache-airflow-providers-discord |                     │ -│apache-airflow-providers-docker | apache-airflow-providers-elasticsearch |                 │ -│apache-airflow-providers-exasol | apache-airflow-providers-facebook |                      │ -│apache-airflow-providers-ftp | apache-airflow-providers-github |                           │ -│apache-airflow-providers-google | apache-airflow-providers-grpc |                          │ -│apache-airflow-providers-hashicorp | apache-airflow-providers-http |                       │ -│apache-airflow-providers-imap | apache-airflow-providers-influxdb |                        │ -│apache-airflow-providers-jdbc | apache-airflow-providers-jenkins |                         │ -│apache-airflow-providers-microsoft-azure | apache-airflow-providers-microsoft-mssql |      │ -│apache-airflow-providers-microsoft-psrp | apache-airflow-providers-microsoft-winrm |       │ -│apache-airflow-providers-mongo | apache-airflow-providers-mysql |                          │ -│apache-airflow-providers-neo4j | apache-airflow-providers-odbc |                           │ -│apache-airflow-providers-openfaas | apache-airflow-providers-openlineage |                 │ -│apache-airflow-providers-opsgenie | apache-airflow-providers-oracle |                      │ -│apache-airflow-providers-pagerduty | apache-airflow-providers-papermill |                  │ -│apache-airflow-providers-plexus | apache-airflow-providers-postgres |                      │ -│apache-airflow-providers-presto | apache-airflow-providers-qubole |                        │ -│apache-airflow-providers-redis | apache-airflow-providers-salesforce |                     │ -│apache-airflow-providers-samba | apache-airflow-providers-segment |                        │ -│apache-airflow-providers-sendgrid | apache-airflow-providers-sftp |                        │ -│apache-airflow-providers-singularity | apache-airflow-providers-slack |                    │ -│apache-airflow-providers-smtp | apache-airflow-providers-snowflake |                       │ -│apache-airflow-providers-sqlite | apache-airflow-providers-ssh |                           │ -│apache-airflow-providers-tableau | apache-airflow-providers-tabular |                      │ -│apache-airflow-providers-telegram | apache-airflow-providers-trino |                       │ -│apache-airflow-providers-vertica | apache-airflow-providers-zendesk | docker-stack |       │ -│helm-chart)                                                                                │ +│(apache-airflow | docker-stack | helm-chart | apache-airflow-providers-airbyte |           │ +│apache-airflow-providers-alibaba | apache-airflow-providers-amazon |                       │ +│apache-airflow-providers-apache-beam | apache-airflow-providers-apache-cassandra |         │ +│apache-airflow-providers-apache-drill | apache-airflow-providers-apache-druid |            │ +│apache-airflow-providers-apache-flink | apache-airflow-providers-apache-hdfs |             │ +│apache-airflow-providers-apache-hive | apache-airflow-providers-apache-impala |            │ +│apache-airflow-providers-apache-kafka | apache-airflow-providers-apache-kylin |            │ +│apache-airflow-providers-apache-livy | apache-airflow-providers-apache-pig |               │ +│apache-airflow-providers-apache-pinot | apache-airflow-providers-apache-spark |            │ +│apache-airflow-providers-apache-sqoop | apache-airflow-providers-apprise |                 │ +│apache-airflow-providers-arangodb | apache-airflow-providers-asana |                       │ +│apache-airflow-providers-atlassian-jira | apache-airflow-providers-celery |                │ +│apache-airflow-providers-cloudant | apache-airflow-providers-cncf-kubernetes |             │ +│apache-airflow-providers-common-sql | apache-airflow-providers-daskexecutor |              │ +│apache-airflow-providers-databricks | apache-airflow-providers-datadog |                   │ +│apache-airflow-providers-dbt-cloud | apache-airflow-providers-dingding |                   │ +│apache-airflow-providers-discord | apache-airflow-providers-docker |                       │ +│apache-airflow-providers-elasticsearch | apache-airflow-providers-exasol |                 │ +│apache-airflow-providers-facebook | apache-airflow-providers-ftp |                         │ +│apache-airflow-providers-github | apache-airflow-providers-google |                        │ +│apache-airflow-providers-grpc | apache-airflow-providers-hashicorp |                       │ +│apache-airflow-providers-http | apache-airflow-providers-imap |                            │ +│apache-airflow-providers-influxdb | apache-airflow-providers-jdbc |                        │ +│apache-airflow-providers-jenkins | apache-airflow-providers-microsoft-azure |              │ +│apache-airflow-providers-microsoft-mssql | apache-airflow-providers-microsoft-psrp |       │ +│apache-airflow-providers-microsoft-winrm | apache-airflow-providers-mongo |                │ +│apache-airflow-providers-mysql | apache-airflow-providers-neo4j |                          │ +│apache-airflow-providers-odbc | apache-airflow-providers-openfaas |                        │ +│apache-airflow-providers-openlineage | apache-airflow-providers-opsgenie |                 │ +│apache-airflow-providers-oracle | apache-airflow-providers-pagerduty |                     │ +│apache-airflow-providers-papermill | apache-airflow-providers-plexus |                     │ +│apache-airflow-providers-postgres | apache-airflow-providers-presto |                      │ +│apache-airflow-providers-qubole | apache-airflow-providers-redis |                         │ +│apache-airflow-providers-salesforce | apache-airflow-providers-samba |                     │ +│apache-airflow-providers-segment | apache-airflow-providers-sendgrid |                     │ +│apache-airflow-providers-sftp | apache-airflow-providers-singularity |                     │ +│apache-airflow-providers-slack | apache-airflow-providers-smtp |                           │ +│apache-airflow-providers-snowflake | apache-airflow-providers-sqlite |                     │ +│apache-airflow-providers-ssh | apache-airflow-providers-tableau |                          │ +│apache-airflow-providers-tabular | apache-airflow-providers-telegram |                     │ +│apache-airflow-providers-trino | apache-airflow-providers-vertica |                        │ +│apache-airflow-providers-zendesk)                                                          │ │--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ │--builderBuildx builder used to perform `docker buildx build` commands.(TEXT)[default: autodetect]│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/images/breeze/output_release-management.svg b/images/breeze/output_release-management.svg index c25e978e3c16a..492e7091e11e4 100644 --- a/images/breeze/output_release-management.svg +++ b/images/breeze/output_release-management.svg @@ -35,8 +35,8 @@ .breeze-release-management-r1 { fill: #c5c8c6;font-weight: bold } .breeze-release-management-r2 { fill: #c5c8c6 } .breeze-release-management-r3 { fill: #d0b344;font-weight: bold } -.breeze-release-management-r4 { fill: #868887 } -.breeze-release-management-r5 { fill: #68a0b3;font-weight: bold } +.breeze-release-management-r4 { fill: #68a0b3;font-weight: bold } +.breeze-release-management-r5 { fill: #868887 } .breeze-release-management-r6 { fill: #98a84b;font-weight: bold } @@ -138,33 +138,33 @@ -Usage: breeze release-management [OPTIONS] COMMAND [ARGS]... +Usage: breeze release-management [OPTIONS] COMMAND [ARGS]... Tools that release managers can use to prepare and manage Airflow releases -╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--help-hShow this message and exit.│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Airflow release commands ───────────────────────────────────────────────────────────────────────────────────────────╮ -│prepare-airflow-package      Prepare sdist/whl package of Airflow.                                                │ -│create-minor-branch          Create a new version branch and update the default branches in main                  │ -│start-rc-process             Start RC process                                                                     │ -│start-release                Start Airflow release process                                                        │ -│release-prod-images          Release production images to DockerHub (needs DockerHub permissions).                │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Providers release commands ─────────────────────────────────────────────────────────────────────────────────────────╮ -│prepare-provider-documentation      Prepare CHANGELOG, README and COMMITS information for providers.              │ -│prepare-provider-packages           Prepare sdist/whl packages of Airflow Providers.                              │ -│install-provider-packages           Installs provider packages that can be found in dist.                         │ -│verify-provider-packages            Verifies if all provider code is following expectations for providers.        │ -│generate-providers-metadata         Generates metadata for providers.                                             │ -│generate-issue-content-providers    Generates content for issue to test the release.                              │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Other release commands ─────────────────────────────────────────────────────────────────────────────────────────────╮ -│publish-docs            Command to publish generated documentation to airflow-site                                │ -│generate-constraints    Generates pinned constraint files with all extras from setup.py in parallel.              │ -│add-back-references     Command to add back references for documentation to make it backward compatible           │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--help-hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Airflow release commands ───────────────────────────────────────────────────────────────────────────────────────────╮ +│prepare-airflow-package      Prepare sdist/whl package of Airflow.                                                │ +│create-minor-branch          Create a new version branch and update the default branches in main                  │ +│start-rc-process             Start RC process                                                                     │ +│start-release                Start Airflow release process                                                        │ +│release-prod-images          Release production images to DockerHub (needs DockerHub permissions).                │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Providers release commands ─────────────────────────────────────────────────────────────────────────────────────────╮ +│prepare-provider-documentation      Prepare CHANGELOG, README and COMMITS information for providers.              │ +│prepare-provider-packages           Prepare sdist/whl packages of Airflow Providers.                              │ +│install-provider-packages           Installs provider packages that can be found in dist.                         │ +│verify-provider-packages            Verifies if all provider code is following expectations for providers.        │ +│generate-providers-metadata         Generates metadata for providers.                                             │ +│generate-issue-content-providers    Generates content for issue to test the release.                              │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Other release commands ─────────────────────────────────────────────────────────────────────────────────────────────╮ +│publish-docs           Command to publish generated documentation to airflow-site                                 │ +│generate-constraints   Generates pinned constraint files with all extras from setup.py in parallel.               │ +│add-back-references    Command to add back references for documentation to make it backward compatible.           │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/images/breeze/output_release-management_add-back-references.svg b/images/breeze/output_release-management_add-back-references.svg index b8ead97cde426..75d6674bb0389 100644 --- a/images/breeze/output_release-management_add-back-references.svg +++ b/images/breeze/output_release-management_add-back-references.svg @@ -1,4 +1,4 @@ - +