From 810d92155ba70acc13f5a1c8dd1cb80cabca06c8 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 14:39:29 +0800 Subject: [PATCH 1/7] Add feature-comparison table and toolset links to common.ai provider docs Reviewer feedback on apache/airflow#69551 (merged) asked for a scannable package-choice table linking to the vendor provider pages, a note that existing LangChain tools work via the langchain extra, and a link to the toolsets doc from the agent bullet. --- providers/common/ai/docs/index.rst | 32 +++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index eb1f768de6050..055894dab531e 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -22,12 +22,33 @@ When to use this provider -------------------------- +.. list-table:: + :header-rows: 1 + :widths: 40 30 30 + + * - You need + - Use + - Package + * - Portable generation, classification, extraction, branching, or a + worker-run agent with toolsets + - ``common.ai`` + - ``apache-airflow-providers-common-ai`` + * - A vendor's native Embeddings, Responses, or Batch API + - The vendor's own provider + - e.g. :doc:`apache-airflow-providers-openai:index`, + :doc:`apache-airflow-providers-anthropic:index` + * - A vendor-managed, server-side agent session (e.g. Anthropic Managed Agents) + - The vendor's own provider + - e.g. :doc:`apache-airflow-providers-anthropic:index` + ``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on `pydantic-ai `__, so the model vendor (OpenAI, Anthropic, Google, Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching providers later -is a connection change, not a Dag rewrite. The AI step is orchestrated by Airflow: the model -calls, the agent loop, and any tools all run in the Airflow worker, where they get retries, -logging, and observability like any other task. +is a connection change, not a Dag rewrite. Existing LangChain tools aren't locked out either: +with the ``langchain`` extra installed, they can be wrapped in ``LangChainToolset`` and +dropped straight into a common.ai agent (see :doc:`toolsets`). The AI step is orchestrated +by Airflow: the model calls, the agent loop, and any tools all run in the Airflow worker, +where they get retries, logging, and observability like any other task. Use it when a Dag needs: @@ -35,8 +56,9 @@ Use it when a Dag needs: :doc:`LLMOperator and @task.llm `, with Pydantic-typed output pushed to XCom. * **Branching on a model's decision** — :doc:`LLMBranchOperator `. * **Agents with tools** — :doc:`AgentOperator ` runs a multi-turn agent loop - in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers), with optional - human-in-the-loop review and durable step replay. + in the worker, calling Airflow-defined :doc:`toolsets ` (SQL, hooks, MCP servers), + with optional human-in-the-loop review and durable step replay — if the task retries after + a failure, completed steps are replayed from cache instead of re-executing. * **Document pipelines** — loading, file analysis, embeddings, and retrieval for RAG (see :doc:`operators/index`). From 26c7998b88cd13cb22da209d5b548a2f44558ed9 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 14:39:53 +0800 Subject: [PATCH 2/7] Fix openai provider docs: orient readers before the comparison, clarify batch tradeoff Reviewer feedback on apache/airflow#69551 (merged) noted the comparison section ran before readers knew what the provider was, that OpenAIHook isn't batch-specific, and that the batch bullet didn't explain why one would pick it over a synchronous call. --- providers/openai/docs/index.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst index 21dd8a9b3dd52..b8775b99cff5c 100644 --- a/providers/openai/docs/index.rst +++ b/providers/openai/docs/index.rst @@ -22,17 +22,20 @@ When to use this provider -------------------------- -Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers over -OpenAI-specific endpoints and options: +The ``openai`` provider gives Dags direct access to OpenAI's own APIs. This page compares +that choice against ``common.ai``. Use ``openai`` when a Dag needs OpenAI's +native API surface — thin wrappers over OpenAI-specific endpoints and options, built on +``OpenAIHook``, the underlying client the operators below share: * ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed a vector store. * ``OpenAIResponseOperator`` — call the `Responses API `__ with OpenAI-specific parameters. -* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a +* ``OpenAITriggerBatchOperator`` — submit a `Batch API `__ job for asynchronous bulk - processing and wait for it to complete. + processing and wait for it to complete; OpenAI prices Batch API calls at roughly half the + cost of the equivalent synchronous call, in exchange for a turnaround of up to ~24 hours. Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step should be run by Airflow itself and stay vendor-neutral: From 7de47a90b825fa024da83b01eb3d96ab3c141c65 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 14:43:11 +0800 Subject: [PATCH 3/7] Fix anthropic provider docs: orient readers before the comparison, add batch cost/turnaround rationale Reviewer feedback on apache/airflow#69551 (merged) noted the comparison section ran before readers knew what the provider was, and that the batch bullet didn't explain why one would pick it over a synchronous call. The new clause reuses the cost/turnaround figures already documented in operators/anthropic.rst for consistency. --- providers/anthropic/docs/index.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/providers/anthropic/docs/index.rst b/providers/anthropic/docs/index.rst index 3794d1ae58bc5..da24c6ec06b2b 100644 --- a/providers/anthropic/docs/index.rst +++ b/providers/anthropic/docs/index.rst @@ -22,12 +22,14 @@ When to use this provider -------------------------- -Use ``anthropic`` when a Dag needs Anthropic's native API surface — services Anthropic runs -for you, which no vendor-neutral operator wraps: +The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs. This page +compares that choice against ``common.ai``. Use ``anthropic`` when a Dag needs Anthropic's +native API surface — services Anthropic runs for you, which no vendor-neutral operator wraps: * ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude `Message Batches `__ - job for asynchronous bulk processing and wait for it to complete. + job for asynchronous bulk processing and wait for it to complete; Message Batches run at + 50% of standard cost, with most completing within an hour and a 24-hour SLA. * ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which the agent loop runs server-side on Anthropic's infrastructure; the Airflow task only kicks off the session and waits for its outcome. From 553c7d363d216fd6efb5a02a6d5735639a37521a Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 15:56:24 +0800 Subject: [PATCH 4/7] Add cohere to the common.ai vendor-provider comparison docs cohere is a third single-vendor AI provider shaped like openai/anthropic (wraps Cohere's own Embed API via one operator). It had no "when to use this provider" section, so common.ai's comparison table and vendor-provider bullet list couldn't link to it yet. --- providers/cohere/docs/index.rst | 16 ++++++++++++++++ providers/common/ai/docs/index.rst | 4 +++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/providers/cohere/docs/index.rst b/providers/cohere/docs/index.rst index 052babe4d9e78..b48c6aaec8177 100644 --- a/providers/cohere/docs/index.rst +++ b/providers/cohere/docs/index.rst @@ -19,6 +19,22 @@ ``apache-airflow-providers-cohere`` ====================================== +When to use this provider +-------------------------- + +The ``cohere`` provider gives Dags direct access to Cohere's own Embed API. This page +compares that choice against ``common.ai``. Use ``cohere`` when a Dag needs Cohere's +native embedding models specifically: + +* ``CohereEmbeddingOperator`` — call Cohere's + `Embed API `__ directly via ``CohereHook``. + +Use :doc:`apache-airflow-providers-common-ai:index` instead when the embedding step should +stay vendor-neutral: + +* Document-to-vector-store pipelines with its document loader, embedding, and retrieval + operators (see :doc:`apache-airflow-providers-common-ai:operators/index`), which are not + tied to Cohere's embedding models. .. toctree:: :hidden: diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index 055894dab531e..1a940c543543e 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -36,7 +36,8 @@ When to use this provider * - A vendor's native Embeddings, Responses, or Batch API - The vendor's own provider - e.g. :doc:`apache-airflow-providers-openai:index`, - :doc:`apache-airflow-providers-anthropic:index` + :doc:`apache-airflow-providers-anthropic:index`, + :doc:`apache-airflow-providers-cohere:index` * - A vendor-managed, server-side agent session (e.g. Anthropic Managed Agents) - The vendor's own provider - e.g. :doc:`apache-airflow-providers-anthropic:index` @@ -69,6 +70,7 @@ a service the vendor runs for you, which no vendor-neutral operator wraps: * :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches API, and Managed Agents sessions where the agent loop runs on Anthropic's infrastructure rather than in the Airflow worker. +* :doc:`apache-airflow-providers-cohere:index` — Cohere's own Embed API. As a rule of thumb: if Airflow should *run* the AI step (and the model should stay swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed service and From 2f92057ca7f71a1a5aff263f94b6bfd009f610e3 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 17:31:59 +0800 Subject: [PATCH 5/7] Soften "You need" table header wording in common.ai docs --- providers/common/ai/docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index 1a940c543543e..a4058dfc26c8d 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -26,7 +26,7 @@ When to use this provider :header-rows: 1 :widths: 40 30 30 - * - You need + * - Use case - Use - Package * - Portable generation, classification, extraction, branching, or a From 032d7b647d84e9bf5baa41572b10844fcac16d46 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 17:43:58 +0800 Subject: [PATCH 6/7] Move provider description before the "When to use this provider" heading The lead-in sentence naming what each provider is was folded into the section body, so a reader still met the comparison heading before the one-line definition. Moving it above the heading, for openai, anthropic, and cohere. --- providers/anthropic/docs/index.rst | 8 +++++--- providers/cohere/docs/index.rst | 7 ++++--- providers/openai/docs/index.rst | 10 ++++++---- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/providers/anthropic/docs/index.rst b/providers/anthropic/docs/index.rst index da24c6ec06b2b..b6840352745e2 100644 --- a/providers/anthropic/docs/index.rst +++ b/providers/anthropic/docs/index.rst @@ -19,12 +19,14 @@ ``apache-airflow-providers-anthropic`` ====================================== +The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs — this page +compares that choice against ``common.ai``. + When to use this provider -------------------------- -The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs. This page -compares that choice against ``common.ai``. Use ``anthropic`` when a Dag needs Anthropic's -native API surface — services Anthropic runs for you, which no vendor-neutral operator wraps: +Use ``anthropic`` when a Dag needs Anthropic's native API surface — services Anthropic runs +for you, which no vendor-neutral operator wraps: * ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude `Message Batches `__ diff --git a/providers/cohere/docs/index.rst b/providers/cohere/docs/index.rst index b48c6aaec8177..092912d34ef20 100644 --- a/providers/cohere/docs/index.rst +++ b/providers/cohere/docs/index.rst @@ -19,12 +19,13 @@ ``apache-airflow-providers-cohere`` ====================================== +The ``cohere`` provider gives Dags direct access to Cohere's own Embed API — this page +compares that choice against ``common.ai``. + When to use this provider -------------------------- -The ``cohere`` provider gives Dags direct access to Cohere's own Embed API. This page -compares that choice against ``common.ai``. Use ``cohere`` when a Dag needs Cohere's -native embedding models specifically: +Use ``cohere`` when a Dag needs Cohere's native embedding models specifically: * ``CohereEmbeddingOperator`` — call Cohere's `Embed API `__ directly via ``CohereHook``. diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst index b8775b99cff5c..678075f53e30b 100644 --- a/providers/openai/docs/index.rst +++ b/providers/openai/docs/index.rst @@ -19,13 +19,15 @@ ``apache-airflow-providers-openai`` ====================================== +The ``openai`` provider gives Dags direct access to OpenAI's own APIs — this page compares +that choice against ``common.ai``. + When to use this provider -------------------------- -The ``openai`` provider gives Dags direct access to OpenAI's own APIs. This page compares -that choice against ``common.ai``. Use ``openai`` when a Dag needs OpenAI's -native API surface — thin wrappers over OpenAI-specific endpoints and options, built on -``OpenAIHook``, the underlying client the operators below share: +Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers over +OpenAI-specific endpoints and options, built on ``OpenAIHook``, the underlying client the +operators below share: * ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed a vector store. From 1d182df20c314257531e174e66dc05661851fc72 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Thu, 9 Jul 2026 17:52:57 +0800 Subject: [PATCH 7/7] Move common.ai provider description before the "When to use this provider" heading Matches the same restructuring already applied to openai, anthropic, and cohere: a short definition sentence before the comparison heading, so a reader knows what the provider is before meeting the table. --- providers/common/ai/docs/index.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index a4058dfc26c8d..7ad6038742296 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -19,6 +19,8 @@ ``apache-airflow-providers-common-ai`` ################################################## +The ``common.ai`` provider is the vendor-neutral way to put LLM and agent steps in a Dag. + When to use this provider -------------------------- @@ -42,14 +44,14 @@ When to use this provider - The vendor's own provider - e.g. :doc:`apache-airflow-providers-anthropic:index` -``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on -`pydantic-ai `__, so the model vendor (OpenAI, Anthropic, Google, -Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching providers later -is a connection change, not a Dag rewrite. Existing LangChain tools aren't locked out either: -with the ``langchain`` extra installed, they can be wrapped in ``LangChainToolset`` and -dropped straight into a common.ai agent (see :doc:`toolsets`). The AI step is orchestrated -by Airflow: the model calls, the agent loop, and any tools all run in the Airflow worker, -where they get retries, logging, and observability like any other task. +``common.ai`` is built on `pydantic-ai `__, so the model vendor +(OpenAI, Anthropic, Google, Bedrock, …) is picked by the connection ``llm_conn_id`` points +at — switching providers later is a connection change, not a Dag rewrite. Existing LangChain +tools aren't locked out either: with the ``langchain`` extra installed, they can be wrapped +in ``LangChainToolset`` and dropped straight into a common.ai agent (see :doc:`toolsets`). +The AI step is orchestrated by Airflow: the model calls, the agent loop, and any tools all +run in the Airflow worker, where they get retries, logging, and observability like any other +task. Use it when a Dag needs: