Skip to content

Add feature-comparison table and toolset links to common.ai provider docs #69649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Lee-W merged 7 commits into from
Jul 10, 2026
+64 −12
Merged

Add feature-comparison table and toolset links to common.ai provider docs #69649

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
810d921
Add feature-comparison table and toolset links to common.ai provider …
Lee-W Jul 9, 2026
26c7998
Fix openai provider docs: orient readers before the comparison, clari…
Lee-W Jul 9, 2026
7de47a9
Fix anthropic provider docs: orient readers before the comparison, ad…
Lee-W Jul 9, 2026
553c7d3
Add cohere to the common.ai vendor-provider comparison docs
Lee-W Jul 9, 2026
2f92057
Soften "You need" table header wording in common.ai docs
Lee-W Jul 9, 2026
032d7b6
Move provider description before the "When to use this provider" heading
Lee-W Jul 9, 2026
1d182df
Move common.ai provider description before the "When to use this prov…
Lee-W Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion providers/anthropic/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@
``apache-airflow-providers-anthropic``
======================================

The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs — this page
compares that choice against ``common.ai``.

When to use this provider
--------------------------

Expand All @@ -27,7 +30,8 @@ for you, which no vendor-neutral operator wraps:

* ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude
`Message Batches <https://docs.claude.com/en/docs/build-with-claude/batch-processing>`__
job for asynchronous bulk processing and wait for it to complete.
job for asynchronous bulk processing and wait for it to complete; Message Batches run at
50% of standard cost, with most completing within an hour and a 24-hour SLA.
* ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which the agent loop
runs server-side on Anthropic's infrastructure; the Airflow task only kicks off the session
and waits for its outcome.
Expand Down
17 changes: 17 additions & 0 deletions providers/cohere/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,23 @@
``apache-airflow-providers-cohere``
======================================

The ``cohere`` provider gives Dags direct access to Cohere's own Embed API — this page
compares that choice against ``common.ai``.

When to use this provider
--------------------------

Use ``cohere`` when a Dag needs Cohere's native embedding models specifically:

* ``CohereEmbeddingOperator`` — call Cohere's
`Embed API <https://docs.cohere.com/docs/embeddings>`__ directly via ``CohereHook``.

Use :doc:`apache-airflow-providers-common-ai:index` instead when the embedding step should
stay vendor-neutral:

* Document-to-vector-store pipelines with its document loader, embedding, and retrieval
operators (see :doc:`apache-airflow-providers-common-ai:operators/index`), which are not
tied to Cohere's embedding models.

.. toctree::
:hidden:
Expand Down
42 changes: 34 additions & 8 deletions providers/common/ai/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,49 @@
``apache-airflow-providers-common-ai``
##################################################

The ``common.ai`` provider is the vendor-neutral way to put LLM and agent steps in a Dag.

When to use this provider
--------------------------

``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on
`pydantic-ai <https://ai.pydantic.dev/>`__, so the model vendor (OpenAI, Anthropic, Google,
Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching providers later
is a connection change, not a Dag rewrite. The AI step is orchestrated by Airflow: the model
calls, the agent loop, and any tools all run in the Airflow worker, where they get retries,
logging, and observability like any other task.
.. list-table::
:header-rows: 1
:widths: 40 30 30

* - Use case
- Use
- Package
* - Portable generation, classification, extraction, branching, or a
worker-run agent with toolsets
- ``common.ai``
- ``apache-airflow-providers-common-ai``
* - A vendor's native Embeddings, Responses, or Batch API
- The vendor's own provider
- e.g. :doc:`apache-airflow-providers-openai:index`,
:doc:`apache-airflow-providers-anthropic:index`,
:doc:`apache-airflow-providers-cohere:index`
* - A vendor-managed, server-side agent session (e.g. Anthropic Managed Agents)
- The vendor's own provider
- e.g. :doc:`apache-airflow-providers-anthropic:index`

``common.ai`` is built on `pydantic-ai <https://ai.pydantic.dev/>`__, so the model vendor
(OpenAI, Anthropic, Google, Bedrock, …) is picked by the connection ``llm_conn_id`` points
at — switching providers later is a connection change, not a Dag rewrite. Existing LangChain
tools aren't locked out either: with the ``langchain`` extra installed, they can be wrapped
in ``LangChainToolset`` and dropped straight into a common.ai agent (see :doc:`toolsets`).
The AI step is orchestrated by Airflow: the model calls, the agent loop, and any tools all
run in the Airflow worker, where they get retries, logging, and observability like any other
task.

Use it when a Dag needs:

* **Generation, classification, summarization, or structured extraction** —
:doc:`LLMOperator and @task.llm <operators/llm>`, with Pydantic-typed output pushed to XCom.
* **Branching on a model's decision** — :doc:`LLMBranchOperator <operators/llm_branch>`.
* **Agents with tools** — :doc:`AgentOperator <operators/agent>` runs a multi-turn agent loop
in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers), with optional
human-in-the-loop review and durable step replay.
in the worker, calling Airflow-defined :doc:`toolsets <toolsets>` (SQL, hooks, MCP servers),
with optional human-in-the-loop review and durable step replay — if the task retries after
a failure, completed steps are replayed from cache instead of re-executing.
* **Document pipelines** — loading, file analysis, embeddings, and retrieval for RAG
(see :doc:`operators/index`).

Expand All @@ -47,6 +72,7 @@ a service the vendor runs for you, which no vendor-neutral operator wraps:
* :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches API, and
Managed Agents sessions where the agent loop runs on Anthropic's infrastructure rather
than in the Airflow worker.
* :doc:`apache-airflow-providers-cohere:index` — Cohere's own Embed API.

As a rule of thumb: if Airflow should *run* the AI step (and the model should stay
swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed service and
Expand Down
11 changes: 8 additions & 3 deletions providers/openai/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,25 @@
``apache-airflow-providers-openai``
======================================

The ``openai`` provider gives Dags direct access to OpenAI's own APIs — this page compares
that choice against ``common.ai``.

When to use this provider
--------------------------

Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers over
OpenAI-specific endpoints and options:
OpenAI-specific endpoints and options, built on ``OpenAIHook``, the underlying client the
operators below share:

* ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed a vector
store.
* ``OpenAIResponseOperator`` — call the
`Responses API <https://platform.openai.com/docs/api-reference/responses>`__ with
OpenAI-specific parameters.
* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a
* ``OpenAITriggerBatchOperator`` — submit a
`Batch API <https://platform.openai.com/docs/guides/batch>`__ job for asynchronous bulk
processing and wait for it to complete.
processing and wait for it to complete; OpenAI prices Batch API calls at roughly half the
cost of the equivalent synchronous call, in exchange for a turnaround of up to ~24 hours.

Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step should be run by
Airflow itself and stay vendor-neutral:
Expand Down
Loading