diff --git a/providers/anthropic/docs/index.rst b/providers/anthropic/docs/index.rst index 3794d1ae58bc5..b6840352745e2 100644 --- a/providers/anthropic/docs/index.rst +++ b/providers/anthropic/docs/index.rst @@ -19,6 +19,9 @@ ``apache-airflow-providers-anthropic`` ====================================== +The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs — this page +compares that choice against ``common.ai``. + When to use this provider -------------------------- @@ -27,7 +30,8 @@ for you, which no vendor-neutral operator wraps: * ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude `Message Batches `__ - job for asynchronous bulk processing and wait for it to complete. + job for asynchronous bulk processing and wait for it to complete; Message Batches run at + 50% of standard cost, with most completing within an hour and a 24-hour SLA. * ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which the agent loop runs server-side on Anthropic's infrastructure; the Airflow task only kicks off the session and waits for its outcome. diff --git a/providers/cohere/docs/index.rst b/providers/cohere/docs/index.rst index 052babe4d9e78..092912d34ef20 100644 --- a/providers/cohere/docs/index.rst +++ b/providers/cohere/docs/index.rst @@ -19,6 +19,23 @@ ``apache-airflow-providers-cohere`` ====================================== +The ``cohere`` provider gives Dags direct access to Cohere's own Embed API — this page +compares that choice against ``common.ai``. + +When to use this provider +-------------------------- + +Use ``cohere`` when a Dag needs Cohere's native embedding models specifically: + +* ``CohereEmbeddingOperator`` — call Cohere's + `Embed API `__ directly via ``CohereHook``. + +Use :doc:`apache-airflow-providers-common-ai:index` instead when the embedding step should +stay vendor-neutral: + +* Document-to-vector-store pipelines with its document loader, embedding, and retrieval + operators (see :doc:`apache-airflow-providers-common-ai:operators/index`), which are not + tied to Cohere's embedding models. .. toctree:: :hidden: diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index eb1f768de6050..7ad6038742296 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -19,15 +19,39 @@ ``apache-airflow-providers-common-ai`` ################################################## +The ``common.ai`` provider is the vendor-neutral way to put LLM and agent steps in a Dag. + When to use this provider -------------------------- -``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on -`pydantic-ai `__, so the model vendor (OpenAI, Anthropic, Google, -Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching providers later -is a connection change, not a Dag rewrite. The AI step is orchestrated by Airflow: the model -calls, the agent loop, and any tools all run in the Airflow worker, where they get retries, -logging, and observability like any other task. +.. list-table:: + :header-rows: 1 + :widths: 40 30 30 + + * - Use case + - Use + - Package + * - Portable generation, classification, extraction, branching, or a + worker-run agent with toolsets + - ``common.ai`` + - ``apache-airflow-providers-common-ai`` + * - A vendor's native Embeddings, Responses, or Batch API + - The vendor's own provider + - e.g. :doc:`apache-airflow-providers-openai:index`, + :doc:`apache-airflow-providers-anthropic:index`, + :doc:`apache-airflow-providers-cohere:index` + * - A vendor-managed, server-side agent session (e.g. Anthropic Managed Agents) + - The vendor's own provider + - e.g. :doc:`apache-airflow-providers-anthropic:index` + +``common.ai`` is built on `pydantic-ai `__, so the model vendor +(OpenAI, Anthropic, Google, Bedrock, …) is picked by the connection ``llm_conn_id`` points +at — switching providers later is a connection change, not a Dag rewrite. Existing LangChain +tools aren't locked out either: with the ``langchain`` extra installed, they can be wrapped +in ``LangChainToolset`` and dropped straight into a common.ai agent (see :doc:`toolsets`). +The AI step is orchestrated by Airflow: the model calls, the agent loop, and any tools all +run in the Airflow worker, where they get retries, logging, and observability like any other +task. Use it when a Dag needs: @@ -35,8 +59,9 @@ Use it when a Dag needs: :doc:`LLMOperator and @task.llm `, with Pydantic-typed output pushed to XCom. * **Branching on a model's decision** — :doc:`LLMBranchOperator `. * **Agents with tools** — :doc:`AgentOperator ` runs a multi-turn agent loop - in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers), with optional - human-in-the-loop review and durable step replay. + in the worker, calling Airflow-defined :doc:`toolsets ` (SQL, hooks, MCP servers), + with optional human-in-the-loop review and durable step replay — if the task retries after + a failure, completed steps are replayed from cache instead of re-executing. * **Document pipelines** — loading, file analysis, embeddings, and retrieval for RAG (see :doc:`operators/index`). @@ -47,6 +72,7 @@ a service the vendor runs for you, which no vendor-neutral operator wraps: * :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches API, and Managed Agents sessions where the agent loop runs on Anthropic's infrastructure rather than in the Airflow worker. +* :doc:`apache-airflow-providers-cohere:index` — Cohere's own Embed API. As a rule of thumb: if Airflow should *run* the AI step (and the model should stay swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed service and diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst index 21dd8a9b3dd52..678075f53e30b 100644 --- a/providers/openai/docs/index.rst +++ b/providers/openai/docs/index.rst @@ -19,20 +19,25 @@ ``apache-airflow-providers-openai`` ====================================== +The ``openai`` provider gives Dags direct access to OpenAI's own APIs — this page compares +that choice against ``common.ai``. + When to use this provider -------------------------- Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers over -OpenAI-specific endpoints and options: +OpenAI-specific endpoints and options, built on ``OpenAIHook``, the underlying client the +operators below share: * ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed a vector store. * ``OpenAIResponseOperator`` — call the `Responses API `__ with OpenAI-specific parameters. -* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a +* ``OpenAITriggerBatchOperator`` — submit a `Batch API `__ job for asynchronous bulk - processing and wait for it to complete. + processing and wait for it to complete; OpenAI prices Batch API calls at roughly half the + cost of the equivalent synchronous call, in exchange for a turnaround of up to ~24 hours. Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step should be run by Airflow itself and stay vendor-neutral: