Description
Azure Data Factory (ADF) is a fully-managed, orchestration and integration service that allows users to run data pipelines easily within the Azure ecosystem. Currently there is only a hook, AzureDataFactoryHook, that exists but no operators or sensors as reusable, Airflow abstractions. Seems beneficial to create an operator and corresponding sensor for running data pipelines in Azure Data Factory at the very least a AzureDataFactoryRunPipelineOperator and AzureDataFactoryPipelineRunStatusSensor (names TBD).
Use case / motivation
The power of Airflow, among many things, is its provider-agnostic stance. Airflow is able to not only orchestrate pipelines itself but also orchestrate other orchestration tools. Being able to bring higher-level orchestration and monitoring to users working with ADF pipelines would be very powerful and improve the developer experience of those trying to manage multiple ADF pipelines as part of their end-to-end architecture.
Are you willing to submit a PR?
Absolutely! 🚀
Related Issues
Slightly related (as example DAGs would be part of creating new operators/sensors for Azure Data Factory): #14758
Description
Azure Data Factory (ADF) is a fully-managed, orchestration and integration service that allows users to run data pipelines easily within the Azure ecosystem. Currently there is only a hook,
AzureDataFactoryHook, that exists but no operators or sensors as reusable, Airflow abstractions. Seems beneficial to create an operator and corresponding sensor for running data pipelines in Azure Data Factory at the very least aAzureDataFactoryRunPipelineOperatorandAzureDataFactoryPipelineRunStatusSensor(names TBD).Use case / motivation
The power of Airflow, among many things, is its provider-agnostic stance. Airflow is able to not only orchestrate pipelines itself but also orchestrate other orchestration tools. Being able to bring higher-level orchestration and monitoring to users working with ADF pipelines would be very powerful and improve the developer experience of those trying to manage multiple ADF pipelines as part of their end-to-end architecture.
Are you willing to submit a PR?
Absolutely! 🚀
Related Issues
Slightly related (as example DAGs would be part of creating new operators/sensors for Azure Data Factory): #14758