Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.

[tsaaii]

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure

Apache Airflow version

2.2.3 (latest released)

Operating System

windows

Deployment

Other Docker-based deployment

Deployment details

I have a docker image with kubernetes which has all the provider dependencies and other pip installations with official helm chart where I overwrote some values for airflow which are mostly credentials related.

What happened

I am trying to test my azure data factory connection and when I hit the test button after giving client ID, secret, Factory name, resource group name, subscription ID , it displays a red banner but it doesn't say what failed.

I am looking to debug this.

My azure connector configuration

red banner indicating something isn't right but it doesn't say what

What you expected to happen

The goal is to see a successful connection to ADF from Airflow so that i can run my ADF pipelines on airflow.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[josh-fell]

I can't necessarily speak to why there is an empty red banner but I can see two things going on:

1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner when clicking on the inactive button and the tooltip below when you hover over the "Test" button:
   

2. The Azure Data Factory connection does not have a test_connection() method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is documentation on testing connections.

FYI there is an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a test_connection() method available via PR #19792 but it has not been released yet. This enhancement also functionally disables the button instead of only the appearance of the button being disabled.

[wesleyemery]

I've was able to enable the test button by enabling the airflow API in the helm chart:

# airflowVersion: "2.2.2"
# defaultAirflowTag: "2.2.2"
executor: "KubernetesExecutor"

images:
  airflow:
    repository: ""
    tag: ""
  useDefaultImageForMigration: true

nodeSelector:
  agentpool: "airflow"

ingress:
  enabled: true
  web:
    annotations: 
      kubernetes.io/ingress.class: "nginx"
    hosts: ["airflow.dev.beertech.com"]
    tls:
      enabled: true
      secretName: ingress-tls-secret
  # flower:
  #   annotations: 
  #     kubernetes.io/ingress.class: "nginx"  
  #   path: "/flower"
  #   hosts: ["dev.airflow.beertech.com"]
  #   tls:
  #     enabled: true
  #     secretName: ingress-tls-secret

config:
  api:
    auth_backend: airflow.api.auth.backend.default  
webserver:
  defaultUser:
    enabled: true
    role: Admin
    username: devops
    email: 
    firstName: devops
    lastName: admin
    password:
dags:
  gitSync:
    enabled: true
    repo: ""
    branch: "main"
    subPath: "dags/"
    sshKeySecret: airflow-ssh-secret
extraSecrets:
  airflow-ssh-secret:
    data: |

However, when I test the connection I still get a red banner with no info for why it's failing.

[josh-fell]

Right on. Even if you enabled the REST API, the second point still holds. The underlying AzureDataFactoryHook doesn't implement a test_connection() method which the API uses to perform the test. Without that method, you can't test the connection.

Unfortunately, I tried to whip something up for testing the ADF connection and ran into #21188 which needs to be fixed first.

[wesleyemery]

Is there any other way to test the connection? Would DAG logging work Stack overflow DAG logging?

[josh-fell]

You can certainly create a PythonOperator/TaskFlow function that calls one of the AzureDataFactoryHook methods (e.g. get_factory(), or something else straightforward) and see if you can retrieve pertinent data. But there is really no way to test prior to executing an Airflow task like you can in the UI if it was all configured for ADF.