Skip to content

Unable to get task log from OpenSearch and instead see 500 error on the Airflow v3.0.0 UI #49657

Description

@sonu1311

Apache Airflow Provider(s)

opensearch

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.6.1
apache-airflow-providers-celery==3.10.6
apache-airflow-providers-cncf-kubernetes==10.4.3
apache-airflow-providers-common-compat==1.6.0
apache-airflow-providers-common-io==1.5.4
apache-airflow-providers-common-messaging==1.0.0
apache-airflow-providers-common-sql==1.26.0
apache-airflow-providers-docker==4.3.1
apache-airflow-providers-elasticsearch==6.2.2
apache-airflow-providers-fab==2.0.1
apache-airflow-providers-ftp==3.12.3
apache-airflow-providers-git==0.0.2
apache-airflow-providers-google==15.1.0
apache-airflow-providers-grpc==3.7.3
apache-airflow-providers-hashicorp==4.1.1
apache-airflow-providers-http==5.2.2
apache-airflow-providers-microsoft-azure==12.3.1
apache-airflow-providers-mysql==6.2.2
apache-airflow-providers-odbc==4.9.2
apache-airflow-providers-openlineage==2.2.0
apache-airflow-providers-opensearch==1.6.3
apache-airflow-providers-postgres==6.1.3
apache-airflow-providers-redis==4.0.2
apache-airflow-providers-sendgrid==4.0.1
apache-airflow-providers-sftp==5.2.1
apache-airflow-providers-slack==9.0.5
apache-airflow-providers-smtp==2.0.3
apache-airflow-providers-snowflake==6.2.2
apache-airflow-providers-ssh==4.0.1
apache-airflow-providers-standard==1.0.0
apache-airflow-task-sdk==1.0.0

Apache Airflow version

3.0.0

Operating System

Debian GNU/Linux 12

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployed the latest Airflow helm chart with airflowVersion: "3.0.0" and KubernetesExecutor for executor on EKS. The logs persistent is diabled and chose Opensearch as an instance of opensearch is running on the same eks cluster.
opensearch:
# Enable opensearch task logging
enabled: true
connection:
scheme: https
user: admin
pass: pass
host: opensearch-cluster-master.opensearch.svc.cluster.local
port: 9200

What happened

Seeing this error when trying to access task logs in the API server pod logs

opensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPSConnection object at 0x7efce05bef50>: Failed to resolve 'https://admin:pass@opensearch-cluster-master.opensearch.svc.cluster.local:9200' ([Errno -2] Name or service not known)) caused by: NameResolutionError(<urllib3.connection.HTTPSConnection object at 0x7efce05bef50>: Failed to resolve 'https://admin:pass@opensearch-cluster-master.opensearch.svc.cluster.local:9200' ([Errno -2]

What you think should happen instead

I should be able to view the loggging in the UI which should be retrived from opensearch.

How to reproduce

Enabled Opensearch option for remote logging in the helm chart and give valid connection setting for the same.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions