Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.25.0
Apache Airflow version
2.9.3
Operating System
Debian GNU/Linux 12
Deployment
Other Docker-based deployment
Deployment details
Using the upstream Airflow docker images customised using by exporting the following env vars:
AIRFLOW__CORE__EXECUTOR=airflow.providers.amazon.aws.executors.ecs.ecs_executor.AwsEcsExecutor
AIRFLOW__AWS_ECS_EXECUTOR__CLUSTER=
AIRFLOW__AWS_ECS_EXECUTOR__CONTAINER_NAME=
AIRFLOW__AWS_ECS_EXECUTOR__TASK_DEFINITION=
AIRFLOW__AWS_ECS_EXECUTOR__LAUNCH_TYPE=EC2
AIRFLOW__AWS_ECS_EXECUTOR__SECURITY_GROUPS=
AIRFLOW__AWS_ECS_EXECUTOR__SUBNETS=
AIRFLOW__AWS_ECS_EXECUTOR__REGION_NAME=eu-west-2
What happened
Currently when specifying the launch type for the ECS executor as "EC2" the current default being set for platform_version is being set to "Latest" which is incompatible for the EC2 launch type. When you don't export the AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION the current upstream airflow docker image will still put platform_version=latest into the airflow.cfg which will cause the ECS executor to fail as the AWS API says that the platform version parameter cannot be specified when an EC2 launch type is used when attempting to run a DAG with this executor.
If you specify an empty string for AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION this stops the airflow.cfg being populated with a value for platform_version but the executor will still pass a platform_version to the resulting RunTask API call at which point AWS returns saying that the platform_version cannot be blank. Setting any other value such as null or anything else results in the original error of the platform version parameter cannot be specified when using EC2 as the launch type
What you think should happen instead
The ECS Executor should verify that if EC2 is used as the launch type that the platform_version parameter is not passed to the API call. It seems the ECS Operator had a similar issue that was fixed as part of #17276
How to reproduce
Setup an ECS Cluster based on EC2 as the launch type with a task definition and anything else required as part of Airflow
Export the above variables into the docker image and run the webserver and scheduler and attempt to schedule a DAG run.
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.25.0
Apache Airflow version
2.9.3
Operating System
Debian GNU/Linux 12
Deployment
Other Docker-based deployment
Deployment details
Using the upstream Airflow docker images customised using by exporting the following env vars:
AIRFLOW__CORE__EXECUTOR=airflow.providers.amazon.aws.executors.ecs.ecs_executor.AwsEcsExecutor
AIRFLOW__AWS_ECS_EXECUTOR__CLUSTER=
AIRFLOW__AWS_ECS_EXECUTOR__CONTAINER_NAME=
AIRFLOW__AWS_ECS_EXECUTOR__TASK_DEFINITION=
AIRFLOW__AWS_ECS_EXECUTOR__LAUNCH_TYPE=EC2
AIRFLOW__AWS_ECS_EXECUTOR__SECURITY_GROUPS=
AIRFLOW__AWS_ECS_EXECUTOR__SUBNETS=
AIRFLOW__AWS_ECS_EXECUTOR__REGION_NAME=eu-west-2
What happened
Currently when specifying the launch type for the ECS executor as "EC2" the current default being set for platform_version is being set to "Latest" which is incompatible for the EC2 launch type. When you don't export the AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION the current upstream airflow docker image will still put platform_version=latest into the airflow.cfg which will cause the ECS executor to fail as the AWS API says that the platform version parameter cannot be specified when an EC2 launch type is used when attempting to run a DAG with this executor.
If you specify an empty string for AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION this stops the airflow.cfg being populated with a value for platform_version but the executor will still pass a platform_version to the resulting RunTask API call at which point AWS returns saying that the platform_version cannot be blank. Setting any other value such as null or anything else results in the original error of the platform version parameter cannot be specified when using EC2 as the launch type
What you think should happen instead
The ECS Executor should verify that if EC2 is used as the launch type that the platform_version parameter is not passed to the API call. It seems the ECS Operator had a similar issue that was fixed as part of #17276
How to reproduce
Setup an ECS Cluster based on EC2 as the launch type with a task definition and anything else required as part of Airflow
Export the above variables into the docker image and run the webserver and scheduler and attempt to schedule a DAG run.
Anything else
No response
Are you willing to submit PR?
Code of Conduct