Added Feature: retry strategy parameter to Amazon AWS Batch Operator #35789
Conversation
…to allow dynamic Batch retry strategies
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
|
Nice indeed :) |
|
@evgenyslab Looks like this change has broken the example_batch system test: https://aws-mwaa.github.io/#/open-source/system-tests/dashboard.html |
| self.scheduling_priority_override = scheduling_priority_override | ||
| self.array_properties = array_properties | ||
| self.parameters = parameters or {} | ||
| self.retry_strategy = retry_strategy or {} |
There was a problem hiding this comment.
I think this needs to default to {"attempts": 1} and likely needs some logic to enforce that a provided value contains a valid "attempts" field.
…to allow dynamic Batch retry strategies (#35789)
…to allow dynamic Batch retry strategies (#35789)

Added
retry_strategyparameter toproviders/amazon/aws/operators/batch.pythat is passed to theboto3client asretryStrategyper the syntax found here:https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch/client/submit_job.html
This allows definition of custom retry strategies from Airflow side instead of relying on job definitions.