Add a retry with wait interval for SSH operator

[Gaurang033]

Description
currently SSH operator fails if authentication fails without retying. We can add two more parameters as mentioned below which will retry in case of Authentication failure after waiting for configured time.

• max_retry - maximum time SSH operator should retry in case of exception
• wait - how many seconds it should wait before next retry

Use case / motivation
We are using SSH operator heavily in our production jobs, And what I have noticed is sometimes SSH operator fails to authenticate, however open re-running jobs it run successfully. And this happens ofthen. We have ended up writing our own custom operator for this. However, if we can implement this, this could help others as well.

Implement the suggested feature for ssh operator.

Are you willing to submit a PR?
I will submit the PR if feature gets approval.

Related Issues
N/A

No

[mik-laj]

SGTM. GoogleCompute retry authorizations, so generic
hook can also do it.

[Gaurang033]

@mik-laj can you elaborate on the generic hook thing? I was thinking to modify the existing hook and add an option for this.

[mik-laj]

We have 2 hooks that allows us to connect to SSH server. Second hook have retry mechanism. See:

airflow/airflow/providers/google/cloud/hooks/compute_ssh.py

Line 251 in 3f7ebfd

for time_to_wait in exponential_sleep_generator(initial=1, maximum=max_time_to_wait):

[Gaurang033]

@mik-laj perfect. we can similar implementation for this as well, then
https://github.com/apache/airflow/blob/3f7ebfdfe2a1fa90b0854028a5db057adacd46c1/airflow/providers/ssh/hooks/ssh.py

[nshetty15]

@eladkal Can I take this up?

[potiuk]

assigned it to you @nshetty15

[eladkal]

@nshetty15 are you still working on it?

[nshetty15]

@eladkal I had some issues setting up airflow. Its good now. I will submit a PR soon for this one.