Support Flink operator

[SteNicholas]

Description

Airflow currently supports Spark operators for kicking off a spark-submit job. In real-time computing or online machine learning scenarios, Flink operator could be designed to submit flink job in streaming workflow.

Use case / motivation

Airflow coud supports definition of FlinkSubmitOperator for DAG composed of multiple Flink jobs. FlinkSubmitOperator is designed to introduce the operator that kicks off a flink-submit task, and provides corresponding to JDBC and SQL operators.

Related Issues

N/A

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[zhongjiajie]

Sound good! looking forward your contribute. BTW, Airflow open to all new and useful operators.

[subkanthi]

If this is not being worked on, I can take it on, please let me know. @zhongjiajie

[mik-laj]

@subkanthi I assigned you to this ticket. 🐈

[SteNicholas]

@subkanthi I thought that you currently couldn't complete this issue. Because Airflow couldn't support the streaming semantic, how do you work on this issue? @mik-laj @zhongjiajie What do we think about?

[mik-laj]

@SteNicholas I think we can add two operators. One that starts streaming operations on a remote cluster, return ID to XCOM and the other that stops these operations. However, I have no experience with Flink. Do you think this is possible?

[zhongjiajie]

I think it depend on what we could add streaming to airflow, if it will task many times, maybe support parts of flink function on flink operator is acceptable.

[subkanthi]

I thought this was very similar to the SparkSubmit operator where it expects spark-submit to be in the path and takes the jar file as a parameter.

[zhongjiajie]

But that parts of flink function, nor full of it. maybe you could see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow for more detail.

[subkanthi]

@mik-laj , @zhongjiajie is it OK if i just work on the simple flinksubmit operator first, maybe we can have another issue for the streaming part?

[SteNicholas]

@subkanthi How does the operator teminate? The flink submit for streaming application doesn't teminate.

[subkanthi]

Is it not the same as spark-submit kill command, we use the flink cancel command.
./bin/flink cancel
Are we all referring to starting and cancelling jobs using the CLI?
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/cli.html

[SteNicholas]

@subkanthi Do you mean that the implemenation of FlinkOperator is based on flink run command? But If you want to kill Flink Job, you must persist with the job id. Therefore, how do you control with this presistence?