update pattern for dataflow job id extraction#41794
Conversation
676c264 to
8019d86
Compare
|
@VladaZakharova when will this be merged? :) |
|
Hi @potiuk ! Can you please merge it? |
When the test pass and someone will merge it. Since you are the first time contributor - we have to manually approve workflows to see if tests pass, then you have to fix them if they don't. but when you submit new version you will have to wait for someone to see it and approve it (you can ask in general without mentioning anyone to approve your workflows) to signal that you think you fixed all the tests. Also see the contribution docs that explain the process https://github.com/apache/airflow/tree/main/contributing-docs |
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Dataflow job id is extracted from the logged output of
javaprocess that starts the Dataflow job, for example, in case ofBeamRunJavaPipelineOperator.Currently job id pattern matches characters until first
"or\nis encountered, which is fine for a following case:[2024-08-27 11:20:22,094] INFO Submitted job: 2024-08-27_04_20_21-79473727258167061512024-08-27_04_20_21-7947372725816706151However, if the logger is configured differently, for example, has a whitespace and a suffix at the end with additional information, the pattern extracts the id together with the suffix:
[2024-08-27 11:20:22,094] INFO Submitted job: 2024-08-27_04_20_21-7947372725816706151 (org.apache.beam.runners.dataflow.DataflowRunner) (main)2024-08-27_04_20_21-7947372725816706151 (org.apache.beam.runners.dataflow.DataflowRunner) (main)In the previous example suffix
(org.apache.beam.runners.dataflow.DataflowRunner) (main)should not be extracted as part of the job id.I updated the pattern by adding the whitespace character
\s(along side existing"and\n), indicating the end of job id.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.