Skip to content

Persist DataprocLink for workflow operators regardless of job status#26986

Merged
potiuk merged 12 commits into
apache:mainfrom
vksunilk:add-dataproc-workflow-link
Nov 16, 2022
Merged

Persist DataprocLink for workflow operators regardless of job status#26986
potiuk merged 12 commits into
apache:mainfrom
vksunilk:add-dataproc-workflow-link

Conversation

@vksunilk

@vksunilk vksunilk commented Oct 11, 2022

Copy link
Copy Markdown
Contributor

Fixes #26987

For DataprocInstantiateInlineWorkflowTemplateOperator and DataprocInstantiateWorkflowTemplateOperator, the dataproc link is available only for the jobs that have succeeded. Incase of Failure, the DataprocLink is not available

Also, As we have moved from LOCATION to REGION in dataproc, have renamed the constant in test file to GCP_REGION


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vksunilk vksunilk requested a review from turbaszek as a code owner October 11, 2022 09:13
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Oct 11, 2022
@boring-cyborg

boring-cyborg Bot commented Oct 11, 2022

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch 2 times, most recently from 109faea to 948bb2d Compare October 12, 2022 03:18
@uranusjr

Copy link
Copy Markdown
Member

Would it be possible to add a test for this?

@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch 2 times, most recently from c1aa036 to b76a29d Compare October 12, 2022 14:07
@vksunilk

Copy link
Copy Markdown
Contributor Author

Would it be possible to add a test for this?
@uranusjr Sure. Have added tests now

@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch 4 times, most recently from 488e0a3 to 0f97294 Compare October 17, 2022 11:26
@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch from 0f97294 to efcdce7 Compare October 21, 2022 05:03
For DataprocInstantiateInlineWorkflowTemplateOperator and DataprocInstantiateWorkflowTemplateOperator, the dataproc link is available only for the jobs that have succeeded. Incase of Failure, the DataprocLink is not available
@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch 2 times, most recently from 2341fda to ac8678f Compare October 23, 2022 00:59
@vksunilk vksunilk force-pushed the add-dataproc-workflow-link branch from ac8678f to 38932a9 Compare October 23, 2022 01:02
@potiuk

potiuk commented Nov 6, 2022

Copy link
Copy Markdown
Member

statuic checks failing.

@vksunilk

vksunilk commented Nov 7, 2022

Copy link
Copy Markdown
Contributor Author

statuic checks failing.

It has passed now @potiuk

@potiuk

potiuk commented Nov 11, 2022

Copy link
Copy Markdown
Member

Nope :( .Static checks are failing. I recommend you to install pre-commit . It will correct the errors for you automatically.

@vksunilk

Copy link
Copy Markdown
Contributor Author

Nope :( .Static checks are failing. I recommend you to install pre-commit . It will correct the errors for you automatically.

Done now. Precommit helped!

@potiuk

potiuk commented Nov 14, 2022

Copy link
Copy Markdown
Member

Done now. Precommit helped!

Let's see. You are a new user, so I still needed to approve the workflow run from you to run tests.

@potiuk

potiuk commented Nov 14, 2022

Copy link
Copy Markdown
Member

Anyone from GCP team wants to comment @bjankie1 ? others ?

@vksunilk

Copy link
Copy Markdown
Contributor Author

Done now. Precommit helped!

Let's see. You are a new user, so I still needed to approve the workflow run from you to run tests.

Finally, it has passed

@potiuk potiuk merged commit 0cb6450 into apache:main Nov 16, 2022
@boring-cyborg

boring-cyborg Bot commented Nov 16, 2022

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataprocLink is not available for dataproc workflow operators

3 participants