Skip to content

Allow the user to use dag timezone to create dag_run_id#19039

Closed
LeonZhao28 wants to merge 6 commits into
apache:mainfrom
LeonZhao28:localize_run_id
Closed

Allow the user to use dag timezone to create dag_run_id#19039
LeonZhao28 wants to merge 6 commits into
apache:mainfrom
LeonZhao28:localize_run_id

Conversation

@LeonZhao28

Copy link
Copy Markdown
Contributor

recreate the PR (origin: #17502)
Why I commit this change:

When checking the airflow features, I found that the dag run_id is generated using the UTC time, but actually, it is not convenient for us to use the UTC time because we are using local time.
For example, I am in Beijing now, I ran a DAG at 19:01, the default run_id is manual__2021-09-08T11:01:02.022226+08:00, but after a few hours, I want to find this run, I have to parse the local time to UTC time to get the specific dag_run. If the run_id is generated using local time, the run_id will be manual__2021-09-08T19:01:02.022226+08:00, I can easily find this dag_run.
How do I change:

Added a config localize_dag_run_id in the core config and the default value is False. It takes no effect if the user doesn't care about the dag run_id format. If the users who just like me want to use the local time to generate the run_id, they can set it to True
Result:

image

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:core-operators area:Scheduler including HA (high availability) scheduler labels Oct 18, 2021
@uranusjr

Copy link
Copy Markdown
Member

We'll need to check all the places that look up DagRun (including DagRun.find() and direct SQLAlchemy calls) to make sure they are passing in execution_date correctly. We have a unique constraint on DagRun.execution_date, and I believe there are some parts of the code relies on the run_id can be uniquely calculated solely from execution_date, which would not be true after this PR. This is similar to #18968 but for scheduled runs.

@uranusjr

Copy link
Copy Markdown
Member

Converting this to draft for now since there are still many things to fix before this PR can be mergable (even without the aforementioned DagRun lookup issue).

@uranusjr uranusjr marked this pull request as draft October 18, 2021 03:34
@LeonZhao28

Copy link
Copy Markdown
Contributor Author

#18968

Sure. But I am busy with my works these days. And I may not continue to check it until early November. I'll do it as soon as I can

@uranusjr

Copy link
Copy Markdown
Member

No problem, there is totally no hurry, take your time. Open source contribution is all about doing what you can when you can, please don't feel pressured to do anything, we understand people have more important things in life than contributing to the project 🙂

@github-actions

github-actions Bot commented Dec 3, 2021

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 3, 2021
@github-actions github-actions Bot closed this Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:Scheduler including HA (high availability) scheduler stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants