Skip to content

move to dag_run.logical_date from execution date in OpenLineage provider#41889

Merged
mobuchowski merged 1 commit into
apache:mainfrom
mobuchowski:move-to-logical-date-ol-provider
Sep 2, 2024
Merged

move to dag_run.logical_date from execution date in OpenLineage provider#41889
mobuchowski merged 1 commit into
apache:mainfrom
mobuchowski:move-to-logical-date-ol-provider

Conversation

@mobuchowski

Copy link
Copy Markdown
Contributor

Remove usage of deprecated execution_date in favor of logical_date

Comment thread tests/providers/openlineage/plugins/test_listener.py Outdated
@potiuk

potiuk commented Sep 1, 2024

Copy link
Copy Markdown
Member

Is it backwards-compatible with Airflow 2?

@mobuchowski

Copy link
Copy Markdown
Contributor Author

@potiuk logical_date property was added back in 2021

Screenshot 2024-09-02 at 11 14 00

@potiuk

potiuk commented Sep 2, 2024

Copy link
Copy Markdown
Member

logical_date teproperty was added back in 2021

I know, just checking if it is ok to change it in the provider - which will be (for the next 6 months or so) only installed on Airflow 2 :)

@potiuk

potiuk commented Sep 2, 2024

Copy link
Copy Markdown
Member

Simply execution_date and logical_date have different semantics, so my question is, if that will impact observed open-lineage events. And if so - whether it should be somehow explained in the changelog so that users are not surprised.

@mobuchowski

Copy link
Copy Markdown
Contributor Author

@potiuk does it really have different semantics?

From what I can find, it's essentially a rename:

From AIP-39

We need a single date value that is the "canonical" date for the DagRun date, that can be used for sorting DagRuns for the UI view, so the current "execution_date" column on DagRun needs to be kept, but we should rename - and we've chosen "logical_date".

Also in the docstring:

This replaces execution_date in Airflow 2.1 and prior. The idea is essentially the same, just a different name.

Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason.

@potiuk

potiuk commented Sep 2, 2024

Copy link
Copy Markdown
Member

Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason.

Yep. You are right. I had confused it with "start/end interval" and other macros available in context, but logical_date and execution_date are really aliases.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants