Deprecate template variables ts etc. and change the default log_filename_template

[uranusjr]

Body

Context: #19058 (comment)

Note that changing the default log_filename_template have a lot of backward compatibility implications, and the solution needs to be cheap enough. Not a trivial job.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ashb]

For AIP-42 we'll need to add some form of "task_mapping_id" in here too

[uranusjr]

Copying from Slack to record the thoughts:

My current best idea is to create a table tracking log template changes and create a fk on TaskInstance pointing to it. A row is inserted to the table when Airflow launches whenever the template does not match the last row in the table.

The plan is to piggyback this operation on #18439, right after migrations are done.

When a TaskInstance is created, a foreign key to this tracking table is populated to signify what log filename template this TI will use. render_log_filename will use this fk to render the filename, instead of accessing the config.

Existing TaskInstance rows will not be backfilled for performance reasons. If this fk is NULL on TaskInstance, it will simply use the first row in the tracking table for rendering.

[ashb]

We can close this as done now can't we @uranusjr?

[uranusjr]

Yes I think so.

[uranusjr]

PR that actually flipped the switch: #21495