Skip to content

Improve handling of string type and non-attribute template_fields#21054

Merged
potiuk merged 1 commit into
apache:mainfrom
josh-fell:operator-template-fields-issue-handling
Jan 26, 2022
Merged

Improve handling of string type and non-attribute template_fields#21054
potiuk merged 1 commit into
apache:mainfrom
josh-fell:operator-template-fields-issue-handling

Conversation

@josh-fell

@josh-fell josh-fell commented Jan 24, 2022

Copy link
Copy Markdown
Contributor

Related: #19047, #19052
Closes: #12876

This PR intends to improve both the handling of template_fields values declared as strings (most typically seen as template_fields = ("some_field") as well as the error message for declaring template_fields that are not attributes of the operator. In the first case, a warning is logged and the template_fields is "automagically" converted to a list for the task execution. For the second case, the error message is updated from the default AttributeError to something more specific to what is happening when attempting to render any template fields.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

Comment thread airflow/models/baseoperator.py Outdated
Comment on lines 747 to 755

@uranusjr uranusjr Jan 24, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
f"The `template_fields` value for {self.task_type} is a string "
"but should be a Sequence type. Converting to a Sequence type for execution. "
f"Please update {self.task_type} accordingly.",
f"The `template_fields` value for {self.task_type} is a string "
"but should be a list of string. Wrapping it in a list for execution. "
f"Please update {self.task_type} accordingly.",

str is technically a sequence, so sequence of string is a better term. But I feel the term sequence itself might not be the easiest to understand for less experienced developers, so let’s just recommend using a list instead, which is the most fool-proof syntax.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent point.

Comment thread airflow/models/baseoperator.py Outdated
Comment on lines 1098 to 1073

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if hasattr(parent, attr_name):
content = getattr(parent, attr_name)
if content:
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
setattr(parent, attr_name, rendered_content)
else:
raise AttributeError(
f"{attr_name!r} is configured as a template field "
f"but {parent.task_type} does not have this attribute."
)
try:
content = getattr(parent, attr_name)
except AttributeError:
raise AttributeError(
f"{attr_name!r} is configured as a template field "
f"but {parent.task_type} does not have this attribute."
)
else:
if content:
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
setattr(parent, attr_name, rendered_content)

EAFP over LBYL.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except ... else not needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes. More succinct as well.

Comment thread airflow/models/baseoperator.py Outdated
Comment on lines 1370 to 1286

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@josh-fell
josh-fell force-pushed the operator-template-fields-issue-handling branch 2 times, most recently from e745924 to abd1b85 Compare January 24, 2022 14:43
@josh-fell
josh-fell force-pushed the operator-template-fields-issue-handling branch from abd1b85 to 69a0d62 Compare January 24, 2022 16:45
@josh-fell

Copy link
Copy Markdown
Contributor Author

Pushed some fixes for static check errors...come on Josh. 🤦

@potiuk

potiuk commented Jan 24, 2022

Copy link
Copy Markdown
Member

Pushed some fixes for static check errors...come on Josh. facepalm

Hint:

pre-commit run --from-ref=HEAD^ --to-ref=HEAD

And a sneaky preview of new Breeze2 equivalent:

./Breeze2 static-checks --last-commit

@potiuk

potiuk commented Jan 24, 2022

Copy link
Copy Markdown
Member

cc: @Bowrna ^^

@josh-fell

Copy link
Copy Markdown
Contributor Author

Pushed some fixes for static check errors...come on Josh. facepalm

Hint:

pre-commit run --from-ref=HEAD^ --to-ref=HEAD

And a sneaky preview of new Breeze2 equivalent:

./Breeze2 static-checks --last-commit

Yeah that's usually my go-to move but I completely spaced on it this time. That new command is lovely. I'll have to try it out.

@github-actions github-actions Bot added the full tests needed We need to run full set of tests for this PR to merge label Jan 26, 2022
@github-actions

Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. Love the explicit communication!

@potiuk
potiuk merged commit beb2a20 into apache:main Jan 26, 2022
@josh-fell
josh-fell deleted the operator-template-fields-issue-handling branch January 26, 2022 19:48
@jedcunningham jedcunningham added the type:improvement Changelog: Improvements label Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests needed We need to run full set of tests for this PR to merge type:improvement Changelog: Improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve error message when template fields contain invalid entries

5 participants