Skip to content

Validate Google operator templated parameters after rendering - #70534

Open
mitre88 wants to merge 1 commit into
apache:mainfrom
mitre88:narrow-google-validation-exceptions
Open

Validate Google operator templated parameters after rendering#70534
mitre88 wants to merge 1 commit into
apache:mainfrom
mitre88:narrow-google-validation-exceptions

Conversation

@mitre88

@mitre88 mitre88 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Three Google provider operators — CloudSpeechToTextRecognizeSpeechOperator, CloudTextToSpeechSynthesizeOperator, and CloudFirestoreExportDatabaseOperator — validate their template fields (audio, config, input_data, voice, audio_config, target_bucket_name, target_filename, body) inside a _validate_inputs() helper called from __init__.

Because the helper reads the fields through a method call, the validate-operators-init check cannot see these reads (same blind spot as the AppFlow subclasses fixed in #70440), but the bug is the same one tracked in #70296: the emptiness checks run against un-rendered Jinja expressions, which are always non-empty, so a templated value rendering to an empty string is never caught. The _validate_inputs() call is moved to the start of execute() in all three operators.

While relocating them, the raise AirflowException usages are narrowed to ValueError per the ongoing exception clean-up (known_airflow_exceptions.txt entries for the three files drop to 0), following the pattern of #66279.

Tests: converted the text-to-speech missing-argument test to the new exception type (it already exercised execute), and added execute-time tests for the speech-to-text and Firestore operators that construct with a templated value and validate the rendered value — both fail against the previous implementation.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines

CloudSpeechToTextRecognizeSpeechOperator, CloudTextToSpeechSynthesizeOperator,
and CloudFirestoreExportDatabaseOperator validate template fields inside
_validate_inputs() called from __init__, so the checks ran against
un-rendered Jinja expressions and an empty rendered value was never
caught. The validation now runs at execute time, and the raises are
narrowed from AirflowException to ValueError per the ongoing exception
clean-up.
@mitre88
mitre88 requested a review from shahar1 as a code owner July 27, 2026 15:40
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Jul 27, 2026
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 28, 2026
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.api_version = api_version
self._validate_inputs()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaxil Because body is a template field, so if not self.body in __init__ reads the unrendered value: a Jinja template string like "{{ ti.xcom_pull(...) }}" is always truthy at construction time, so the check can never catch a body that renders empty — and per the template-field validation burn-down (#70296), value checks on templated parameters belong after rendering. Moving the call to execute() validates the rendered value. (An argument-provision check would stay in __init__ per #70505, but not self.body is a truthiness/value check, which is exactly the kind that must move.)

Same rationale for the speech_to_text/text_to_speech changes in this PR.


Drafted-by: Claude Code (Fable 5) (no human review before posting)

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up — but I think the classification here needs revisiting, and the conclusion changes the shape of the PR.

These are provision checks, and #70296 says they must stay in __init__

Your reply to kaxil cites #70505 as sanctioning the move, but #70505 is the PR that narrowed the rule in the opposite direction — to keep exactly these checks in the constructor. #70296's "False positives" section is explicit:

A check that only asks whether an argument was passed … belongs in __init__ and must not be moved. … Fix these by rewriting in place, not by moving.

with two reasons that both apply here: under render_template_as_native_obj=True a provided field renders to None, so the same check in execute() reports a supplied argument as missing; and raising at construction surfaces a static authoring mistake as a Dag import error rather than on a worker once per task instance and per retry.

Five already-merged burn-down PRs moved provision checks that should have stayed — reverting them is tracked in #70503. I'd rather not add three more.

What I think the right shape is

The bug you're describing is real — a templated audio / body that renders empty is never caught. But #70296 answers that case directly: "a provision check in __init__ guarantees nothing about the rendered value: code in execute() that needs the field set still needs its own guard." The documented model is S3DeleteObjectsOperator, where both copies stay — the __init__ copy catches the authoring mistake at parse time, the execute() copy catches the rendered value.

So: keep the constructor check, and add a guard in execute() — rather than relocating the single check.

Two things that make this more than a mechanical edit, worth thinking through:

  • if not self.body and if self.audio == "" are truthiness / equality tests. #70296 describes truthiness as asking a third question that matches neither provision nor value, so neither existing check is already in the sanctioned shape.
  • #70505 warns that rewriting a required-argument check to is None loosens it. body, audio, config, input_data, voice, audio_config, target_bucket_name and target_filename are all required keyword arguments, so if body is None would start accepting body={}, which the current check rejects. Please don't just swap the polarity.

On AirflowExceptionValueError

No objection — ValueError is the right call for a constructor-time authoring error, and it avoids introducing a new Airflow-exception usage that the ongoing clean-up would only have to unwind later. Keep that part.

Scope note

None of these three classes appears in validate_operators_init_exemptions.txt — the hook never flagged them, since it can't see reads behind a _validate_inputs() call. So this is voluntary work outside the burn-down's tracked set, which is likely why the provision/value classification didn't get caught by the usual guardrails. Not a problem in itself, just worth knowing why there was no safety net.

Verified, and holds up regardless of the above

  • All eight parameters really are template_fields, so the underlying gap is real.
  • 13/13 tests pass on the branch; reverting only the three source files fails 7 of them, including both new tests. The coverage is honest.
  • generated/known_airflow_exceptions.txt drops exactly the three entries; no leftover references; no :raises: docstrings to update.
  • cloud/operators/translate_speech.py already raises inside execute() — no adjacent gap.
  • No changelog entry needed; the google changelog is regenerated from git log.

Separate follow-up, not for this PR

The _validate_inputs()-behind-a-helper blind spot isn't limited to these three: cloud/operators/bigtable.py:51 (BigtableValidationMixin, whose REQUIRED_ATTRIBUTES for BigtableCreateInstanceOperator are all template_fields) and cloud/operators/compute.py:70 have the same shape and are equally invisible to the hook. Worth teaching validate_operators_init.py to follow one level of helper call so the burn-down can actually see them.


This review was drafted by an AI-assisted tool and confirmed by an Apache Airflow maintainer. After you've addressed the points above and pushed an update, an Apache Airflow maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review: contributing-docs/05_pull_requests.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting

raise ValueError("The required parameter 'body' is missing")

def execute(self, context: Context):
self._validate_inputs()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the move I'd like to reconsider — and it's the same question kaxil raised above.

if not self.body asks whether a meaningful body was supplied, which #70296 classes as a provision check, and provision checks are the documented exception that must stay in __init__:

A check that only asks whether an argument was passed … belongs in __init__ and must not be moved. … Fix these by rewriting in place, not by moving.

The two stated reasons both bite here: with render_template_as_native_obj=True a provided field can render to None, so this check in execute() will report a supplied argument as missing; and moving it turns a static authoring mistake into a per-task-instance, per-retry runtime failure instead of a Dag import error.

#70505 — which you cite above — is the PR that narrowed the hook to permit this check in the constructor, not to require moving it. Five merged PRs already made this move and are queued for revert in #70503.

The rendered-value gap you're fixing is real, though. #70296's answer is to have both: "a provision check in __init__ guarantees nothing about the rendered value: code in execute() that needs the field set still needs its own guard." S3DeleteObjectsOperator is the worked example — both copies stay. So I'd keep the __init__ check and add this one, rather than relocate it.

One trap if you do rewrite the constructor copy: #70505 warns that turning a required-argument check into is None loosens it. body is required, so if body is None would start accepting body={}, which today's check rejects.


Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants