Fix RecursionError in common.compat hook lineage add_extra polyfill#68735
Merged
kaxil merged 1 commit intoJun 19, 2026
Merged
Conversation
kacpermuda
approved these changes
Jun 19, 2026
Collaborator
|
Thanks Kaxil for fixing this, missed that for sure when writing tests |
_add_extra_polyfill re-patched the collector's collected_assets/has_collected class properties on every call (the trigger gate keys on instance-level _extra), stacking a wrapper layer per fresh collector until property access raised RecursionError on the Compat 3.1.x provider test matrix. Patch each class exactly once via a per-class marker set after all patches; production is unaffected (singleton collector).
kaxil
force-pushed
the
fix-compat-lineage-collector-recursion
branch
from
June 19, 2026 09:43
0508b97 to
7a07bce
Compare
vatsrahul1001
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Airflow < 3.2,
common.compat'sget_hook_lineage_collector()polyfillsadd_extrabymonkeypatching the
collected_assetsandhas_collectedclass properties of the lineagecollector.
_add_extra_polyfillwas not idempotent: the trigger gate (_lacks_add_extra_method)checks instance-level
_extra, so a fresh collector instance of an already-patched classre-enters the polyfill and re-wraps the class property, capturing the previously installed wrapper
as the "original". Each call stacks another layer onto the getter chain, so once enough collectors
are created the next
collected_assets/has_collectedaccess exceeds the recursion limit andraises
RecursionError.Where it shows up
In production the global collector is a process singleton (the accessor is
@cache-decorated) andthe polyfill only runs on Airflow < 3.2, so it is applied exactly once -- no impact. The failure
surfaces on the Compat 3.1.x provider test matrix, where a fresh
HookLineageCollectoris builtper test: the class accumulates one wrapper per test until the lineage edge-case tests blow the
recursion limit. This is a test-suite/robustness fix, not a production incident.
Fix
Capture the collector's true original
collected_assets/has_collectedgetters once per class(stashed in the class's own
__dict__) and always wrap that original, rather than whatever getter iscurrently installed:
layer re-patches
collected_assetsbefore each call. Wrapping the current value would capture theprevious compat wrapper as the "original", so the getter chain grew one layer per call until access
hit the recursion limit.
polyfill runs, while still re-applying the
extrawrapper each call -- so it composes correctlywith the Airflow 2 asset-naming layer that re-patches
collected_assets.__dict__, so a subclass that overrides these properties iscaptured in its own right rather than reusing a base class's original.
_extra/_extra_countsare initialized only when missing, so re-applying never clears acollector that already accumulated extras.
Behavior is otherwise unchanged: every collector still gets
add_extraand the extendedcollected_assets/has_collected.