feat(providers/amazon): allow disabling hook-level lineage in S3Hook … by shnhdan · Pull Request #63499 · apache/airflow

shnhdan · 2026-03-13T09:17:35Z

Description

When set to False, all get_hook_lineage_collector() calls in the hook are
skipped, preventing asset spam during operations like multipart uploads.
Default behavior is unchanged.

Changes:

hooks/s3.py — new param; all lineage calls across load_file, load_string,
load_bytes, load_file_obj, copy_object, and download_file consistently gated
tests/test_s3.py — tests for default, disabled, and explicit enabled behavior

Note: Complementary to #66992 — that PR controls whether the OL listener
reads from HookLineageCollector at task end; this PR prevents the calls from
firing at the source in s3.py entirely.

Closes #63371

Was generative AI tooling used to co-author this PR?

Yes (Claude)

Generated-by: Claude following the guidelines

Important

🛠️ Maintainer triage note for @shnhdan · by @potiuk · 2026-06-22 06:31 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed (see the Pull Request quality criteria):

❌ Unaddressed Copilot review — a Copilot review thread has been open for ≥7 days with no reply; please respond to or resolve it. See the Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

…pache#63371)

eladkal · 2026-03-13T10:04:52Z

cc @kacpermuda can you clarify what is the desired behavior? I think this apply also for google storage and other cloud vendors. I'd rather we won't have different customizations for each provider.

SamWheating · 2026-03-14T16:47:09Z

@eladkal see the attached ticket which I opened earlier this week with a description of the motivations.

#63371

Agreed that this is definitely something we want to standardize across all relevant hooks, lets discuss it on the issue and maybe some of the other lineage contributors can weigh in.

kacpermuda · 2026-03-16T10:16:05Z

@shnhdan I believe the PR now contains only test, with no implementation on the actual hook. Maybe some faulty merge/rebase happened?

kacpermuda · 2026-03-16T10:18:24Z

Responded on the linked issue, not sure how, but we should make sure this arg is named consistently across providers.

shnhdan · 2026-03-16T13:14:25Z

@kacpermuda Fixed the rebase error and restored the implementation.All 154 S3 tests are passing locally. I’m following the naming discussion and am open to renaming the parameter to align with Airflow's consistency standards.

… kwargs.pop

Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>

kacpermuda · 2026-03-16T15:51:16Z

Thanks for working on this - the use case makes sense, and having a way to limit hook-level lineage emission can definitely be useful.

One thing I’d like to raise before we settle on the exact shape of the solution is that hook-level lineage is fundamentally a core feature, but the places where we will likely want to control it live in multiple providers (S3, GCS, etc.). Because of that, it would be good to think a bit about how to keep it consistent across providers without introducing a dependency on newer core versions.

In particular:

Global vs local control
It probably makes sense to have a cluster-level/default configuration, and then allow providers or hooks to override it when needed. That way DAG authors do not need to explicitly configure every hook if a cluster operator wants to disable or limit hook-level lineage globally. We can already effectively disable hook-level lineage since AF 3.2 by setting the asset collection limit to 0 (PR feat: Make Hook Level Lineage limits configurable #62010), but maybe we should consider introducing a more explicit config (for example a simple boolean to enable/disable HLL), and then allow hooks/providers to override it if needed.
Consistency across providers without a core dependency
This is a core feature that we will likely want to control in multiple providers, so we probably want to keep naming and semantics consistent. At the same time, we would like to do this without introducing a dependency on core, so that providers can implement it while still supporting multiple Airflow versions (basically so that what we introduce today still works on AF2.11).
It might be good to discuss this with the more task-sdk oriented contributors as well, as it feels more like an API boundary question between core and providers. One possible idea could be to keep an internal _BaseHook in core, while also exposing a BaseHook in a common provider layer, so that _BaseHook remains an internal implementation detail and BaseHook can be used more freely by providers. cc @ashb , curious what you think.

Because of that, it might be worth having a short discussion (either here or on devlist) about the intended approach. Maybe we end up doing exactly what this PR proposes, but it would be good to confirm that this is the direction we want and that we apply it consistently across providers.

Curious what others think. cc @eladkal @potiuk @mobuchowski

ashb · 2026-03-16T15:56:49Z

Honestly the name of this isn't right - "enable_hook_level_lineage" is what it does right now, but the intent is "no auto create assets". Not about disabling lineage entirely...

Also this sounds like it is too broad -- we might want to disable the auto creation of assets, but keep the metadata that "hook X accessed asset Y".

Additionally back to the original ask, maybe you need this at a per call level too -- you might want to disable it for the individual parts when uploaded, but still keep it when the single "upload complete" file is uploaded.

As for into basehook or not: yes perhaps, but I think that also depends on answers to ^^

shnhdan · 2026-03-17T07:37:14Z

I agree that standardising this in BaseHook is the better path for long-term consistency. Even if this shift moves the work toward the Core team, I'd like to remain involved and work alongside the maintainers to implement the final version. Following this conversation for the final direction.

ashb · 2026-03-17T09:37:06Z

@shnhdan There is no "core team" (or there is, but we don't have a monopoly on making changes to task-sdk etc.) PRs welcome in other words. Feel free to update this PR in place to become change to core.

shnhdan · 2026-03-17T10:30:41Z

@ashb Thanks for the clarification ! I'm happy to take this on. I'll work on moving the implementation to Core/Task SDK and will update this PR once the refactor is ready for review.

Copilot

Pull request overview

This PR adds a configuration switch to S3Hook intended to let users disable hook-level lineage emission (motivated by large numbers of per-object assets when uploading many files).

Changes:

Adds an enable_hook_level_lineage flag on S3Hook and gates some get_hook_lineage_collector() usage behind it.
Adds unit tests covering default/explicit values and load_string() lineage behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
`providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py`	Introduces the `enable_hook_level_lineage` flag and conditionally skips some lineage emission calls.
`providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py`	Adds tests for the new flag and for lineage behavior in `load_string()`.

    def __init__(
        self,
        aws_conn_id: str | None | ArgNotSet = AwsBaseHook.default_conn_name,
        transfer_config_args: dict | None = None,
        extra_args: dict | None = None,
        *args,
        **kwargs,
    ) -> None:
        kwargs["client_type"] = "s3"
        kwargs["aws_conn_id"] = aws_conn_id
        self._requester_pays = kwargs.pop("requester_pays", False)
+        self.enable_hook_level_lineage = kwargs.pop("enable_hook_level_lineage", True)



        get_hook_lineage_collector().add_input_asset(
            context=self, scheme="file", asset_kwargs={"path": filename}
        )
-        get_hook_lineage_collector().add_output_asset(
-            context=self, scheme="s3", asset_kwargs={"bucket": bucket_name, "key": key}
-        )
+        if self.enable_hook_level_lineage:
+            get_hook_lineage_collector().add_output_asset(
+                context=self, scheme="s3", asset_kwargs={"bucket": bucket_name, "key": key}
+            )


+class TestS3HookLineageConfig:
+    def test_hook_lineage_enabled_by_default(self):
+        from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+
+        hook = S3Hook()
+        assert hook.enable_hook_level_lineage is True
+
+    def test_hook_lineage_disabled_when_flag_false(self):
+        from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+
+        hook = S3Hook(enable_hook_level_lineage=False)
+        assert hook.enable_hook_level_lineage is False
+
+    from unittest import mock
+
+    @mock.patch("airflow.providers.amazon.aws.hooks.s3.get_hook_lineage_collector")
+    @mock.patch("airflow.providers.amazon.aws.hooks.s3.S3Hook.get_conn")
+    def test_load_string_skips_lineage_when_disabled(self, mock_conn, mock_collector):
+        from airflow.providers.amazon.aws.hooks.s3 import S3Hook


ramitkataria · 2026-05-27T15:18:01Z

Hey @shnhdan, any updates? Are you still planning to continue working on this or would you like me to take over?

kacpermuda · 2026-05-27T15:36:41Z

Just FYI, I'm working on a more generic solution on the OL provider level, that would allow controlling event emission and content in more details, for a more detailed scope. Draft PR is #66992. This might solve the above problem as well. We might think about adding a hook scope to the hook_lineage control there, to cover all occurrences of hook across operators, but even the current approach might be enough.

ashb · 2026-06-04T05:57:33Z

No response from OP. Closing

shnhdan · 2026-06-04T06:26:57Z

Hey @ashb — sorry for the delayed response, I've been busy. I did comment yesterday but it accidentally went to #66992 instead of here. I'm still actively working on this and would love to continue — would you be willing to reopen?

…ineage Previously load_file (input asset), copy_object (both assets), and download_file (both assets) were ungated — firing even when enable_hook_level_lineage=False. All get_hook_lineage_collector() calls in S3Hook are now consistently gated behind the flag. Also normalized getattr() guard to direct attribute access.

kacpermuda · 2026-06-08T11:47:03Z

Thanks @shnhdan, I think we can work without overlap here, those two PRs have the same goal, but you're focusing on producing side and I'm taking care of consuming side, which makes sense. For this PR, I think we can go with the solution you proposed, but maybe we can add a new config in lineage section of config, something very similar to default_deferrable, and then we can use this value as default for enable_hook_level_lineage in init of the hooks, as we do with deferrable in operators . This way, we keep the ability to provide this for single hook, but we also have a way to disable for all with single flag. Then, we can add this gating flag to more hooks. If we clearly state in the config description that not all hooks can honor this, we should be fine. Not all operators support deferrable as well, but we still have a default config.

@kacpermuda

Following the default_deferrable pattern, add a new [lineage] section to airflow.cfg with default_hook_lineage boolean (default: True). S3Hook.__init__ now reads this as the default for enable_hook_level_lineage instead of hardcoding True. Allows global disabling via config while permitting per-instance override. Usage: [lineage] default_hook_lineage = False # airflow.cfg S3Hook(enable_hook_level_lineage=False) # per-instance As suggested by @kacpermuda in review. Pattern can be extended to other hooks in future PRs. Related: apache#66992

shnhdan · 2026-06-14T14:39:56Z

@kacpermuda implemented as suggested — following the default_deferrable pattern:

Added [lineage] section to default_airflow.cfg with default_hook_lineage = True
S3Hook.__init__ now reads conf.getboolean("lineage", "default_hook_lineage", fallback=True) as the default
Per-instance enable_hook_level_lineage kwarg still works for individual override

Test results: 158 S3 tests passing, 988 lineage tests passing across amazon/openlineage/common.compat providers.

kacpermuda · 2026-06-15T10:15:32Z

+
+[lineage]
+
+# When enabled, hooks will collect lineage data by default.
+# Individual hooks can still override this via enable_hook_level_lineage kwarg.
+# Set to False to disable hook-level lineage collection globally (e.g. to suppress
+# intermediate asset spam during multipart uploads).
+default_hook_lineage = True
+


There already is a lineage section in airflow config, you can just add a new key there

potiuk · 2026-07-02T18:55:43Z

Note

🗑️ Closing stale draft · by @potiuk · 2026-07-02 18:53 UTC

This draft has been inactive for ~10 days. Closing to keep the queue tidy — no judgment on the change itself. Please reopen (or open a fresh PR) whenever you pick it back up, and a maintainer will be glad to take a look.

_{Automated triage — may be imperfect.}

feat(providers/amazon): allow disabling hook-level lineage in S3Hook (a…

275e30d

…pache#63371)

shnhdan requested a review from o-nikolas as a code owner March 13, 2026 09:17

boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 13, 2026

Merge branch 'main' into feature/disable-s3-hook-lineage

5316108

shnhdan added 2 commits March 14, 2026 12:13

Merge branch 'main' into feature/disable-s3-hook-lineage

adc141b

Merge branch 'apache:main' into feature/disable-s3-hook-lineage

3ff3ac4

fix: move enable_hook_level_lineage from S3Hook __init__ signature to…

0b3dec6

… kwargs.pop

shnhdan force-pushed the feature/disable-s3-hook-lineage branch from 9bb9c06 to 0b3dec6 Compare March 16, 2026 14:07

Merge branch 'main' into feature/disable-s3-hook-lineage

8ab405b

vincbeck reviewed Mar 16, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py Outdated

Comment thread providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py Outdated

shnhdan and others added 2 commits March 16, 2026 20:33

Update providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py

563e565

Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>

Merge branch 'main' into feature/disable-s3-hook-lineage

033c11f

Merge branch 'main' into feature/disable-s3-hook-lineage

8d94bdd

shnhdan added 3 commits March 19, 2026 11:56

Merge branch 'main' into feature/disable-s3-hook-lineage

bf9d56e

Merge branch 'apache:main' into feature/disable-s3-hook-lineage

aeeab02

Merge branch 'main' into feature/disable-s3-hook-lineage

e4cf3cb

kacpermuda mentioned this pull request Apr 2, 2026

Allow disabling hook-level lineage in Hook constructors #63371

Open

2 tasks

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 6, 2026

Merge branch 'apache:main' into feature/disable-s3-hook-lineage

8ec9a79

kaxil requested a review from Copilot April 10, 2026 19:55

Copilot started reviewing on behalf of kaxil April 10, 2026 19:56 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Merge branch 'apache:main' into feature/disable-s3-hook-lineage

935e04e

Merge branch 'main' into feature/disable-s3-hook-lineage

a85cb56

ashb closed this Jun 4, 2026

o-nikolas reopened this Jun 4, 2026

shnhdan added 2 commits June 8, 2026 13:12

Merge branch 'main' into feature/disable-s3-hook-lineage

624e0b2

shnhdan mentioned this pull request Jun 8, 2026

Add unified emission_policy config for OpenLineage provider #66992

Merged

1 task

shnhdan force-pushed the feature/disable-s3-hook-lineage branch from 57e6158 to b1c7886 Compare June 14, 2026 14:35

Merge branch 'main' into feature/disable-s3-hook-lineage

83e451b

kacpermuda reviewed Jun 15, 2026

View reviewed changes

potiuk assigned shnhdan Jun 17, 2026

potiuk marked this pull request as draft June 18, 2026 21:46

potiuk closed this Jul 2, 2026

Uh oh!

Conversation

shnhdan commented Mar 13, 2026 • edited by potiuk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Closes #63371

Was generative AI tooling used to co-author this PR?

Uh oh!

eladkal commented Mar 13, 2026

Uh oh!

SamWheating commented Mar 14, 2026

Uh oh!

kacpermuda commented Mar 16, 2026

Uh oh!

kacpermuda commented Mar 16, 2026

Uh oh!

shnhdan commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kacpermuda commented Mar 16, 2026

Uh oh!

ashb commented Mar 16, 2026

Uh oh!

shnhdan commented Mar 17, 2026

Uh oh!

ashb commented Mar 17, 2026

Uh oh!

shnhdan commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ramitkataria commented May 27, 2026

Uh oh!

kacpermuda commented May 27, 2026

Uh oh!

ashb commented Jun 4, 2026

Uh oh!

shnhdan commented Jun 4, 2026

Uh oh!

kacpermuda commented Jun 8, 2026

Uh oh!

shnhdan commented Jun 14, 2026

Uh oh!

kacpermuda Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

potiuk commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

shnhdan commented Mar 13, 2026 •

edited by potiuk

Loading

shnhdan commented Mar 16, 2026 •

edited

Loading