Skip to content

Set larger limit get_partitions_by_filter in HiveMetastoreHook #21504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Feb 11, 2022
Merged

Set larger limit get_partitions_by_filter in HiveMetastoreHook #21504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion airflow/providers/apache/hive/hooks/hive.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,9 @@ def check_for_partition(self, schema: str, table: str, partition: str) -> bool:
True
"""
with self.metastore as client:
partitions = client.get_partitions_by_filter(schema, table, partition, 1)
partitions = client.get_partitions_by_filter(
schema, table, partition, HiveMetastoreHook.MAX_PART_COUNT

@eladkal eladkal Feb 10, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a breaking change?
is 1 is not desirable in all cases? (if there are valid use cases for 1 then maybe we should make the 4th parameter (limit) configurable thus allowing each user to set whatever value he wishes?

@pingzh pingzh Feb 11, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call. it isn't a breaking change since it is invoked inside check_for_partition, which returns bool.

@potiuk potiuk Feb 11, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct! Nice one :).

)

return bool(partitions)

Expand Down
8 changes: 6 additions & 2 deletions tests/providers/apache/hive/hooks/test_hive.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,15 +456,19 @@ def test_check_for_partition(self):

assert self.hook.check_for_partition(self.database, self.table, partition)

metastore.get_partitions_by_filter(self.database, self.table, partition, 1)
metastore.get_partitions_by_filter(
self.database, self.table, partition, HiveMetastoreHook.MAX_PART_COUNT
)

# Check for non-existent partition.
missing_partition = f"{self.partition_by}='{self.next_day}'"
metastore.get_partitions_by_filter = mock.MagicMock(return_value=[])

assert not self.hook.check_for_partition(self.database, self.table, missing_partition)

metastore.get_partitions_by_filter.assert_called_with(self.database, self.table, missing_partition, 1)
metastore.get_partitions_by_filter.assert_called_with(
self.database, self.table, missing_partition, HiveMetastoreHook.MAX_PART_COUNT
)

def test_check_for_named_partition(self):

Expand Down