-
Notifications
You must be signed in to change notification settings - Fork 271
Simplify gpu transform operations in iceberg #13844
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Renjie Liu <[email protected]>
Greptile OverviewGreptile SummaryRefactored Iceberg transform expression handling to eliminate code duplication. Previously, Key changes:
Confidence Score: 5/5
Important Files ChangedFile Analysis
Sequence DiagramsequenceDiagram
participant Client
participant IcebergProviderImpl
participant GpuTransform
participant Transform Objects
Note over Client,Transform Objects: tagForGpu Flow
Client->>IcebergProviderImpl: tagForGpu(StaticInvoke, meta)
IcebergProviderImpl->>GpuTransform: apply(StaticInvoke)
GpuTransform->>GpuTransform: Match staticObject class name
alt BucketInt/BucketLong
GpuTransform-->>IcebergProviderImpl: Success(GpuBucket)
else Years/Months/Days/Hours Functions
GpuTransform-->>IcebergProviderImpl: Success(GpuYears/Months/Days/Hours)
else Unsupported
GpuTransform-->>IcebergProviderImpl: Failure(IllegalArgumentException)
IcebergProviderImpl->>Client: meta.willNotWorkOnGpu()
end
IcebergProviderImpl->>Transform Objects: transform.tagExprForGpu(meta)
Transform Objects->>Transform Objects: Validate argument types
Transform Objects-->>IcebergProviderImpl: Tag as supported/unsupported
Note over Client,Transform Objects: convertToGpu Flow
Client->>IcebergProviderImpl: convertToGpu(StaticInvoke, meta)
IcebergProviderImpl->>GpuTransform: apply(StaticInvoke)
GpuTransform-->>IcebergProviderImpl: Success(transform)
IcebergProviderImpl->>Transform Objects: transform.convertToGpu(expr, meta)
Transform Objects->>Transform Objects: Convert child expressions
Transform Objects-->>IcebergProviderImpl: GpuExpression
IcebergProviderImpl-->>Client: GpuExpression
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
7 files reviewed, no comments
| def apply(expr: StaticInvoke): Try[GpuTransform] = { | ||
| val staticObj = expr.staticObject | ||
| val name = staticObj.getName | ||
| if (name.equals(classOf[BucketInt].getName) || |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you working inequality due to classloading via name comparison ? Otherwise could do
def apply(expr: StaticInvoke): Try[GpuTransform] = Try {
expr.staticObject match {
case c if c == classOf[BucketInt] || c == classOf[BucketLong] => GpuBucket(0)
case c if c == classOf[DateToYearsFunction] || c == classOf[TimestampToYearsFunction] => GpuYears
...
}
}|
NOTE: release/26.02 has been created from main. Please retarget your PR to release/26.02 if it should be included in the release. |
Fixes #13832 .
Description
A little refactoring about gpu support of iceberg's transform, removing redundant codes.
Checklists
(Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)