Skip to content

[BUG] Min/Max on derived fields cannot be pushed down #4571

@qianheng-aws

Description

@qianheng-aws

Query Information

PPL Command/Query:

POST localhost:9200/_plugins/_ppl/_explain

source=opensearch_dashboards_sample_data_logs 
| eval RamMB=CEIL(machine.ram/1000000.0) 
| stats MIN(RamMB) as Min_RAM_MB

Expected Result:

"physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch_dashboards_sample_data_logs]], PushDownContext=[[AGGREGATION->rel#80:LogicalAggregate.NONE.[](input=RelSubset#79,group={},Min_RAM_MB=MIN($0)), LIMIT->10000]...

Actual Result:

"physical": "
EnumerableLimit
    EnumerableAggregate
        CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch_dashboards_sample_data_logs]], PushDownContext=[[AGGREGATION->rel#80:LogicalAggregate.NONE.[](input=RelSubset#79,group={},Min_RAM_MB=MIN($0)), LIMIT->10000]...

Dataset Information

Dataset/Schema Type

  • OpenTelemetry (OTEL)
  • Simple Schema for Observability (SS4O)
  • Open Cybersecurity Schema Framework (OCSF)
  • Custom (details below)

Index Mapping

{
  "mappings": {
    "properties": {
      "machine": {
        "properties": {
          "ram": { "type": "long" },
          "os": { "type": "text" }
        }
      }
    }
  }
}

Sample Data

{
  "machine": {
    "ram": 8589934592,
    "os": "win 8"
  }
}

Bug Description

Issue Summary:
MIN/MAX on derived fields should be pushed down to be aggregate script, while failing to be pushed down with the latest code.

Steps to Reproduce:

  1. Create index with mapping like above
  2. Run explain command on the above PPL

Impact:

Environment Information

OpenSearch Version:
3.3.0-snapshot

Additional Details:
Seems to be introduced by this PR: #4281. Firstly found when investigating this issue: #4460

Screenshots

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languagebugSomething isn't workingpushdownpushdown related issues

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions