[BUG] PPL `rename` command breaks `WHERE` clause with functions - NullPointerException in Calcite filter pushdown

[alexey-temnikov]

Query Information

PPL Command/Query:

source=test-rename-bug 
| rename timestamp_field as @timestamp 
| where TIMESTAMPDIFF(MINUTE, @timestamp, NOW()) <= 60 
| timechart span=1minute COUNT()

Expected Result:
Query executes successfully, filtering documents where the renamed @timestamp field is within the last 60 minutes, then aggregating by 1-minute intervals.

Actual Result:

{
  "error": {
    "reason": "Error occurred in OpenSearch engine: all shards failed",
    "details": "QueryShardException[failed to create query: Failed to compile inline script [...] using lang [opensearch_compounded_script]]; nested: GeneralScriptException[...]; nested: NullPointerException",
    "type": "SearchPhaseExecutionException"
  },
  "status": 400
}

Reproduction Steps

Execute the following curl commands to reproduce:

# 1. Create test index
curl -X PUT "http://localhost:9200/test-rename-bug" -H 'Content-Type: application/json' -d '{
  "mappings": {
    "properties": {
      "timestamp_field": {"type": "date_nanos"},
      "value": {"type": "integer"}
    }
  }
}'

# 2. Insert test data
curl -X POST "http://localhost:9200/test-rename-bug/_doc" -H 'Content-Type: application/json' -d '{
  "timestamp_field": "2025-10-24T23:45:00.000Z",
  "value": 100
}'

# 3. Refresh index
curl -X POST "http://localhost:9200/test-rename-bug/_refresh"

# 4. Run failing query
curl -X POST "http://localhost:9200/_plugins/_ppl" -H 'Content-Type: application/json' -d '{
  "query": "source=test-rename-bug | rename timestamp_field as @timestamp | where TIMESTAMPDIFF(MINUTE, @timestamp, NOW()) <= 60 | timechart span=1minute COUNT()"
}'

Result: NullPointerException error

Bug Description

Issue Summary:
When a field is renamed using the rename command and then used in a where clause with a function (like TIMESTAMPDIFF), the query fails with a NullPointerException during Calcite script compilation. This occurs because the script engine receives the renamed field name but only has access to the original field name mappings.

Impact:
This bug prevents users from using renamed fields in WHERE clauses with functions when Calcite pushdown is enabled. It's particularly problematic for observability use cases where users need to rename timestamp fields to @timestamp for compatibility with timechart (see related issue #4576).

Comparison:

❌ Failing (rename before where):

source=test-rename-bug 
| rename timestamp_field as @timestamp 
| where TIMESTAMPDIFF(MINUTE, @timestamp, NOW()) <= 60 
| timechart span=1minute COUNT()

Result: NullPointerException

✅ Working (where before rename):

source=test-rename-bug 
| where TIMESTAMPDIFF(MINUTE, timestamp_field, NOW()) <= 60 
| rename timestamp_field as @timestamp 
| timechart span=1minute COUNT()

Result: Returns correct data

Environment Information

OpenSearch Version: 3.4.0-SNAPSHOT (reproduced on latest main branch)

Additional Details:

• Requires plugins.calcite.enabled=true (default in 3.x)
• Only affects queries where renamed fields are used in WHERE clauses with functions
• Does not affect simple comparisons without functions

Root Cause Analysis

Tentative Root Cause:

The bug occurs in two locations:

1. Filter Pushdown (CalciteLogicalIndexScan.java:125-128):

List<String> schema = this.getRowType().getFieldNames();
Map<String, ExprType> fieldTypes =
    this.osIndex.getAllFieldTypes().entrySet().stream()
        .filter(entry -> schema.contains(entry.getKey()))
        .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));

After a rename operation:

• schema contains renamed field names (e.g., @timestamp)
• osIndex.getAllFieldTypes() contains original field names (e.g., timestamp_field)
• The filter fails to match, so fieldTypes doesn't include the renamed field mapping

2. Script Execution (CalciteScriptEngine.java:186-187):

String fieldName = rowType.getFieldList().get(index).getName();
ExprType exprType = fieldTypes.get(fieldName);

During script execution:

• fieldName gets the renamed field name from rowType (e.g., @timestamp)
• fieldTypes.get(fieldName) returns null because fieldTypes only has original names
• Causes NullPointerException when trying to use the null exprType

Workaround

Place the where clause before the rename command:

curl -X POST "http://localhost:9200/_plugins/_ppl" -H 'Content-Type: application/json' -d '{
  "query": "source=test-rename-bug | where TIMESTAMPDIFF(MINUTE, timestamp_field, NOW()) <= 60 | rename timestamp_field as @timestamp | timechart span=1minute COUNT()"
}'

Related Issues

• [BUG] PPL rename command breaks dedup operation - returns null values #4563 - rename breaks dedup operation (same root cause, different pushdown operation)
• [FEATURE] timechart with option to specify timestamp column, overriding the implicit @timestamp field #4576 - Feature request for timechart custom timestamp field (mentions workaround)

[songkant-aws]

I think a simple fix is probably removing this line .filter(entry -> schema.contains(entry.getKey())). Schema doesn't need to be the same with rowType as rowType is usually physical plan specific and schema is probably not.

Anyway, need to check the impact.

[LantaoJin]

I think a simple fix is probably removing this line .filter(entry -> schema.contains(entry.getKey())). Schema doesn't need to be the same with rowType as rowType is usually physical plan specific and schema is probably not.

Anyway, need to check the impact.

#4670 provided a similar fixing. Since the interface analyzeExpression requires the rowType as a parameter, just uses table scan's rowType() instead of filter.getRowType() in filter pushdown could also resolve the issue.