Fix path navigation on map columns for spath command

[dai-chen]

Description

QualifiedNameResolver.resolveFieldAccess previously wrapped nested map access with an alias unconditionally, causing issues in non-projection contexts. As a quick fix to unblock spath, this PR removes alias wrapping from resolveFieldAccess and adds aliasing back only in expandProjectFields. Longer term, we can consider clarifying the path navigation semantics similar to PartiQL and address issues tracked in #5152.

Example

spath input=doc | where doc.user.age = 30 is translated to LogicalFilter(condition=[=(SAFE_CAST(AS(ITEM($7, 'user.age'), 'doc.user.age')), 30.0E0)]), which later triggers an execution exception in the script engine because AS() is not valid.

opensearchsql> source=test_spath | spath input=doc | where doc.user.age = 30;

{'error': {'reason': 'Error occurred in OpenSearch engine: all shards failed',
'details': 'Shard[0]: java.lang.RuntimeException: cannot translate call AS($t3, $t4)\n\n
For more details, please send request for Json format to see the raw response from OpenSearch engine.', 'type': 'SearchPhaseExecutionException'}, 'status': 500})

opensearchsql> explain source=test_spath | spath input=doc | where doc.user.age = 30;

OpenSearchException({'calcite': {'logical': "
LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n
  LogicalProject(case=[$0], doc=[$7])\n
    LogicalFilter(condition=[=(SAFE_CAST(AS(ITEM($7, 'user.age'), 'doc.user.age')), 30.0E0)])\n      
      LogicalProject(case=[$1], _id=[$2], _index=[$3], _score=[$4], _maxscore=[$5], _sort=[$6], _routing=[$7], doc=[JSON_EXTRACT_ALL($0)])\n
        CalciteLogicalIndexScan(table=[[OpenSearch, test_spath]])\n",
'physical': '
EnumerableCalc(expr#0..1=[{inputs}], expr#2=[JSON_EXTRACT_ALL($t1)], case=[$t0], doc=[$t2])\n
  CalciteEnumerableIndexScan(table=[[OpenSearch, test_spath]],
    PushDownContext=[[PROJECT->[case, doc], SCRIPT->=(SAFE_CAST(AS(ITEM(JSON_EXTRACT_ALL($1), \'user.age\'), \'doc.user.age\')), 30.0E0), LIMIT->10000],
    OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":10000,"timeout":"1m","query":{"script":{"script":{"source":"{\\"langType\\":\\"calcite\\",\\"script\\":\\"..."}","lang":"opensearch_compounded_script","params":{"utcTimestamp":1771270144416573000,"SOURCES":[0,2,2,2],"DIGESTS":["doc","user.age","doc.user.age",30.0]}},"boost":1.0}},"_source":{"includes":["case","doc"],"excludes":[]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])\n'}})

Related Issues

Resolves #4307

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR modifies how dotted field access preserves user-visible aliases in Calcite PPL. Key changes fix alias handling in CalciteRelNodeVisitor's field expansion and QualifiedNameResolver's item access resolution, ensuring projected columns retain their original names when accessing nested paths. Documentation and tests are updated to reflect the auto-extract mode behavior.

Changes

Cohort / File(s)	Summary
Core field alias & item access handling core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java, core/src/main/java/org/opensearch/sql/calcite/QualifiedNameResolver.java	Modified field resolution for dotted field access (e.g., map.path): CalciteRelNodeVisitor now re-applies user-visible aliases when encountering ITEM function results, while QualifiedNameResolver removes alias wrapping and directly returns item access. This preserves column names in projections.
Documentation docs/user/ppl/cmd/spath.md	Updated auto-extract mode examples with guidance on dotted path navigation (e.g., doc.user.name resolves to user.name) and backtick quoting for keys with special characters. Revised sample queries and result tables to reflect new field extraction layout.
Test coverage for auto-extract mode integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathCommandIT.java, ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLSpathTest.java	Added four new integration test methods covering eval, where, stats, and sort operators with auto-extract mode. Added corresponding unit tests covering combinations with multiple operators and validating expected Logical and SparkSQL plans.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

• Support spath with dynamic fields #5058 — Extensive spath/dynamic-fields work that directly intersects with modifications to CalciteRelNodeVisitor and field-access/item-access resolution logic
• Add auto-extract mode for spath command #5140 — Introduces json_extract_all and flattened map key handling that builds upon the alias preservation changes for dotted field/item access
• Refactor alias type field by adding another project with alias #4881 — Refactors alias handling by moving it into a separate project/wrapper around scans, aligning with the alias handling changes in QualifiedNameResolver

Suggested labels

calcite

Suggested reviewers

• ykmr1224
• ps48
• kavithacm
• derek-ho
• joshuali925
• anirudha
• LantaoJin
• qianheng-aws
• yuancu

🚥 Pre-merge checks | ✅ 4 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (22 files): ⚔️ core/src/main/java/org/opensearch/sql/analysis/Analyzer.java (content) ⚔️ core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java (content) ⚔️ core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java (content) ⚔️ core/src/main/java/org/opensearch/sql/calcite/QualifiedNameResolver.java (content) ⚔️ core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java (content) ⚔️ core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/ArrayFunctionImpl.java (content) ⚔️ core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java (content) ⚔️ docs/category.json (content) ⚔️ docs/user/ppl/cmd/spath.md (content) ⚔️ docs/user/ppl/index.md (content) ⚔️ integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java (content) ⚔️ integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java (content) ⚔️ integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathCommandIT.java (content) ⚔️ integ-test/src/test/java/org/opensearch/sql/ppl/NewAddedCommandsIT.java (content) ⚔️ integ-test/src/test/java/org/opensearch/sql/security/CalciteCrossClusterSearchIT.java (content) ⚔️ ppl/src/main/antlr/OpenSearchPPLLexer.g4 (content) ⚔️ ppl/src/main/antlr/OpenSearchPPLParser.g4 (content) ⚔️ ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java (content) ⚔️ ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java (content) ⚔️ ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLFunctionTypeTest.java (content) ⚔️ ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLSpathTest.java (content) ⚔️ ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java (content) These conflicts must be resolved before merging into main.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Fix path navigation on map columns for spath command' clearly summarizes the main bug fix addressing nested map access issues in the spath command.
Linked Issues check	✅ Passed	The PR addresses issue #4307 by enabling spath to support dynamic columns through proper path navigation on map columns, allowing extraction of attributes as columns.
Out of Scope Changes check	✅ Passed	All changes are focused on fixing path navigation for the spath command: core resolver/visitor modifications, documentation updates, and comprehensive unit/integration tests.
Description check	✅ Passed	The pull request description clearly relates to the changeset, detailing a specific bug fix in QualifiedNameResolver and its resolution by moving alias wrapping logic.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

⚔️ Resolve merge conflicts (beta)

• Auto-commit resolved conflicts to branch support-map-path-navigation
• Post resolved changes as copyable diffs in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/user/ppl/cmd/spath.md (1)

146-173: ⚠️ Potential issue | 🟡 Minor

Verify single‑element array rendering in Example 5.
Row 2 shows doc.tags{} as python while the rules say arrays stringify like [a, b, c]. Please confirm the actual output and align either the rule or the row value (likely [python]).

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/user/ppl/cmd/spath.md` around lines 146 - 173, The example output for
spath flattening is inconsistent: the rules state arrays stringify as [a, b, c]
but the second row shows doc.tags{} as python; update the example so
single-element arrays are rendered as [python] (or, if behaviour is different,
change the rule text to state single-element arrays render without brackets) by
editing the spath example block and the bullet that describes arrays
(references: the spath command, output column doc.tags{}, input doc_auto and the
fields line) so the example and rule are consistent.

🧹 Nitpick comments (1)

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java (1)

433-484: Consider extracting helpers from expandProjectFields (>50 LOC).
This method now exceeds the 50‑line threshold; pulling wildcard handling and ITEM aliasing into helpers would keep the visitor easier to scan.

As per coding guidelines: “core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java: Flag methods >50 lines - this file is known to be hard to read”.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java`
around lines 433 - 484, The expandProjectFields method is too long; extract two
helpers to simplify it: 1) extract wildcard expansion logic into a helper like
expandWildcardFields(String fieldName, List<String> currentFields, Set<String>
addedFields, CalcitePlanContext context) that uses
WildcardUtils.expandWildcardPattern, filters with isMetadataField and
addedFields::add and adds context.relBuilder.field(...) to expandedFields; and
2) extract dotted-path ITEM handling into a helper like
resolveFieldWithItemAlias(Field field, CalcitePlanContext context) that calls
rexVisitor.analyze(field, context), checks for SqlKind.ITEM and applies
context.relBuilder.alias(..., fieldName) when needed; replace the inlined blocks
in expandProjectFields with calls to these helpers and keep the final
validateWildcardPatterns(projectList, currentFields) behavior unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@docs/user/ppl/cmd/spath.md`:
- Around line 146-173: The example output for spath flattening is inconsistent:
the rules state arrays stringify as [a, b, c] but the second row shows
doc.tags{} as python; update the example so single-element arrays are rendered
as [python] (or, if behaviour is different, change the rule text to state
single-element arrays render without brackets) by editing the spath example
block and the bullet that describes arrays (references: the spath command,
output column doc.tags{}, input doc_auto and the fields line) so the example and
rule are consistent.

---

Nitpick comments:
In `@core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java`:
- Around line 433-484: The expandProjectFields method is too long; extract two
helpers to simplify it: 1) extract wildcard expansion logic into a helper like
expandWildcardFields(String fieldName, List<String> currentFields, Set<String>
addedFields, CalcitePlanContext context) that uses
WildcardUtils.expandWildcardPattern, filters with isMetadataField and
addedFields::add and adds context.relBuilder.field(...) to expandedFields; and
2) extract dotted-path ITEM handling into a helper like
resolveFieldWithItemAlias(Field field, CalcitePlanContext context) that calls
rexVisitor.analyze(field, context), checks for SqlKind.ITEM and applies
context.relBuilder.alias(..., fieldName) when needed; replace the inlined blocks
in expandProjectFields with calls to these helpers and keep the final
validateWildcardPatterns(projectList, currentFields) behavior unchanged.

[dai-chen]

cc: @yuancu as this reverted changes in #4618