#3016 New `expand` PPL command by currantw · Pull Request #3305 · opensearch-project/sql

currantw · 2025-02-06T01:05:52Z

Signed-off-by: currantw taylor.curran@improving.com

Description

Adds a new expand PPL command to OpenSearch SQL that create multiple rows corresponding to the elements on an multi-valued field. If the field contains a single value (not an array, including null or missing), the row is not modified.

This does not support expanding object fields. This is handled by the flatten command - see #3030.

Related Issues

Resolves #3030

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
~~API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

currantw · 2025-02-06T01:09:19Z

+  public boolean equal(ExprValue other) {
+
+    if (!(other instanceof ExprTupleValue)) {
      return false;
-    } else {
-      ExprTupleValue other = (ExprTupleValue) o;
-      Iterator<Entry<String, ExprValue>> thisIterator = this.valueMap.entrySet().iterator();
-      Iterator<Entry<String, ExprValue>> otherIterator = other.valueMap.entrySet().iterator();
-      while (thisIterator.hasNext() && otherIterator.hasNext()) {
-        Entry<String, ExprValue> thisEntry = thisIterator.next();
-        Entry<String, ExprValue> otherEntry = otherIterator.next();
-        if (!(thisEntry.getKey().equals(otherEntry.getKey())
-            && thisEntry.getValue().equals(otherEntry.getValue()))) {
-          return false;
-        }
-      }
-      return !(thisIterator.hasNext() || otherIterator.hasNext());
    }
+
+    /**
+     * {@link Map#equals} returns true if the two maps' entry sets are equal, and works properly
+     * across all implementation of the {@link Map} interface. See {@link
+     * https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#equals-java.lang.Object-} for
+     * more details.
+     */
+    return valueMap.equals(other.tupleValue());
  }


Change pulled from flatten PR. See discussion there.
#3267

currantw · 2025-02-06T01:18:08Z


 Usage: nullif(field1, field2) return null if two parameters are same, otherwise return field1.

-Argument type: all the supported data type, (NOTE : if two parameters has different type, if two parameters has different type, you will fail semantic check)


Just a typo that I came across. Unrelated.

currantw · 2025-02-06T01:18:59Z

+        result,
+        rows("Seattle Seahawks", 2014),
+        rows("Seattle Kraken", null),
+        rows("Vancouver Canucks", null),


Apologies to any Canucks fans... 😢

currantw · 2025-02-06T01:26:49Z

+import org.opensearch.sql.exception.SemanticCheckException;
+
+@UtilityClass
+public class PathUtils {


Added this new utilities module for dealing with paths. Includes unit tests. Once flatten has been merged (see here, I will also update that implementation to use these methods.

currantw · 2025-02-06T01:48:27Z

+The ``expand`` command expands a field that contains an array of values to produce a seperate row for each value in the
+array. If the field does not contain an array, the row is not modified.
+
+Syntax
+============
+
+``expand <field>``
+
+* ``field``: reference to the field to flatten.
+
+Example 1: Expand a field
+=========================
+
+PPL query::
+
+    os> source=expand | expand team | fields city, team.name
+    fetched rows / total rows = 7/7
+    +--------------+-------------------+
+    | city         | team.name         |
+    |--------------+-------------------|
+    | Seattle      | Seattle Seahawks  |
+    | Seattle      | Seattle Kraken    |
+    | Vancouver    | Vancouver Canucks |
+    | Vancouver    | BC Lions          |
+    | San Antonio  | San Antonio Spurs |
+    | Null Team    | null              |
+    | Missing Team | null              |
+    +--------------+-------------------+


Bit of a difference from both Splunk and OpenSearch Spark.

This support multi-values and single values. If expand is applied to a field that contains both scalars (i.e. single values, including null and missing) and multi-values (i.e. arrays), then the arrays are expanded into separate rows, while the scalar rows remain unchanged. An empty array appears to be converted to null "under-the-hood" by OpenSearch Spark (see OpenSearchExprValueFactoryTest.constructNullArrayValue), so an empty array is handled the same as a null value.

currantw · 2025-02-06T01:52:27Z

+    /** With {@link org.opensearch.sql.data.model.ExprMissingValue} */
+    inputRow = ExprValueUtils.tupleValue(Map.of());
+    mockInput(inputRow);
+
+    actualRows = execute(expand(inputPlan, DSL.ref("scalar_missing", ARRAY)));
+    expectedRows = List.of(inputRow);
+
+    assertEquals(expectedRows, actualRows);
+
+    /** Without {@link org.opensearch.sql.data.model.ExprMissingValue} */
+    inputRow = ExprValueUtils.tupleValue(Map.of("scalar_missing", missingExprValue));
+    mockInput(inputRow);
+
+    actualRows = execute(expand(inputPlan, DSL.ref("scalar_missing", ARRAY)));
+    expectedRows = List.of(inputRow);
+
+    assertEquals(expectedRows, actualRows);


When debugging, I found that "missing" values seem to simply by missing from the ExprValue mapping when they get read into OpenSearch SQL, rather than being explicitly mapped to ExprMissingValue. I test both cases here.

Signed-off-by: currantw <taylor.curran@improving.com>

…ation. Signed-off-by: currantw <taylor.curran@improving.com>

Signed-off-by: currantw <taylor.curran@improving.com>

LantaoJin · 2025-06-23T12:26:03Z

Resolved by #3745

currantw requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners February 6, 2025 01:05

currantw commented Feb 6, 2025

View reviewed changes

currantw marked this pull request as draft February 6, 2025 01:54

currantw force-pushed the #3016_expand_command branch 2 times, most recently from 8f6838b to 7e0a31b Compare February 12, 2025 23:50

currantw added 7 commits February 12, 2025 18:34

Add flatten command to ANTLR lexer and parser.

f462e2a

Signed-off-by: currantw <taylor.curran@improving.com>

Skeleton implementation, tests, and documents with lots of TODOs.

69f0b1a

Signed-off-by: currantw <taylor.curran@improving.com>

Initial implementation

c1ac737

Signed-off-by: currantw <taylor.curran@improving.com>

Fix typo

366e162

Signed-off-by: currantw <taylor.curran@improving.com>

Initial implementation

0cbd8d4

Signed-off-by: currantw <taylor.curran@improving.com>

Update/fix tests.

26e9443

Signed-off-by: currantw <taylor.curran@improving.com>

Update integration tests to align with doc tests.

237b69e

Signed-off-by: currantw <taylor.curran@improving.com>

currantw added 26 commits February 12, 2025 18:48

Initial implementation of skeleton classes and methods.

ab5a2fe

Signed-off-by: currantw <taylor.curran@improving.com>

Implement some of the expand logic

e7e5a5a

Signed-off-by: currantw <taylor.curran@improving.com>

Add PathUtils and unit tests.

a8f6855

Signed-off-by: currantw <taylor.curran@improving.com>

Update ExpandOperator and unit tests.

cf357ea

Signed-off-by: currantw <taylor.curran@improving.com>

Implement integration tests, update Expand logic, rename data set.

c260f6b

Signed-off-by: currantw <taylor.curran@improving.com>

Implement integration tests, update Expand logic, rename data set.

0d3c33c

Signed-off-by: currantw <taylor.curran@improving.com>

Add expand.rst documentation and further updates to tests/implement…

209326d

…ation. Signed-off-by: currantw <taylor.curran@improving.com>

Unrelated typo fix

dd9a024

Signed-off-by: currantw <taylor.curran@improving.com>

Cleanup, modify to return null for an empty array.

e31df37

Signed-off-by: currantw <taylor.curran@improving.com>

Fix test_docs.py typo, order alphabetically.

274719d

Signed-off-by: currantw <taylor.curran@improving.com>

Minor cleanup, mostly alphabetizing constants.

459ca24

Signed-off-by: currantw <taylor.curran@improving.com>

Add new doc and integration tests

eef907f

Signed-off-by: currantw <taylor.curran@improving.com>

Fix missing test coverage

77afac7

Signed-off-by: currantw <taylor.curran@improving.com>

Move PathUtils to ExprValueUtils and update tests.

49078e9

Signed-off-by: currantw <taylor.curran@improving.com>

Use ExprValueUtils to simplify FlattenOperator

ff0d5ae

Signed-off-by: currantw <taylor.curran@improving.com>

Simplify and make consistent flatten and expand operators.

790104c

Signed-off-by: currantw <taylor.curran@improving.com>

Update ExprValueUtils and unit tests.

ec47f8f

Signed-off-by: currantw <taylor.curran@improving.com>

Make constants private

101b653

Signed-off-by: currantw <taylor.curran@improving.com>

Spotless

ec0f44f

Signed-off-by: currantw <taylor.curran@improving.com>

Add hashCode unit tests

da7738a

Signed-off-by: currantw <taylor.curran@improving.com>

Trivial documentation cleanup.

4284f6f

Signed-off-by: currantw <taylor.curran@improving.com>

Fix doctest

1e51881

Signed-off-by: currantw <taylor.curran@improving.com>

General cleanup, combine flatten and expand datasets.

c4f732e

Signed-off-by: currantw <taylor.curran@improving.com>

Spotless

582fdaa

Signed-off-by: currantw <taylor.curran@improving.com>

Fix failing doctests

e8494f2

Signed-off-by: currantw <taylor.curran@improving.com>

Fix ExplainIT test

028e074

Signed-off-by: currantw <taylor.curran@improving.com>

currantw force-pushed the #3016_expand_command branch from 7e0a31b to 028e074 Compare February 13, 2025 03:10

Handle empty collections.

01569ae

Signed-off-by: currantw <taylor.curran@improving.com>

LantaoJin closed this Jun 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3016 New `expand` PPL command#3305

#3016 New `expand` PPL command#3305
currantw wants to merge 81 commits into
opensearch-project:mainfrom
Bit-Quill:#3016_expand_command

currantw commented Feb 6, 2025

Uh oh!

currantw Feb 6, 2025

Uh oh!

currantw Feb 6, 2025

Uh oh!

currantw Feb 6, 2025

Uh oh!

currantw Feb 6, 2025

Uh oh!

currantw Feb 6, 2025 •

edited

Loading

Uh oh!

currantw Feb 6, 2025

Uh oh!

LantaoJin commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		Usage: nullif(field1, field2) return null if two parameters are same, otherwise return field1.

		Argument type: all the supported data type, (NOTE : if two parameters has different type, if two parameters has different type, you will fail semantic check)

Conversation

currantw commented Feb 6, 2025

Description

Related Issues

Check List

Uh oh!

currantw Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

currantw Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

currantw Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

currantw Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

currantw Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

currantw Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

currantw Feb 6, 2025 •

edited

Loading