val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
Comet does not throw an exception but Spark does.
scala> spark.conf.set("spark.comet.ansi.enabled", true)
scala> spark.conf.set("spark.sql.ansi.enabled", true)
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+------------+
| a|round(a, -1)|
+-----------+------------+
| 2147483647| -2147483646|
|-2147483648| 2147483646|
+-----------+------------+
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:37 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:38 ERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 11) ==
select a, round(a,-1) from t
^^^^^^^^^^^
What is the problem the feature request solves?
Comet does not support ANSI mode for
round.Create test data
Test with ANSI enabled
Comet does not throw an exception but Spark does.
Describe the potential solution
No response
Additional context
No response