Describe the bug
Comet currently fails with an overflow when ANSI mode is disabled, but we should return the original value instead.
Create test data
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
Test with ansi mode disabled
With ansi mode disabled we expect abs to return the original value if there would be an overflow calculating the abs value, but Comet fails with an overflow.
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, abs(a) from t").show
+-----------+-----------+
| a| abs(a)|
+-----------+-----------+
| 2147483647| 2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, abs(a) from t").show
24/05/23 13:26:17 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
24/05/23 13:26:17 ERROR Executor: Exception in task 0.0 in stage 7.0 (TID 8)
org.apache.comet.CometNativeException: Arrow error: Compute error: Int32Array overflow on abs(-2147483648)
With ansi mode enabled, we do expect an exception, but we should try and match Spark's exception.
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] integer overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
Comet currently has this error:
org.apache.comet.CometNativeException: Arrow error: Compute error: Int32Array overflow on abs(-2147483648)
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Describe the bug
Comet currently fails with an overflow when ANSI mode is disabled, but we should return the original value instead.
Create test data
Test with ansi mode disabled
With ansi mode disabled we expect abs to return the original value if there would be an overflow calculating the abs value, but Comet fails with an overflow.
With ansi mode enabled, we do expect an exception, but we should try and match Spark's exception.
Comet currently has this error:
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response