`SingleDistinctToGroupBy` panicked at No field named 'SPM_23134806_INFLUENCE_DEFAULT.period'

[waynexia]

Describe this problem

Backtrace

2022-03-10 11:42:59.471 ERRO [common_util/src/panic.rs:40] thread 'cse-read' panicked 'called `Result::unwrap()` on an `Err` value: Plan("No field named 'SPM_23134806_INFLUENCE_DEFAULT.period'. Valid fields are 'time_bucket(SPM_23134806_INFLUENCE_DEFAULT.period,Utf8(\"PT1M\"),Utf8(\"yyyy-MM-dd HH:mm:ss\"))', 'alias1'.")' at "/root/.cargo/git/checkouts/arrow-datafusion-78dc86050a92b8a7/9a77541/datafusion/src/optimizer/single_distinct_to_groupby.rs:106"
   0: common_util::panic::set_panic_hook::{{closure}}
             at /app/ceresdb/ceresdbx/common_util/src/panic.rs:39:18
   1: std::panicking::rust_panic_with_hook
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/std/src/panicking.rs:610:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/std/src/panicking.rs:502:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/std/src/sys_common/backtrace.rs:139:18
   4: rust_begin_unwind
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/std/src/panicking.rs:498:5
   5: core::panicking::panic_fmt
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/panicking.rs:107:14
   6: core::result::unwrap_failed
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/result.rs:1690:5
   7: core::result::Result<T,E>::unwrap
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/result.rs:1018:23
      datafusion::optimizer::single_distinct_to_groupby::optimize::{{closure}}
             at /root/.cargo/git/checkouts/arrow-datafusion-78dc86050a92b8a7/9a77541/datafusion/src/optimizer/single_distinct_to_groupby.rs:106:41
      core::iter::adapters::map::map_fold::{{closure}}
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/iter/adapters/map.rs:84:28
      core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/ops/function.rs:269:13
      core::iter::traits::iterator::Iterator::fold
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/iter/traits/iterator.rs:2171:21
      <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/iter/adapters/chain.rs:119:19
   8: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/iter/adapters/map.rs:124:9
      core::iter::traits::iterator::Iterator::for_each
             at /rustc/f1ce0e6a00593493a12e0e3662119786c761f375/library/core/src/iter/traits/iterator.rs:736:9
      <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at

Steps to reproduce

SQL:

SELECT   time_bucket(`period`, \'PT1M\', \'yyyy-MM-dd HH:mm:ss\') AS `time`,
         count(DISTINCT(pid))                                     AS `pid`
FROM     spm_23134806_influence_default
WHERE    `period` >= \'2022-03-10 10:41:00\'
AND      `period` < \'2022-03-10 11:42:00\'
GROUP BY time_bucket(`period`, \'PT1M\', \'yyyy-MM-dd HH:mm:ss\')
ORDER BY `time`

Expected behavior

Do not panic

Additional Information

To avoid this we have temporary comment out the optimizer SingleDistinctToGroupBy.

[waynexia]

apache/datafusion#2909 allows skipping failed opt rules. But this case will panic...

[jiacai2050]

Report to upstream

• single_distinct_to_groupby panic when group by expr is a binaryExpr datafusion#2994

[waynexia]

I tried apache/datafusion#3305 in my local env and it can pass:

~/repo/CeresDB main *5 !3 ?1                                                                                                                                       ruihang@ruihang 17:15:19
> curl --location --request POST 'http://127.0.0.1:5000/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "CREATE TABLE `spm_23134806_influence_default` (`pid` double NOT NULL, `period` timestamp NOT NULL, TIMESTAMP KEY(period)) ENGINE=Analytic with (enable_ttl='\''false'\'')"
}'
{"affected_rows":0}%


~/repo/CeresDB main *5 !3 ?1                                                                                                                                       ruihang@ruihang 17:16:38
> curl --location --request POST 'http://127.0.0.1:5000/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "select * from spm_23134806_influence_default"
}'
{"rows":[]}%


~/repo/CeresDB main *5 !3 ?1                                                                                                                                       ruihang@ruihang 17:16:46
> curl --location --request POST 'http://127.0.0.1:5000/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "SELECT   time_bucket(`period`, '\''PT1M'\'', '\''yyyy-MM-dd HH:mm:ss'\'') AS `time`, count(DISTINCT(pid)) AS `pid` FROM     spm_23134806_influence_default WHERE    `period` >= '\''2022-03-10 10:41:00'\'' AND      `period` < '\''2022-03-10 11:42:00'\'' GROUP BY time_bucket(`period`, '\''PT1M'\'', '\''yyyy-MM-dd HH:mm:ss'\'') ORDER BY `time`"
}'
{"rows":[]}%

[jiacai2050]

After #269, this should be fixed now 🚀

• https://github.com/apache/arrow-datafusion/blob/master/datafusion/CHANGELOG.md#1200-2022-09-12

[jiacai2050]

Ooop, the optimizer rule is still commented out, we should enable it again
https://github.com/CeresDB/ceresdb/blob/e5995be68e318fd66061c7034d6d1633081e4351/query_engine/src/context.rs#L77