Error using `IN` list on dictionary encoded data: `InList does not support datatype Dictionary(Int32, Utf8).`

[alamb]

Describe the bug
I am writing a query to select some values from a dictionary encoded string column trace_id and I get an error

select * from spans where trace_id   IN ('187dcbbc68a83a0d', '335424324532');

Results in this error

This feature is not implemented: InList does not support datatype Dictionary(Int32, Utf8).

To Reproduce
Run this test:

diff --git a/datafusion/core/tests/sql/predicates.rs b/datafusion/core/tests/sql/predicates.rs
index 07e016a27..5109f0167 100644
--- a/datafusion/core/tests/sql/predicates.rs
+++ b/datafusion/core/tests/sql/predicates.rs
@@ -427,6 +427,32 @@ async fn csv_in_set_test() -> Result<()> {
     Ok(())
 }

+#[tokio::test]
+async fn in_set_string_dictionaries() -> Result<()> {
+    let input = vec![Some("foo"), Some("bar"), None, Some("fazzz")]
+        .into_iter()
+        .collect::<DictionaryArray<Int32Type>>();
+
+    let batch = RecordBatch::try_from_iter(vec![("c1", Arc::new(input) as _)]).unwrap();
+
+    let ctx = SessionContext::new();
+    ctx.register_batch("test", batch)?;
+
+    let sql = "SELECT * FROM test WHERE c1 IN ('foo', 'Bar', 'fazz')";
+    let actual = execute_to_batches(&ctx, sql).await;
+    let expected = vec![
+        "+-------+",
+        "| c1    |",
+        "+-------+",
+        "| foo   |",
+        "| fazzz |",
+        "+-------+",
+    ];
+
+    assert_batches_eq!(expected, &actual);
+    Ok(())
+}
+

Results in

thread 'sql::predicates::in_set_string_dictionaries' panicked at 'called `Result::unwrap()` on an `Err` value: "ArrowError(ExternalError(NotImplemented(\"InList does not support datatype Dictionary(Int32, Utf8).\"))) at ...

Expected behavior
Test should pass

Additional context
Found this in IOx as part of https://github.com/influxdata/influxdb_iox/issues/5959

[alamb]

I plan to work on this later in the week if no one beats me to it

[jackwener]

related issue: #3766

[alamb]

@NGA-TRAN says she plans to work on this

[NGA-TRAN]

I am actively working on this