4.0.3 Release Notes

[yiguolei]

4.0.2

Features

AI & Search

• Add inverted index NORMALIZER support
• Implement es-like boolean query [feature](inverted index) Implement es-like boolean query #58545
• Introduce lucene bool mode for search function [feature](search) introduce lucene bool mode for search function #59394

Lakehouse

• Support for Loading Catalog Credentials via AwsCredentialsProviderChain [feat](catalog)Support for Loading Catalog Credentials via AwsCredentialsProviderChain #58740
• Support Paimon DLF Catalog Using OSSHDFS Storage [feat](Catalog)Support Paimon DLF Catalog Using OSSHDFS Storage #59245
• Add manifest-level cache for Iceberg tables [Feature](iceberg) Add manifest-level cache for Iceberg tables to reduce I/O and parsing overhead #59056

Query Engine

• Support INTERVAL function and fix EXPORT_SET [Feature](func) Support INTERVAL function and fix EXPORT_SET constant process #58885
• Support function TIME_FORMAT [Feature](function) Support function TIME_FORMAT #58592
• Support function QUANTILE_STATE_TO/FROM_BASE64 [Feature](func) Support function QUANTILE_STATE_TO/FROM_BASE64 #59664

Improvements

• Introduce load job system table [feat](load) introduce load job system table #57421
• Enables views, materialized views, generated columns, and alias functions to persist session variables [feature](variable) enables views, materialized views, generated columns, and alias functions to persist session variables #58031
• Add sql received from table query plan action into audit log [feature](QueryPlanAction)add sql received from table query plan action into audit log #58739
• Enable stream load record to audit log system table [feat](load) enable stream load record to audit log system table #57530
• Optimize Complex Type Column Reading with Column Purning
• Compatible with mysql MOD syntax [improve](syntax) compatiable with mysql MOD syntax #58432
• Add dynamic configuration for sql_digest generation [feature](audit) add dynamic configuration for sql_digest generation #59102
• Use Youngs-Cramer for REGR_SLOPE/INTERCEPT to align with PG [fix](regr) Use Youngs-Cramer for REGR_SLOPE/INTERCEPT to align with PG #55940

Bugfixes

• Fix JNI global reference leak in JdbcConnector close [fix](jdbc) Fix JNI global reference leak in JdbcConnector close #58574
• Fix sync mv could not be chosen by cbo stable because stats upload from be not in time [fix](mv) Fix sync mv could not be chosen by cbo stable because stats upload from be not in time #58720
• Replace invalid JSONB with default JSONB null value [fix](json) Replace invalid JSONB with default JSONB null value #59007
• Fix NPE in OlapTableSink.createPaloNodesInfo due to concurrent drop backend [Fix](fe) Fix NPE in OlapTableSink.createPaloNodesInfo due to concurrent drop backend #58999
• Fix FROM DUAL incorrectly matching table names starting with dual [fix](parser) Fix FROM DUAL incorrectly matching table names starting with dual #59003
• Fix warm up cancel failure when BE is down [fix](filecache) fix warm up cancel failure when BE is down #58035
• Fix mv rewrite failed when mv is rewritten by LimitAggToTopNAgg but query is not [fix](mtmv) Fix mv rewrite failed when mv is rewritten by LimitAggToTopNAgg but query is not #58974
• Fix lastUpdateTime not updated on refresh and add scheduled refresh logs [fix](catalog) Fix lastUpdateTime not updated on refresh and add scheduled refresh logs #58997
• Fix core if hll_from_base64 input invalid [fix](core) fix core if hll_from_base64 input invalid #59106
• Fix the sensitivity issue of load column mapping with expressions [fix](load) Fix the sensitivity issue of load column mapping with expressions #59149
• Fix drop table not drop constraint related info [fix](pkfk) Fix drop table not drop constraint related info #58958
• Fix parquet topn lazy mat complex data error result [fix](parquet)fix parquet topn lazy mat complex data error result #58785
• Always create data and index page cache to avoid null pointer [fix](cache) always create data and index page cache to avoid null pointer #59266
• Modify tablet cooldownConfLock to reduce memory [fix](fe) modify tablet cooldownConfLock to reduce memory #59356
• Fix read parquet footer missing profile
• Fix potential use-after-free in Exception::to_string [Fix](Exception) Fix potential use-after-free because Exception::to_string is not thread safe #59558
• Fix float field to_string
• Fix hudi parquet read cause be core [fix](parquet)fix hudi parquet read hoodie.datasource.write.drop.partition.columns prop table cause be core. #58532
• Fix Kerberos auth config detection [Fix](Hdfs)Fix Kerberos auth config detection and auto-enable hive.metastore.sasl #59748
• Fix sync failure under empty tables [Fix](Streamingjob) Fix the issue of synchronization failure under empty tables #59735
• Fix parquet type not handle float16 [bug](parquet) fix parquet type not handle float16 type #58528
• Fix BM25 LENGTH_TABLE norm decoding [fix](inverted index) fix BM25 LENGTH_TABLE using byte4_to_int for correct norm decoding #59713
• Avoid false alarm of some datelike functions [Fix](function) avoid false alarm of some datelike functions #59897

[Nj-kol]

@yiguolei Could you please enable Azure vault for compute storage separation in this release? We are stuck because of this

Ref - #58605 (comment)

[CalvinKirs]

@yiguolei Could you please enable Azure vault for compute storage separation in this release? We are stuck because of this

Ref - #58605 (comment)

@Nj-kol Are you using ARM or x64, and is this the official build?
We have ARM binaries, but they don’t include Azure since Azure packages aren’t available on ARM.
Azure is enabled as a compile-time option only for x86, so it’s included in the official x86 releases.

[Nj-kol]

@yiguolei Could you please enable Azure vault for compute storage separation in this release? We are stuck because of this
Ref - #58605 (comment)

@Nj-kol Are you using ARM or x64, and is this the official build? We have ARM binaries, but they don’t include Azure since Azure packages aren’t available on ARM. Azure is enabled as a compile-time option only for x86, so it’s included in the official x86 releases.

Hi @CalvinKirs ,

We are currently running Doris 3.1.3 in disaggregated mode with Azure Blob as Vault. It is running perfectly fine, but when we tried to run Doris 4.0.1 with the same it complained it is not supported and @yiguolei mentioned it is a bug and should be solved in Doris 4.0.3

Also, we run Doris in Kubernetes so we rely on docker images published to docker hub. And yes we used x86 arch because arm is not supported for foundation DB 7.1.x which Doris expects.

Kindly help us in unblocking this in 4.0.3

[CalvinKirs]

@yiguolei Could you please enable Azure vault for compute storage separation in this release? We are stuck because of this
Ref - #58605 (comment)

@Nj-kol Are you using ARM or x64, and is this the official build? We have ARM binaries, but they don’t include Azure since Azure packages aren’t available on ARM. Azure is enabled as a compile-time option only for x86, so it’s included in the official x86 releases.

Hi @CalvinKirs ,

We are currently running Doris 3.1.3 in disaggregated mode with Azure Blob as Vault. It is running perfectly fine, but when we tried to run Doris 4.0.1 with the same it complained it is not supported and @yiguolei mentioned it is a bug and should be solved in Doris 4.0.3

Also, we run Doris in Kubernetes so we rely on docker images published to docker hub. And yes we used x86 arch because arm is not supported for foundation DB 7.1.x which Doris expects.

Kindly help us in unblocking this in 4.0.3

@Nj-kol The 4.0.3 version works fine. The 4.0.1 one indeed doesn’t include the Azure package.

[Nj-kol]

@yiguolei Could you please enable Azure vault for compute storage separation in this release? We are stuck because of this
Ref - #58605 (comment)

@Nj-kol Are you using ARM or x64, and is this the official build? We have ARM binaries, but they don’t include Azure since Azure packages aren’t available on ARM. Azure is enabled as a compile-time option only for x86, so it’s included in the official x86 releases.

Hi @CalvinKirs ,
We are currently running Doris 3.1.3 in disaggregated mode with Azure Blob as Vault. It is running perfectly fine, but when we tried to run Doris 4.0.1 with the same it complained it is not supported and @yiguolei mentioned it is a bug and should be solved in Doris 4.0.3

Also, we run Doris in Kubernetes so we rely on docker images published to docker hub. And yes we used x86 arch because arm is not supported for foundation DB 7.1.x which Doris expects.
Kindly help us in unblocking this in 4.0.3

@Nj-kol The 4.0.3 version works fine. The 4.0.1 one indeed doesn’t include the Azure package.

@CalvinKirs I see, so you are confirming that when 4.0.3 will be released it will have export BUILD_AZURE=ON by default.

[CalvinKirs]

export
@Nj-kol

Yeah, I’d recommend just using our x64 binary package for version 4.0.3.

[Nj-kol]

@CalvinKirs ok, and when are you guys planning to publish the 4.0.3 images to dockerhub? Because we run on k8s

[hcy99999]

Why hasn't the new version of docker been released yet?

[CalvinKirs]

@hcy99999 @Nj-kol It’s already up on Docker Hub. Have fun trying it out! https://hub.docker.com/r/apache/doris