[Refactor](Variant) refactor flush logic to support partial update by eldenmoon · Pull Request #34925 · apache/doris

eldenmoon · 2024-05-15T12:31:46Z

Proposed changes

Moved variant flatten and flush logic to _append_block_with_variant_subcolumns in segment writer to simplify handling partial updates.
Ensured rowset schema is refreshed during partial updates for tables with variant types, preventing stale schemas and incorrect data reads.
Removed _output_as_raw_json as the segment writer now handles variant flatten and flush logic, which was used for schema change, but t flatten and flush already handled in segment writer now.
Implemented element_at function in BE for non-scalar variants to enhance functionality.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-15T12:31:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

eldenmoon · 2024-05-15T12:32:09Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-15T12:40:08Z

 }

-Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column) {
+Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column,


warning: function '_create_column_writer' has cognitive complexity of 51 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:154: +1, including nesting penalty of 0, nesting level increased to 1

if (tablet_index) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:163: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.rowset_ctx != nullptr) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:168: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.write_type == DataWriteType::TYPE_DIRECT && ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:174: +1, including nesting penalty of 0, nesting level increased to 1

if (!InvertedIndexColumnWriter::check_column_valid(column)) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:213: +1, including nesting penalty of 0, nesting level increased to 1

if (column.is_row_store_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-15T12:40:08Z

 }

-Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column) {
+Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column,


warning: function '_create_column_writer' exceeds recommended size/complexity thresholds [readability-function-size]

Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:142: 81 lines including whitespace and comments (threshold 80)

Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column, ^

github-actions · 2024-05-15T12:40:08Z

 //       2.3 fill block
 // 3. set columns to data convertor and then write all columns
-Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data) {
+Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data,


warning: function '_append_block_with_partial_content' has cognitive complexity of 57 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:351: +1, including nesting penalty of 0, nesting level increased to 1

if (const vectorized::ColumnWithTypeAndName* delete_sign_column = ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:356: +2, including nesting penalty of 1, nesting level increased to 2

if (delete_sign_col.size() >= data.row_pos + data.num_rows) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +1, including nesting penalty of 0, nesting level increased to 1

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:35: expanded from macro 'DBUG_EXECUTE_IF'

if (UNLIKELY(config::enable_debug_points)) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +2, including nesting penalty of 1, nesting level increased to 2

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:37: expanded from macro 'DBUG_EXECUTE_IF'

if (dp) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:367: +1, including nesting penalty of 0, nesting level increased to 1

if (specified_rowsets.size() != _mow_context->rowset_ids.size()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:376: +2, including nesting penalty of 1, nesting level increased to 2

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:387: +1, including nesting penalty of 0, nesting level increased to 1

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:398: +2, including nesting penalty of 1, nesting level increased to 2

if (have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:404: +2, including nesting penalty of 1, nesting level increased to 2

if (!_tablet_schema->has_sequence_col() || have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:410: +1

(delete_sign_column_data != nullptr && delete_sign_column_data[block_pos] != 0); ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:417: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_NOT_FOUND>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:418: +3, including nesting penalty of 2, nesting level increased to 3

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:424: +1, nesting level increased to 3

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:425: +4, including nesting penalty of 3, nesting level increased to 4

if (!_opts.rowset_ctx->partial_update_info->can_insert_new_rows_in_partial_update) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:446: +2, including nesting penalty of 1, nesting level increased to 2

if (!st.ok() && !st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:456: +2, including nesting penalty of 1, nesting level increased to 2

if (have_delete_sign && !_tablet_schema->has_sequence_col()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:459: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:466: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:472: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:479: +1, including nesting penalty of 0, nesting level increased to 1

if (config::enable_merge_on_write_correctness_check) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:489: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->store_row_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:513: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->has_sequence_col() && !have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:520: +2, including nesting penalty of 1, nesting level increased to 2

if (_num_rows_written != data.row_pos || ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:527: +2, including nesting penalty of 1, nesting level increased to 2

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-15T12:40:08Z

 //       2.3 fill block
 // 3. set columns to data convertor and then write all columns
-Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data) {
+Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data,


warning: function '_append_block_with_partial_content' exceeds recommended size/complexity thresholds [readability-function-size]

Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:307: 233 lines including whitespace and comments (threshold 80)

Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data, ^

github-actions · 2024-05-15T12:40:08Z

    return Status::OK();
 }

+Status VerticalSegmentWriter::_append_block_with_variant_subcolumns(RowsInBlock& data) {


warning: function '_append_block_with_variant_subcolumns' exceeds recommended size/complexity thresholds [readability-function-size]

Status VerticalSegmentWriter::_append_block_with_variant_subcolumns(RowsInBlock& data) { ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:709: 98 lines including whitespace and comments (threshold 80)

Status VerticalSegmentWriter::_append_block_with_variant_subcolumns(RowsInBlock& data) { ^

doris-robot · 2024-05-15T13:13:56Z

TPC-H: Total hot run time: 40785 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c44af038a16624d5a019df9953082dcfa9991682, data reload: false

------ Round 1 ----------------------------------
q1	17766	4460	4333	4333
q2	2522	195	194	194
q3	11575	1288	1205	1205
q4	10725	790	834	790
q5	7526	2775	2645	2645
q6	211	130	130	130
q7	1016	576	570	570
q8	9228	2094	2064	2064
q9	8747	6576	6546	6546
q10	8921	3750	3692	3692
q11	457	259	239	239
q12	442	228	218	218
q13	17761	2949	2955	2949
q14	267	215	224	215
q15	496	471	465	465
q16	486	388	383	383
q17	973	736	677	677
q18	8075	7583	7506	7506
q19	5823	1531	1464	1464
q20	657	304	300	300
q21	5082	3920	4115	3920
q22	345	280	287	280
Total cold run time: 119101 ms
Total hot run time: 40785 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4286	4177	4184	4177
q2	378	269	274	269
q3	2999	2710	2725	2710
q4	1867	1633	1571	1571
q5	5306	5287	5285	5285
q6	215	123	124	123
q7	2242	1866	1837	1837
q8	3201	3401	3312	3312
q9	8366	8401	8360	8360
q10	3883	3735	3664	3664
q11	588	484	501	484
q12	760	565	624	565
q13	16574	2941	2950	2941
q14	299	278	265	265
q15	521	461	481	461
q16	467	412	407	407
q17	1786	1464	1467	1464
q18	7762	7769	7479	7479
q19	1642	1529	1549	1529
q20	1979	1773	1755	1755
q21	5030	4958	4986	4958
q22	553	489	480	480
Total cold run time: 70704 ms
Total hot run time: 54096 ms

doris-robot · 2024-05-15T13:25:42Z

TPC-DS: Total hot run time: 187021 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c44af038a16624d5a019df9953082dcfa9991682, data reload: false

query1	911	385	378	378
query2	6453	2358	2166	2166
query3	6659	202	205	202
query4	25110	21209	21285	21209
query5	4151	414	411	411
query6	264	185	168	168
query7	4585	297	292	292
query8	251	186	187	186
query9	8457	2462	2440	2440
query10	445	251	261	251
query11	14975	14164	14308	14164
query12	145	91	88	88
query13	1652	380	374	374
query14	10531	8288	7554	7554
query15	229	169	174	169
query16	8154	265	256	256
query17	1858	543	536	536
query18	2099	271	274	271
query19	191	143	148	143
query20	90	86	88	86
query21	193	131	128	128
query22	5092	4778	4840	4778
query23	34093	33468	33707	33468
query24	11750	2967	2895	2895
query25	657	369	367	367
query26	1747	164	157	157
query27	2925	324	335	324
query28	7456	2091	2065	2065
query29	994	620	599	599
query30	319	172	180	172
query31	1043	755	741	741
query32	93	52	53	52
query33	744	244	245	244
query34	1070	485	474	474
query35	804	670	667	667
query36	1103	899	949	899
query37	135	71	69	69
query38	2918	2777	2747	2747
query39	1605	1548	1555	1548
query40	269	131	123	123
query41	46	47	42	42
query42	102	94	95	94
query43	582	530	530	530
query44	1253	723	728	723
query45	270	254	248	248
query46	1069	729	757	729
query47	1951	1870	1867	1867
query48	373	303	306	303
query49	1165	424	401	401
query50	772	390	382	382
query51	6750	6613	6587	6587
query52	102	90	90	90
query53	348	285	276	276
query54	1043	428	427	427
query55	72	71	73	71
query56	236	218	223	218
query57	1240	1134	1139	1134
query58	223	200	208	200
query59	3592	2974	3031	2974
query60	260	230	237	230
query61	92	89	91	89
query62	651	466	471	466
query63	333	275	278	275
query64	9766	7370	7332	7332
query65	3114	3064	3099	3064
query66	1395	337	331	331
query67	15259	15141	15093	15093
query68	4733	532	551	532
query69	531	310	313	310
query70	1176	1153	1084	1084
query71	420	273	281	273
query72	7993	2708	2487	2487
query73	725	331	330	330
query74	6456	6095	6148	6095
query75	3696	2684	2618	2618
query76	3534	1041	1058	1041
query77	659	270	270	270
query78	10579	10114	10171	10114
query79	2725	520	516	516
query80	1330	461	452	452
query81	514	248	251	248
query82	1389	99	101	99
query83	205	176	176	176
query84	267	91	92	91
query85	1523	321	315	315
query86	455	325	300	300
query87	3309	3115	3122	3115
query88	4290	2448	2445	2445
query89	479	377	378	377
query90	2004	194	193	193
query91	135	109	107	107
query92	64	50	50	50
query93	3575	510	496	496
query94	1165	189	200	189
query95	410	305	317	305
query96	598	268	273	268
query97	3178	2955	2983	2955
query98	248	218	215	215
query99	1326	919	918	918
Total cold run time: 293970 ms
Total hot run time: 187021 ms

doris-robot · 2024-05-15T13:31:12Z

ClickBench: Total hot run time: 31.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c44af038a16624d5a019df9953082dcfa9991682, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.05	0.04
query3	0.24	0.04	0.05
query4	1.68	0.09	0.09
query5	0.49	0.48	0.52
query6	1.11	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.55	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.79	0.76	0.78
query15	0.81	0.82	0.81
query16	0.37	0.36	0.37
query17	1.03	0.95	1.02
query18	0.23	0.24	0.24
query19	1.76	1.76	1.70
query20	0.01	0.01	0.01
query21	15.71	0.66	0.64
query22	4.23	7.07	2.43
query23	18.32	1.33	1.28
query24	1.57	0.33	0.24
query25	0.14	0.09	0.07
query26	0.26	0.16	0.17
query27	0.07	0.08	0.07
query28	13.24	1.02	1.01
query29	13.07	3.33	3.33
query30	0.25	0.06	0.05
query31	2.88	0.39	0.37
query32	3.28	0.46	0.46
query33	2.82	2.83	2.82
query34	17.17	4.44	4.44
query35	4.51	4.50	4.51
query36	0.65	0.45	0.47
query37	0.17	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.58 s
Total hot run time: 31.03 s

doris-robot · 2024-05-15T13:31:51Z

TeamCity be ut coverage result:
Function Coverage: 35.71% (9001/25208)
Line Coverage: 27.37% (74468/272074)
Region Coverage: 26.61% (38493/144672)
Branch Coverage: 23.43% (19640/83836)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c44af038a16624d5a019df9953082dcfa9991682_c44af038a16624d5a019df9953082dcfa9991682/report/index.html

github-actions

clang-tidy made some suggestions

eldenmoon · 2024-05-16T04:30:59Z

run buildall

doris-robot · 2024-05-16T05:29:39Z

TeamCity be ut coverage result:
Function Coverage: 35.70% (9001/25215)
Line Coverage: 27.36% (74476/272169)
Region Coverage: 26.61% (38505/144704)
Branch Coverage: 23.44% (19647/83836)
Coverage Report: http://coverage.selectdb-in.cc/coverage/78b2c79c10b9ba4c07f1a00863e8b659856cbc91_78b2c79c10b9ba4c07f1a00863e8b659856cbc91/report/index.html

eldenmoon · 2024-05-17T06:48:34Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-17T06:55:59Z

-
-Status parse_variant_columns(Block& block, const std::vector<int>& variant_pos,
-                             const ParseContext& ctx) {
+Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos,


warning: function '_parse_variant_columns' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]

Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos, ^

Additional context

be/src/vec/common/schema_util.cpp:481: +1, including nesting penalty of 0, nesting level increased to 1

for (int i = 0; i < variant_pos.size(); ++i) { ^

be/src/vec/common/schema_util.cpp:491: nesting level increased to 2

auto encode_rowstore = [&]() { ^

be/src/vec/common/schema_util.cpp:492: +3, including nesting penalty of 2, nesting level increased to 3

if (!ctx.record_raw_json_column) { ^

be/src/vec/common/schema_util.cpp:496: +3, including nesting penalty of 2, nesting level increased to 3

if (record_raw_string_with_serialization) { ^

be/src/vec/common/schema_util.cpp:499: +4, including nesting penalty of 3, nesting level increased to 4

for (size_t i = 0; i < var->rows(); ++i) { ^

be/src/vec/common/schema_util.cpp:501: +5, including nesting penalty of 4, nesting level increased to 5

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:501: +6, including nesting penalty of 5, nesting level increased to 6

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:505: +1, nesting level increased to 3

} else { ^

be/src/vec/common/schema_util.cpp:515: +2, including nesting penalty of 1, nesting level increased to 2

if (!var.is_scalar_variant()) { ^

be/src/vec/common/schema_util.cpp:518: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:518: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:523: +2, including nesting penalty of 1, nesting level increased to 2

if (WhichDataType(remove_nullable(var.get_root_type())).is_json()) { ^

be/src/vec/common/schema_util.cpp:526: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:528: +4, including nesting penalty of 3, nesting level increased to 4

? make_nullable(std::make_shared<DataTypeString>()) ^

be/src/vec/common/schema_util.cpp:526: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:531: +3, including nesting penalty of 2, nesting level increased to 3

if (raw_json_column->is_nullable()) { ^

be/src/vec/common/schema_util.cpp:535: +1, nesting level increased to 2

} else { ^

be/src/vec/common/schema_util.cpp:539: +3, including nesting penalty of 2, nesting level increased to 3

? assert_cast<const ColumnNullable&>(root).get_nested_column_ptr() ^

be/src/vec/common/schema_util.cpp:549: +2, including nesting penalty of 1, nesting level increased to 2

if (is_nullable) { ^

be/src/vec/common/schema_util.cpp:555: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:570: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:555: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:572: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

doris-robot · 2024-05-17T08:18:32Z

TeamCity be ut coverage result:
Function Coverage: 35.70% (9009/25234)
Line Coverage: 27.37% (74547/272374)
Region Coverage: 26.61% (38546/144833)
Branch Coverage: 23.44% (19664/83904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a9c1dcd84b4f407b8a437204e38dcede70b0e4d4_a9c1dcd84b4f407b8a437204e38dcede70b0e4d4/report/index.html

eldenmoon · 2024-05-17T09:44:29Z

run buildall

doris-robot · 2024-05-17T10:44:00Z

TeamCity be ut coverage result:
Function Coverage: 35.71% (9011/25234)
Line Coverage: 27.36% (74522/272354)
Region Coverage: 26.61% (38548/144839)
Branch Coverage: 23.44% (19670/83920)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fbed502ba8191a7e2f548e8624ae4ec33c2b4f5d_fbed502ba8191a7e2f548e8624ae4ec33c2b4f5d/report/index.html

eldenmoon · 2024-05-18T05:06:02Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-18T05:13:12Z

 }

-Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column) {
+Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column,


warning: function '_create_column_writer' has cognitive complexity of 51 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:154: +1, including nesting penalty of 0, nesting level increased to 1

if (tablet_index) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:163: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.rowset_ctx != nullptr) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:168: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.write_type == DataWriteType::TYPE_DIRECT && ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:174: +1, including nesting penalty of 0, nesting level increased to 1

if (!InvertedIndexColumnWriter::check_column_valid(column)) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:213: +1, including nesting penalty of 0, nesting level increased to 1

if (column.is_row_store_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-18T05:13:12Z

 //       2.3 fill block
 // 3. set columns to data convertor and then write all columns
-Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data) {
+Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data,


warning: function '_append_block_with_partial_content' has cognitive complexity of 57 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:351: +1, including nesting penalty of 0, nesting level increased to 1

if (const vectorized::ColumnWithTypeAndName* delete_sign_column = ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:356: +2, including nesting penalty of 1, nesting level increased to 2

if (delete_sign_col.size() >= data.row_pos + data.num_rows) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +1, including nesting penalty of 0, nesting level increased to 1

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:35: expanded from macro 'DBUG_EXECUTE_IF'

if (UNLIKELY(config::enable_debug_points)) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +2, including nesting penalty of 1, nesting level increased to 2

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:37: expanded from macro 'DBUG_EXECUTE_IF'

if (dp) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:367: +1, including nesting penalty of 0, nesting level increased to 1

if (specified_rowsets.size() != _mow_context->rowset_ids.size()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:376: +2, including nesting penalty of 1, nesting level increased to 2

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:387: +1, including nesting penalty of 0, nesting level increased to 1

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:398: +2, including nesting penalty of 1, nesting level increased to 2

if (have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:404: +2, including nesting penalty of 1, nesting level increased to 2

if (!_tablet_schema->has_sequence_col() || have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:410: +1

(delete_sign_column_data != nullptr && delete_sign_column_data[block_pos] != 0); ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:417: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_NOT_FOUND>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:418: +3, including nesting penalty of 2, nesting level increased to 3

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:424: +1, nesting level increased to 3

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:425: +4, including nesting penalty of 3, nesting level increased to 4

if (!_opts.rowset_ctx->partial_update_info->can_insert_new_rows_in_partial_update) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:446: +2, including nesting penalty of 1, nesting level increased to 2

if (!st.ok() && !st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:456: +2, including nesting penalty of 1, nesting level increased to 2

if (have_delete_sign && !_tablet_schema->has_sequence_col()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:459: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:466: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:472: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:479: +1, including nesting penalty of 0, nesting level increased to 1

if (config::enable_merge_on_write_correctness_check) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:489: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->store_row_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:513: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->has_sequence_col() && !have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:520: +2, including nesting penalty of 1, nesting level increased to 2

if (_num_rows_written != data.row_pos || ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:527: +2, including nesting penalty of 1, nesting level increased to 2

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-18T05:13:13Z

-
-Status parse_variant_columns(Block& block, const std::vector<int>& variant_pos,
-                             const ParseContext& ctx) {
+Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos,


warning: function '_parse_variant_columns' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]

Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos, ^

Additional context

be/src/vec/common/schema_util.cpp:481: +1, including nesting penalty of 0, nesting level increased to 1

for (int i = 0; i < variant_pos.size(); ++i) { ^

be/src/vec/common/schema_util.cpp:491: nesting level increased to 2

auto encode_rowstore = [&]() { ^

be/src/vec/common/schema_util.cpp:492: +3, including nesting penalty of 2, nesting level increased to 3

if (!ctx.record_raw_json_column) { ^

be/src/vec/common/schema_util.cpp:496: +3, including nesting penalty of 2, nesting level increased to 3

if (record_raw_string_with_serialization) { ^

be/src/vec/common/schema_util.cpp:499: +4, including nesting penalty of 3, nesting level increased to 4

for (size_t i = 0; i < var->rows(); ++i) { ^

be/src/vec/common/schema_util.cpp:501: +5, including nesting penalty of 4, nesting level increased to 5

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:501: +6, including nesting penalty of 5, nesting level increased to 6

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:505: +1, nesting level increased to 3

} else { ^

be/src/vec/common/schema_util.cpp:515: +2, including nesting penalty of 1, nesting level increased to 2

if (!var.is_scalar_variant()) { ^

be/src/vec/common/schema_util.cpp:518: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:518: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:523: +2, including nesting penalty of 1, nesting level increased to 2

if (WhichDataType(remove_nullable(var.get_root_type())).is_json()) { ^

be/src/vec/common/schema_util.cpp:526: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:528: +4, including nesting penalty of 3, nesting level increased to 4

? make_nullable(std::make_shared<DataTypeString>()) ^

be/src/vec/common/schema_util.cpp:526: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:531: +3, including nesting penalty of 2, nesting level increased to 3

if (raw_json_column->is_nullable()) { ^

be/src/vec/common/schema_util.cpp:535: +1, nesting level increased to 2

} else { ^

be/src/vec/common/schema_util.cpp:539: +3, including nesting penalty of 2, nesting level increased to 3

? assert_cast<const ColumnNullable&>(root).get_nested_column_ptr() ^

be/src/vec/common/schema_util.cpp:549: +2, including nesting penalty of 1, nesting level increased to 2

if (is_nullable) { ^

be/src/vec/common/schema_util.cpp:555: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:555: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

doris-robot · 2024-05-18T06:00:00Z

TeamCity be ut coverage result:
Function Coverage: 35.71% (9011/25236)
Line Coverage: 27.36% (74518/272363)
Region Coverage: 26.61% (38546/144850)
Branch Coverage: 23.44% (19666/83914)
Coverage Report: http://coverage.selectdb-in.cc/coverage/24269d30a973fb57d2fc47adfb7de1b5dd74154d_24269d30a973fb57d2fc47adfb7de1b5dd74154d/report/index.html

eldenmoon · 2024-05-21T02:00:50Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-21T02:07:56Z

 }

-Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column) {
+Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column,


warning: function '_create_column_writer' has cognitive complexity of 51 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletColumn& column, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:154: +1, including nesting penalty of 0, nesting level increased to 1

if (tablet_index) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:163: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.rowset_ctx != nullptr) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:168: +1, including nesting penalty of 0, nesting level increased to 1

if (_opts.write_type == DataWriteType::TYPE_DIRECT && ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:174: +1, including nesting penalty of 0, nesting level increased to 1

if (!InvertedIndexColumnWriter::check_column_valid(column)) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:202: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(STRUCT, "struct") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:203: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(ARRAY, "array") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:204: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(JSONB, "jsonb") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:205: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(AGG_STATE, "agg_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:206: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(MAP, "map") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:207: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(OBJECT, "object") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:208: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(HLL, "hll") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +1, including nesting penalty of 0, nesting level increased to 1

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:192: expanded from macro 'CHECK_FIELD_TYPE'

if (column.type() == FieldType::OLAP_FIELD_TYPE_##TYPE) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:194: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bloom_filter) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:209: +2, including nesting penalty of 1, nesting level increased to 2

CHECK_FIELD_TYPE(QUANTILE_STATE, "quantile_state") ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:197: expanded from macro 'CHECK_FIELD_TYPE'

if (opts.need_bitmap_index) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:213: +1, including nesting penalty of 0, nesting level increased to 1

if (column.is_row_store_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:218: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(ColumnWriter::create(opts, &column, _file_writer, &writer)); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:219: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(writer->init()); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-21T02:07:56Z

 //       2.3 fill block
 // 3. set columns to data convertor and then write all columns
-Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data) {
+Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data,


warning: function '_append_block_with_partial_content' has cognitive complexity of 57 (threshold 50) [readability-function-cognitive-complexity]

Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& data, ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:351: +1, including nesting penalty of 0, nesting level increased to 1

if (const vectorized::ColumnWithTypeAndName* delete_sign_column = ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:356: +2, including nesting penalty of 1, nesting level increased to 2

if (delete_sign_col.size() >= data.row_pos + data.num_rows) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +1, including nesting penalty of 0, nesting level increased to 1

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:35: expanded from macro 'DBUG_EXECUTE_IF'

if (UNLIKELY(config::enable_debug_points)) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:363: +2, including nesting penalty of 1, nesting level increased to 2

DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_partial_content.sleep", ^

be/src/util/debug_points.h:37: expanded from macro 'DBUG_EXECUTE_IF'

if (dp) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:367: +1, including nesting penalty of 0, nesting level increased to 1

if (specified_rowsets.size() != _mow_context->rowset_ids.size()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:376: +2, including nesting penalty of 1, nesting level increased to 2

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:387: +1, including nesting penalty of 0, nesting level increased to 1

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:398: +2, including nesting penalty of 1, nesting level increased to 2

if (have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:404: +2, including nesting penalty of 1, nesting level increased to 2

if (!_tablet_schema->has_sequence_col() || have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:405: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:410: +1

(delete_sign_column_data != nullptr && delete_sign_column_data[block_pos] != 0); ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:417: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_NOT_FOUND>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:418: +3, including nesting penalty of 2, nesting level increased to 3

if (_opts.rowset_ctx->partial_update_info->is_strict_mode) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:424: +1, nesting level increased to 3

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:425: +4, including nesting penalty of 3, nesting level increased to 4

if (!_opts.rowset_ctx->partial_update_info->can_insert_new_rows_in_partial_update) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:446: +2, including nesting penalty of 1, nesting level increased to 2

if (!st.ok() && !st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:456: +2, including nesting penalty of 1, nesting level increased to 2

if (have_delete_sign && !_tablet_schema->has_sequence_col()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:459: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:466: +2, including nesting penalty of 1, nesting level increased to 2

if (st.is<KEY_ALREADY_EXISTS>()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:472: +1, nesting level increased to 2

} else { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:479: +1, including nesting penalty of 0, nesting level increased to 1

if (config::enable_merge_on_write_correctness_check) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:486: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(_fill_missing_columns(mutable_full_columns, use_default_or_null_flag, ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:489: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->store_row_column()) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:513: +1, including nesting penalty of 0, nesting level increased to 1

if (_tablet_schema->has_sequence_col() && !have_input_seq_column) { ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:520: +2, including nesting penalty of 1, nesting level increased to 2

if (_num_rows_written != data.row_pos || ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:527: +2, including nesting penalty of 1, nesting level increased to 2

for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:531: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_primary_key_index_builder->add_item(key)); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

github-actions · 2024-05-21T02:07:56Z

-
-Status parse_variant_columns(Block& block, const std::vector<int>& variant_pos,
-                             const ParseContext& ctx) {
+Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos,


warning: function '_parse_variant_columns' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]

Status _parse_variant_columns(Block& block, const std::vector<int>& variant_pos, ^

Additional context

be/src/vec/common/schema_util.cpp:485: +1, including nesting penalty of 0, nesting level increased to 1

for (int i = 0; i < variant_pos.size(); ++i) { ^

be/src/vec/common/schema_util.cpp:495: nesting level increased to 2

auto encode_rowstore = [&]() { ^

be/src/vec/common/schema_util.cpp:496: +3, including nesting penalty of 2, nesting level increased to 3

if (!ctx.record_raw_json_column) { ^

be/src/vec/common/schema_util.cpp:500: +3, including nesting penalty of 2, nesting level increased to 3

if (record_raw_string_with_serialization) { ^

be/src/vec/common/schema_util.cpp:503: +4, including nesting penalty of 3, nesting level increased to 4

for (size_t i = 0; i < var->rows(); ++i) { ^

be/src/vec/common/schema_util.cpp:505: +5, including nesting penalty of 4, nesting level increased to 5

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:505: +6, including nesting penalty of 5, nesting level increased to 6

RETURN_IF_ERROR(var->serialize_one_row_to_string(i, &raw_str)); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:509: +1, nesting level increased to 3

} else { ^

be/src/vec/common/schema_util.cpp:519: +2, including nesting penalty of 1, nesting level increased to 2

if (!var.is_scalar_variant()) { ^

be/src/vec/common/schema_util.cpp:522: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:522: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:527: +2, including nesting penalty of 1, nesting level increased to 2

if (WhichDataType(remove_nullable(var.get_root_type())).is_json()) { ^

be/src/vec/common/schema_util.cpp:530: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:532: +4, including nesting penalty of 3, nesting level increased to 4

? make_nullable(std::make_shared<DataTypeString>()) ^

be/src/vec/common/schema_util.cpp:530: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(cast_column({var.get_root(), var.get_root_type(), ""}, ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/common/schema_util.cpp:535: +3, including nesting penalty of 2, nesting level increased to 3

if (raw_json_column->is_nullable()) { ^

be/src/vec/common/schema_util.cpp:539: +1, nesting level increased to 2

} else { ^

be/src/vec/common/schema_util.cpp:543: +3, including nesting penalty of 2, nesting level increased to 3

? assert_cast<const ColumnNullable&>(root).get_nested_column_ptr() ^

be/src/vec/common/schema_util.cpp:553: +2, including nesting penalty of 1, nesting level increased to 2

if (is_nullable) { ^

be/src/vec/common/schema_util.cpp:559: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/common/schema_util.cpp:559: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(encode_rowstore()); ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

doris-robot · 2024-05-21T03:36:59Z

TeamCity be ut coverage result:
Function Coverage: 35.69% (9016/25259)
Line Coverage: 27.34% (74551/272703)
Region Coverage: 26.59% (38564/145010)
Branch Coverage: 23.42% (19673/83992)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b3ddff42cdf65989f3f6d2e4a08c923334704b0_3b3ddff42cdf65989f3f6d2e4a08c923334704b0/report/index.html

doris-robot · 2024-06-11T07:27:28Z

TeamCity be ut coverage result:
Function Coverage: 36.44% (8979/24643)
Line Coverage: 27.95% (73336/262418)
Region Coverage: 27.37% (38050/139030)
Branch Coverage: 24.00% (19320/80516)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ee22526610ce2da836e51b2dda51babfd4c5a01_4ee22526610ce2da836e51b2dda51babfd4c5a01/report/index.html

doris-robot · 2024-06-11T07:36:38Z

TPC-DS: Total hot run time: 173010 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ee22526610ce2da836e51b2dda51babfd4c5a01, data reload: false

query1	916	375	378	375
query2	6380	2389	2317	2317
query3	6631	211	215	211
query4	18865	17358	17102	17102
query5	3677	502	492	492
query6	240	159	157	157
query7	4585	295	303	295
query8	310	304	310	304
query9	8414	2378	2363	2363
query10	594	311	313	311
query11	10532	9940	10061	9940
query12	123	85	89	85
query13	1642	370	365	365
query14	9512	7660	7189	7189
query15	236	189	188	188
query16	7774	264	262	262
query17	1804	555	525	525
query18	1908	273	275	273
query19	197	153	171	153
query20	93	85	88	85
query21	208	128	130	128
query22	4278	4097	3985	3985
query23	33523	33937	33452	33452
query24	11300	2846	2848	2846
query25	621	377	366	366
query26	725	155	159	155
query27	2307	344	338	338
query28	6613	2111	2118	2111
query29	887	657	664	657
query30	255	156	151	151
query31	985	770	746	746
query32	98	53	53	53
query33	751	287	318	287
query34	1006	481	488	481
query35	750	646	654	646
query36	1138	978	995	978
query37	150	74	77	74
query38	2930	2822	2743	2743
query39	896	862	859	859
query40	220	139	144	139
query41	63	62	83	62
query42	124	95	99	95
query43	589	570	576	570
query44	1246	739	756	739
query45	195	161	165	161
query46	1073	778	715	715
query47	1815	1728	1747	1728
query48	391	298	294	294
query49	838	413	416	413
query50	782	395	398	395
query51	6690	6626	6726	6626
query52	102	96	91	91
query53	366	298	299	298
query54	862	471	454	454
query55	77	73	73	73
query56	287	266	257	257
query57	1101	1072	1030	1030
query58	255	248	263	248
query59	3383	3214	3071	3071
query60	298	272	282	272
query61	148	92	88	88
query62	599	434	457	434
query63	333	295	301	295
query64	8907	2212	1717	1717
query65	3240	3104	3078	3078
query66	750	323	340	323
query67	15159	14846	14757	14757
query68	4845	549	546	546
query69	619	459	405	405
query70	1214	1162	1171	1162
query71	504	281	278	278
query72	7638	5160	5332	5160
query73	772	327	329	327
query74	5847	5475	5375	5375
query75	3447	2696	2681	2681
query76	2945	967	901	901
query77	636	300	308	300
query78	10313	9807	9742	9742
query79	2036	534	518	518
query80	2199	484	462	462
query81	560	221	224	221
query82	887	106	104	104
query83	287	174	172	172
query84	264	90	82	82
query85	1261	278	259	259
query86	454	317	331	317
query87	3218	3061	3029	3029
query88	3691	2444	2429	2429
query89	459	401	390	390
query90	1726	200	196	196
query91	133	108	171	108
query92	60	50	50	50
query93	1874	509	507	507
query94	1024	189	183	183
query95	407	315	317	315
query96	593	274	271	271
query97	3198	3036	3053	3036
query98	222	198	200	198
query99	1117	815	821	815
Total cold run time: 268197 ms
Total hot run time: 173010 ms

doris-robot · 2024-06-11T07:42:09Z

ClickBench: Total hot run time: 31.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ee22526610ce2da836e51b2dda51babfd4c5a01, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.46
query6	1.14	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.55	0.55	0.53
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.59	0.60	0.59
query14	0.79	0.78	0.79
query15	0.85	0.81	0.80
query16	0.37	0.36	0.38
query17	0.95	1.00	1.05
query18	0.23	0.24	0.23
query19	1.78	1.81	1.81
query20	0.02	0.01	0.02
query21	15.41	0.66	0.65
query22	3.86	6.84	2.65
query23	18.29	1.50	1.34
query24	2.11	0.21	0.22
query25	0.16	0.09	0.08
query26	0.26	0.17	0.16
query27	0.09	0.09	0.08
query28	13.28	1.04	1.00
query29	12.56	3.26	3.22
query30	0.26	0.08	0.06
query31	2.84	0.39	0.39
query32	3.29	0.48	0.47
query33	2.84	2.91	2.92
query34	17.32	4.65	4.60
query35	4.73	4.67	4.48
query36	0.65	0.46	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.03
query40	0.15	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.43 s
Total hot run time: 31.61 s

github-actions · 2024-06-11T08:15:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-11T08:15:39Z

PR approved by anyone and no changes requested.

``` Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnObject Check failure stack trace: *** @ 0x562dcebca976 google::LogMessage::SendToLog() @ 0x562dcebc73c0 google::LogMessage::Flush() @ 0x562dcebcb1b9 google::LogMessageFatal::~LogMessageFatal() @ 0x562d9ded39e6 assert_cast<>() @ 0x562d9df1e599 doris::segment_v2::HierarchicalDataReader::process_read<>() @ 0x562d9df1e106 doris::segment_v2::HierarchicalDataReader::next_batch() @ 0x562d9df3373e doris::segment_v2::ColumnIterator::next_batch() .... @ 0x562d9ea56d31 doris::EngineChecksumTask::_compute_checksum() @ 0x562d9ea55cc2 doris::EngineChecksumTask::execute() @ 0x562d9b558355 doris::check_consistency_callback() ``` introduced by apache#34925

) ``` Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnObject Check failure stack trace: *** @ 0x562dcebca976 google::LogMessage::SendToLog() @ 0x562dcebc73c0 google::LogMessage::Flush() @ 0x562dcebcb1b9 google::LogMessageFatal::~LogMessageFatal() @ 0x562d9ded39e6 assert_cast<>() @ 0x562d9df1e599 doris::segment_v2::HierarchicalDataReader::process_read<>() @ 0x562d9df1e106 doris::segment_v2::HierarchicalDataReader::next_batch() @ 0x562d9df3373e doris::segment_v2::ColumnIterator::next_batch() .... @ 0x562d9ea56d31 doris::EngineChecksumTask::_compute_checksum() @ 0x562d9ea55cc2 doris::EngineChecksumTask::execute() @ 0x562d9b558355 doris::check_consistency_callback() ``` introduced by #34925

…34925) 1. Moved variant flatten and flush logic to `_append_block_with_variant_subcolumns` in segment writer to simplify handling partial updates. 2. Ensured rowset schema is refreshed during partial updates for tables with variant types, preventing stale schemas and incorrect data reads. 3. Removed `_output_as_raw_json` as the segment writer now handles variant flatten and flush logic, which was used for schema change, but t flatten and flush already handled in segment writer now. 4. Implemented element_at function in BE for non-scalar variants to enhance functionality.

) ``` Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnObject Check failure stack trace: *** @ 0x562dcebca976 google::LogMessage::SendToLog() @ 0x562dcebc73c0 google::LogMessage::Flush() @ 0x562dcebcb1b9 google::LogMessageFatal::~LogMessageFatal() @ 0x562d9ded39e6 assert_cast<>() @ 0x562d9df1e599 doris::segment_v2::HierarchicalDataReader::process_read<>() @ 0x562d9df1e106 doris::segment_v2::HierarchicalDataReader::next_batch() @ 0x562d9df3373e doris::segment_v2::ColumnIterator::next_batch() .... @ 0x562d9ea56d31 doris::EngineChecksumTask::_compute_checksum() @ 0x562d9ea55cc2 doris::EngineChecksumTask::execute() @ 0x562d9b558355 doris::check_consistency_callback() ``` introduced by #34925

) 1. Variant use serialize_one_row_to_string to string and then parse to jsonb as row store.Since we could not get the original string after #34925 2. Remove redundant code

…pache#34925) 1. Moved variant flatten and flush logic to `_append_block_with_variant_subcolumns` in segment writer to simplify handling partial updates. 2. Ensured rowset schema is refreshed during partial updates for tables with variant types, preventing stale schemas and incorrect data reads. 3. Removed `_output_as_raw_json` as the segment writer now handles variant flatten and flush logic, which was used for schema change, but t flatten and flush already handled in segment writer now. 4. Implemented element_at function in BE for non-scalar variants to enhance functionality.

…che#36201) ``` Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnObject Check failure stack trace: *** @ 0x562dcebca976 google::LogMessage::SendToLog() @ 0x562dcebc73c0 google::LogMessage::Flush() @ 0x562dcebcb1b9 google::LogMessageFatal::~LogMessageFatal() @ 0x562d9ded39e6 assert_cast<>() @ 0x562d9df1e599 doris::segment_v2::HierarchicalDataReader::process_read<>() @ 0x562d9df1e106 doris::segment_v2::HierarchicalDataReader::next_batch() @ 0x562d9df3373e doris::segment_v2::ColumnIterator::next_batch() .... @ 0x562d9ea56d31 doris::EngineChecksumTask::_compute_checksum() @ 0x562d9ea55cc2 doris::EngineChecksumTask::execute() @ 0x562d9b558355 doris::check_consistency_callback() ``` introduced by apache#34925

#37526)

github-actions Bot reviewed May 15, 2024

View reviewed changes

eldenmoon force-pushed the var-rf-seg branch 3 times, most recently from d7fd687 to 826e2c9 Compare May 16, 2024 04:24

github-actions Bot reviewed May 16, 2024

View reviewed changes

Comment thread be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp

Comment thread be/src/vec/common/schema_util.cpp

eldenmoon changed the title ~~refactor to support partial update~~ [Refactor](Variant) refactor flush logic to support partial update May 16, 2024

eldenmoon marked this pull request as ready for review May 16, 2024 06:19

eldenmoon force-pushed the var-rf-seg branch from 78b2c79 to a9c1dcd Compare May 17, 2024 06:48

eldenmoon added the dev/2.1.x label May 17, 2024

github-actions Bot reviewed May 17, 2024

View reviewed changes

eldenmoon force-pushed the var-rf-seg branch from a9c1dcd to fbed502 Compare May 17, 2024 09:44

github-actions Bot reviewed May 18, 2024

View reviewed changes

eldenmoon force-pushed the var-rf-seg branch from 24269d3 to 3b3ddff Compare May 21, 2024 02:00

github-actions Bot reviewed May 21, 2024

View reviewed changes

eldenmoon requested a review from xiaokang June 11, 2024 08:13

qidaye approved these changes Jun 11, 2024

View reviewed changes

github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 11, 2024

xiaokang approved these changes Jun 11, 2024

View reviewed changes

Comment thread be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated

Comment thread be/src/olap/rowset/rowset.h Outdated

Comment thread be/src/olap/base_tablet.cpp Outdated

Comment thread be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp

eldenmoon merged commit 5745274 into apache:master Jun 11, 2024

eldenmoon deleted the var-rf-seg branch June 11, 2024 10:23

eldenmoon mentioned this pull request Jun 12, 2024

[Fix](Variant) fix checksum task read with unmatched column type #36201

Merged

xiaokang added the p0_c label Jun 19, 2024

eldenmoon removed the dev/2.1.x label Jun 21, 2024

eldenmoon mentioned this pull request Jun 25, 2024

[Fix](Variant) fix variant partial update with row store enabled #36793

Merged

eldenmoon mentioned this pull request Jul 9, 2024

[pick](Variant) pick some refactor and fix #37526

Merged

xiaokang pushed a commit that referenced this pull request Jul 11, 2024

[pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (

217eac7

#37526)

wm1581066 added the dev/2.1.5-merged label Jul 11, 2024

yiguolei mentioned this pull request Jul 19, 2024

2.1.5 Release Notes #38111

Closed

1 task

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Conversation

eldenmoon commented May 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

eldenmoon commented May 15, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 15, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 15, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 15, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 15, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 15, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented May 16, 2024

Uh oh!

doris-robot commented May 16, 2024

Uh oh!

eldenmoon commented May 17, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 17, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 17, 2024

Uh oh!

eldenmoon commented May 17, 2024

Uh oh!

doris-robot commented May 17, 2024

Uh oh!

eldenmoon commented May 18, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 18, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 18, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 18, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 18, 2024

Uh oh!

eldenmoon commented May 21, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented May 15, 2024 •

edited

Loading