[Feature](partial update) Support flexible partial update in stream load with json files

[bobhan1]

This PR add the ability to update different columns for each row in one stream load
Doc: apache/doris-website#1140

Example

MySQL root@127.1:d1> CREATE TABLE t1 (
                  -> `k` int(11) NULL, 
                  -> `v1` BIGINT NULL,
                  -> `v2` BIGINT NULL DEFAULT "9876",
                  -> `v3` BIGINT NOT NULL,
                  -> `v4` BIGINT NOT NULL DEFAULT "1234",
                  -> `v5` BIGINT NULL
                  -> ) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  -> PROPERTIES(
                  -> "replication_num" = "1",
                  -> "enable_unique_key_merge_on_write" = "true"); 
Query OK, 0 rows affected
Time: 0.013s
MySQL root@127.1:d1> insert into t1 select number, number, number, number, number, number from numbers("number" = "6");
Query OK, 6 rows affected
Time: 0.107s
MySQL root@127.1:d1> select * from t1;
+---+----+----+----+----+----+
| k | v1 | v2 | v3 | v4 | v5 |
+---+----+----+----+----+----+
| 0 | 0  | 0  | 0  | 0  | 0  |
| 1 | 1  | 1  | 1  | 1  | 1  |
| 2 | 2  | 2  | 2  | 2  | 2  |
| 3 | 3  | 3  | 3  | 3  | 3  |
| 4 | 4  | 4  | 4  | 4  | 4  |
| 5 | 5  | 5  | 5  | 5  | 5  |
+---+----+----+----+----+----+

test1.json:

{"k": 1, "v1": 10}
{"k": 2, "v2": 20, "v5": 25}
{"k": 3, "v3": 30}
{"k": 4, "v4": 20, "v1": 43, "v3": 99}
{"k": 5, "v5": null}
{"k": 6, "v1": 999, "v3": 777}
{"k": 2, "v4": 222}
{"k": 1, "v2": 111, "v3": 111}

curl --location-trusted -u root: \
-H "strict_mode:false" \
-H "format:json" \
-H "read_json_by_line:true" \
-H "unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS" \
-T test1.json \
-XPUT http://<host>:<http_port>/api/d1/t1/_stream_load

MySQL root@127.1:d1> select * from t1;
+---+-----+------+-----+------+--------+
| k | v1  | v2   | v3  | v4   | v5     |
+---+-----+------+-----+------+--------+
| 0 | 0   | 0    | 0   | 0    | 0      |
| 1 | 10  | 111  | 111 | 1    | 1      |
| 2 | 2   | 20   | 2   | 222  | 25     |
| 3 | 3   | 3    | 30  | 3    | 3      |
| 4 | 43  | 4    | 99  | 20   | 4      |
| 5 | 5   | 5    | 5   | 5    | <null> |
| 6 | 999 | 9876 | 777 | 1234 | <null> |
+---+-----+------+-----+------+--------+

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[bobhan1]

run buildall

[github-actions]

clang-tidy made some suggestions

[github-actions]

warning: function '_merge_rows_for_sequence_column' exceeds recommended size/complexity thresholds [readability-function-size]

Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
                              ^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:830: 91 lines including whitespace and comments (threshold 80)

Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
                              ^

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.23% (9638/25888)
Line Coverage: 28.54% (79918/280040)
Region Coverage: 28.01% (41327/147559)
Branch Coverage: 24.62% (21055/85508)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2569de09e81dd504a3267b5e991c552ca678fbb0_2569de09e81dd504a3267b5e991c552ca678fbb0/report/index.html

[bobhan1]

run cloud_p0

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[bobhan1]

run buildall

[bobhan1]

run buildall

[github-actions]

clang-tidy made some suggestions

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.25% (9645/25895)
Line Coverage: 28.56% (79987/280104)
Region Coverage: 27.99% (41350/147725)
Branch Coverage: 24.60% (21047/85574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e5b703dbd9f6361ac4bc3456066488cc78091f2f_e5b703dbd9f6361ac4bc3456066488cc78091f2f/report/index.html

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[zhannngchen]

LGTM

[Carl-Zhou-CN]

@dataroaring @zhannngchen hi, is there any plan to release this PR? What risks are involved

[bobhan1]

@Carl-Zhou-CN the feature will probably be in doris 4.0

[Carl-Zhou-CN]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

[bobhan1]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

This PR has problems and some them are fixed in #41701.
This feature may require high iops and io throughput.

[Carl-Zhou-CN]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

This PR has problems and some them are fixed in #41701. This feature may require high iops and io throughput.

Ok. Thank you.