Skip to content

[fix] (inverted index) fix query errors caused by ignore_above#37679

Merged
airborne12 merged 2 commits intoapache:masterfrom
csun5285:fix_ignore
Jul 12, 2024
Merged

[fix] (inverted index) fix query errors caused by ignore_above#37679
airborne12 merged 2 commits intoapache:masterfrom
csun5285:fix_ignore

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented Jul 11, 2024 •
edited
Loading

Proposed changes

If the written value exceeds ignore_above, it will be written as null.
The queried value exceeds ignore_above means the written value cannot be found.
The query needs to be downgraded to read from the segment file.

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 11, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993
zzzxl1993 approved these changes Jul 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

TPC-H: Total hot run time: 40070 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9d30ee5c3ef9c7768ee5c870179ff28fd18983dd, data reload: false

------ Round 1 ----------------------------------
q1	17632	4703	4275	4275
q2	2007	192	181	181
q3	10463	1177	1136	1136
q4	10185	787	812	787
q5	7556	2701	2663	2663
q6	221	134	137	134
q7	957	597	607	597
q8	9207	2057	2098	2057
q9	8867	6532	6555	6532
q10	8803	3766	3813	3766
q11	461	239	241	239
q12	437	230	231	230
q13	17764	3007	3034	3007
q14	284	230	232	230
q15	520	478	493	478
q16	500	394	375	375
q17	972	700	733	700
q18	8026	7438	7324	7324
q19	2671	1455	1444	1444
q20	687	308	317	308
q21	4979	3279	3436	3279
q22	394	337	328	328
Total cold run time: 113593 ms
Total hot run time: 40070 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4292	4252	4252
q2	374	284	270	270
q3	3024	2780	2905	2780
q4	1926	1712	1696	1696
q5	5611	5631	5567	5567
q6	233	139	136	136
q7	2208	1801	1875	1801
q8	3252	3390	3362	3362
q9	8768	8750	8875	8750
q10	4107	3869	3992	3869
q11	584	502	489	489
q12	796	619	616	616
q13	17226	3182	3171	3171
q14	312	305	288	288
q15	547	509	499	499
q16	510	453	445	445
q17	1793	1519	1524	1519
q18	8119	7902	7843	7843
q19	2743	1571	1634	1571
q20	2165	1899	1855	1855
q21	5184	4993	4909	4909
q22	614	549	564	549
Total cold run time: 74501 ms
Total hot run time: 56237 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

TPC-DS: Total hot run time: 175586 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9d30ee5c3ef9c7768ee5c870179ff28fd18983dd, data reload: false

query1	920	367	367	367
query2	6466	2468	2431	2431
query3	6640	213	222	213
query4	23488	17528	17348	17348
query5	3690	486	492	486
query6	263	167	165	165
query7	4583	312	288	288
query8	314	282	288	282
query9	8623	2471	2457	2457
query10	450	289	284	284
query11	11935	10307	10004	10004
query12	115	91	82	82
query13	1650	379	385	379
query14	10289	7720	7583	7583
query15	243	194	190	190
query16	7414	338	325	325
query17	1747	558	551	551
query18	1619	291	288	288
query19	205	159	161	159
query20	92	85	82	82
query21	204	129	127	127
query22	4324	4109	4024	4024
query23	34048	33750	33851	33750
query24	10810	2894	2934	2894
query25	623	406	406	406
query26	734	155	156	155
query27	2238	280	290	280
query28	6355	2191	2198	2191
query29	920	668	750	668
query30	246	154	155	154
query31	938	775	786	775
query32	100	55	60	55
query33	691	290	292	290
query34	905	503	508	503
query35	673	596	578	578
query36	1120	989	986	986
query37	141	84	87	84
query38	3029	2836	2829	2829
query39	909	844	815	815
query40	208	121	124	121
query41	53	49	59	49
query42	118	108	102	102
query43	594	551	557	551
query44	1109	753	732	732
query45	193	166	167	166
query46	1081	738	715	715
query47	1832	1739	1763	1739
query48	369	309	309	309
query49	829	402	407	402
query50	806	400	398	398
query51	6833	6830	6765	6765
query52	101	97	92	92
query53	364	298	297	297
query54	881	463	485	463
query55	75	75	75	75
query56	298	274	279	274
query57	1186	1041	1071	1041
query58	243	247	247	247
query59	3345	3229	3133	3133
query60	302	285	278	278
query61	99	95	94	94
query62	811	644	668	644
query63	334	302	295	295
query64	9214	2205	1645	1645
query65	3197	3074	3102	3074
query66	752	324	328	324
query67	15549	14924	14781	14781
query68	8361	580	585	580
query69	631	496	416	416
query70	1183	1162	1175	1162
query71	512	285	280	280
query72	8151	5516	5904	5516
query73	1699	332	331	331
query74	5960	5523	5516	5516
query75	4146	2711	2693	2693
query76	4844	942	966	942
query77	706	336	323	323
query78	9653	9655	9139	9139
query79	8547	535	540	535
query80	1542	482	489	482
query81	590	230	217	217
query82	778	135	135	135
query83	226	172	169	169
query84	274	90	87	87
query85	1325	317	295	295
query86	335	311	328	311
query87	3287	3128	3110	3110
query88	4301	2474	2483	2474
query89	535	387	390	387
query90	1818	199	200	199
query91	138	103	104	103
query92	65	50	51	50
query93	6348	522	529	522
query94	764	220	211	211
query95	407	322	323	322
query96	622	275	274	274
query97	3189	3033	3015	3015
query98	229	201	192	192
query99	1682	1274	1264	1264
Total cold run time: 293378 ms
Total hot run time: 175586 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9d30ee5c3ef9c7768ee5c870179ff28fd18983dd, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.64	0.09	0.08
query5	0.48	0.48	0.49
query6	1.13	0.72	0.75
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.52	0.49
query10	0.54	0.55	0.53
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.60	0.58
query14	0.78	0.76	0.83
query15	0.84	0.82	0.82
query16	0.38	0.37	0.35
query17	1.00	1.00	0.98
query18	0.24	0.22	0.22
query19	1.80	1.76	1.71
query20	0.01	0.01	0.01
query21	15.43	0.74	0.65
query22	4.04	7.52	2.18
query23	18.30	1.41	1.20
query24	2.11	0.24	0.23
query25	0.15	0.09	0.08
query26	0.30	0.22	0.22
query27	0.45	0.23	0.23
query28	13.28	1.01	1.01
query29	12.63	3.34	3.31
query30	0.26	0.06	0.06
query31	2.87	0.41	0.39
query32	3.25	0.47	0.47
query33	2.89	2.92	2.94
query34	17.11	4.32	4.36
query35	4.39	4.47	4.39
query36	0.65	0.46	0.47
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 30.86 s

xiaokang
xiaokang approved these changes Jul 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2024
This was referenced Jul 11, 2024
Merged
Merged
@airborne12 airborne12 merged commit fa31aa1 into apache:master Jul 12, 2024
airborne12 pushed a commit that referenced this pull request Jul 12, 2024
@csun5285
[cherry-pick] (branch-2.1) fix query errors caused by ignore_above (#…
4dc933b 
…37685)

## Proposed changes
pick from master #37679
airborne12 pushed a commit that referenced this pull request Jul 12, 2024
@csun5285
[cherry-pick] (branch-2.0) fix query errors caused by ignore_above (#…
dff141f 
…37686)

## Proposed changes

pick from master #37679
@csun5285 csun5285 mentioned this pull request Jul 17, 2024
Closed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@csun5285 @seawinde
[fix] (inverted index) fix query errors caused by ignore_above (apach…
e85fa76 
…e#37679)

## Proposed changes
If the written value exceeds ignore_above, it will be written as null.
The queried value exceeds ignore_above means the written value cannot be
found.
The query needs to be downgraded to read from the segment file.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[fix] (inverted index) fix query errors caused by ignore_above (#37679)
886f9fe 
## Proposed changes
If the written value exceeds ignore_above, it will be written as null.
The queried value exceeds ignore_above means the written value cannot be
found.
The query needs to be downgraded to read from the segment file.
@xiaokang xiaokang added the doing label Jul 18, 2024
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@csun5285
[cherry-pick] (branch-2.0) fix query errors caused by ignore_above (a…
821f367 
…pache#37686)

## Proposed changes

pick from master apache#37679
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @doris-robot @xiaokang @zzzxl1993 @airborne12 @wm1581066