Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](inverted index) Fix for Inaccurate match_phrase_prefix Cache in Query Processing #46310

Merged
merged 1 commit into from
Jan 6, 2025

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Jan 2, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

  1. different values of inverted_index_max_expansions require separate caches.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32807 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb1b92e73d46b56bbea8f7d77a2472588c9cf231, data reload: false

------ Round 1 ----------------------------------
q1	17586	6095	6017	6017
q2	2048	305	176	176
q3	10563	1247	718	718
q4	10228	866	430	430
q5	7928	2177	1978	1978
q6	209	184	152	152
q7	903	754	612	612
q8	9244	1366	1221	1221
q9	5293	4967	4966	4966
q10	6767	2313	1850	1850
q11	474	277	254	254
q12	351	353	218	218
q13	17760	3644	3092	3092
q14	231	234	212	212
q15	561	498	499	498
q16	638	607	609	607
q17	587	844	325	325
q18	7228	6633	6463	6463
q19	1703	967	560	560
q20	307	313	189	189
q21	2829	2249	1964	1964
q22	361	328	305	305
Total cold run time: 103799 ms
Total hot run time: 32807 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6267	6201	6402	6201
q2	238	323	235	235
q3	2301	2682	2313	2313
q4	1426	1876	1384	1384
q5	4367	4776	4969	4776
q6	188	176	143	143
q7	2136	1964	1789	1789
q8	2608	2832	2661	2661
q9	7358	7219	7244	7219
q10	3039	3331	2809	2809
q11	575	520	497	497
q12	657	774	614	614
q13	3492	3860	3289	3289
q14	313	318	280	280
q15	565	517	503	503
q16	664	683	641	641
q17	1226	1726	1257	1257
q18	7777	7375	7241	7241
q19	823	1112	1045	1045
q20	1949	2000	1857	1857
q21	5426	5096	4839	4839
q22	582	606	547	547
Total cold run time: 53977 ms
Total hot run time: 52140 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189805 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb1b92e73d46b56bbea8f7d77a2472588c9cf231, data reload: false

query1	956	374	378	374
query2	6524	2403	2327	2327
query3	6718	227	214	214
query4	33415	23556	23461	23461
query5	4342	620	443	443
query6	273	190	183	183
query7	4610	493	312	312
query8	302	232	223	223
query9	9413	2623	2597	2597
query10	471	336	260	260
query11	17994	15415	15163	15163
query12	159	105	101	101
query13	1695	537	425	425
query14	10512	7365	6969	6969
query15	239	201	183	183
query16	8271	592	434	434
query17	1584	738	572	572
query18	2144	410	302	302
query19	225	191	157	157
query20	119	111	114	111
query21	218	123	116	116
query22	4295	4393	4181	4181
query23	34705	33745	33488	33488
query24	6375	2297	2248	2248
query25	478	432	378	378
query26	1196	277	150	150
query27	2012	475	332	332
query28	5328	2391	2367	2367
query29	753	553	397	397
query30	228	184	153	153
query31	995	923	807	807
query32	89	62	61	61
query33	502	341	310	310
query34	751	831	514	514
query35	792	821	733	733
query36	1027	1053	947	947
query37	119	96	69	69
query38	4276	4321	3997	3997
query39	1598	1463	1451	1451
query40	213	112	97	97
query41	47	44	48	44
query42	114	100	111	100
query43	516	541	492	492
query44	1294	787	801	787
query45	182	168	165	165
query46	872	1032	660	660
query47	1904	1929	1873	1873
query48	374	414	324	324
query49	776	471	381	381
query50	634	647	391	391
query51	7157	7203	7094	7094
query52	101	101	93	93
query53	224	256	182	182
query54	483	484	420	420
query55	82	77	82	77
query56	246	274	248	248
query57	1214	1161	1103	1103
query58	233	219	228	219
query59	3234	3309	2844	2844
query60	270	289	265	265
query61	111	101	125	101
query62	860	794	726	726
query63	228	193	197	193
query64	4591	989	644	644
query65	3243	3196	3274	3196
query66	1065	426	312	312
query67	15994	15886	15512	15512
query68	9509	715	515	515
query69	469	290	264	264
query70	1203	1073	1095	1073
query71	426	291	255	255
query72	6207	3815	3897	3815
query73	669	747	348	348
query74	10236	9032	9020	9020
query75	4564	3176	2664	2664
query76	5457	1218	777	777
query77	1026	367	320	320
query78	10032	10071	9476	9476
query79	2842	785	594	594
query80	692	518	454	454
query81	461	262	223	223
query82	635	151	128	128
query83	198	162	151	151
query84	284	93	73	73
query85	793	374	317	317
query86	345	317	315	315
query87	4380	4377	4432	4377
query88	3225	2176	2130	2130
query89	417	338	297	297
query90	1911	187	191	187
query91	131	138	105	105
query92	65	99	56	56
query93	1185	867	533	533
query94	665	379	297	297
query95	342	258	252	252
query96	487	597	278	278
query97	2914	3008	2822	2822
query98	221	199	198	198
query99	1712	1566	1438	1438
Total cold run time: 295668 ms
Total hot run time: 189805 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb1b92e73d46b56bbea8f7d77a2472588c9cf231, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.59	0.10	0.10
query5	0.42	0.39	0.42
query6	1.16	0.66	0.65
query7	0.03	0.01	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.56	0.57	0.56
query11	0.15	0.11	0.10
query12	0.13	0.11	0.11
query13	0.59	0.61	0.59
query14	2.73	2.73	2.76
query15	0.90	0.83	0.82
query16	0.37	0.36	0.40
query17	1.06	1.08	1.05
query18	0.22	0.20	0.21
query19	2.00	1.77	1.97
query20	0.01	0.01	0.02
query21	15.36	0.96	0.58
query22	0.76	0.86	0.61
query23	15.32	1.47	0.52
query24	2.53	1.81	0.78
query25	0.27	0.16	0.10
query26	0.28	0.14	0.14
query27	0.07	0.05	0.04
query28	13.70	1.55	1.05
query29	12.59	3.85	3.22
query30	0.24	0.09	0.06
query31	2.83	0.59	0.38
query32	3.22	0.54	0.48
query33	3.08	3.02	3.09
query34	16.57	5.15	4.52
query35	4.54	4.52	4.60
query36	0.73	0.48	0.49
query37	0.10	0.06	0.07
query38	0.04	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.47 s
Total hot run time: 30.94 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10129/26039)
Line Coverage: 29.92% (85668/286312)
Region Coverage: 29.03% (43729/150625)
Branch Coverage: 25.58% (22326/87282)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fb1b92e73d46b56bbea8f7d77a2472588c9cf231_fb1b92e73d46b56bbea8f7d77a2472588c9cf231/report/index.html

@zzzxl1993 zzzxl1993 force-pushed the 202401022034 branch 2 times, most recently from 5d5d117 to f5380b3 Compare January 3, 2025 11:18
airborne12
airborne12 previously approved these changes Jan 6, 2025
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 6, 2025
Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by anyone and no changes requested.

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 6, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32727 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f95797077a3ec0e30ec9c7ab5cb1734412f0e3de, data reload: false

------ Round 1 ----------------------------------
q1	17580	6236	6063	6063
q2	2043	308	171	171
q3	10455	1260	756	756
q4	10197	855	438	438
q5	7498	2214	1989	1989
q6	198	176	145	145
q7	913	758	595	595
q8	9228	1339	1142	1142
q9	5290	4902	4998	4902
q10	6759	2289	1899	1899
q11	470	285	259	259
q12	347	364	222	222
q13	17765	3769	3092	3092
q14	257	235	214	214
q15	558	515	500	500
q16	632	629	599	599
q17	564	864	345	345
q18	7017	6350	6522	6350
q19	2553	957	544	544
q20	310	314	187	187
q21	3013	2213	2002	2002
q22	374	340	313	313
Total cold run time: 104021 ms
Total hot run time: 32727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6335	6289	6236	6236
q2	237	319	234	234
q3	2379	2651	2359	2359
q4	1419	1836	1349	1349
q5	4393	4740	4869	4740
q6	186	176	144	144
q7	2107	1911	1812	1812
q8	2637	2799	2781	2781
q9	7322	7186	7315	7186
q10	3103	3355	2889	2889
q11	569	520	501	501
q12	641	723	622	622
q13	3597	3888	3157	3157
q14	271	304	299	299
q15	569	524	503	503
q16	678	710	625	625
q17	1238	1728	1247	1247
q18	7790	7457	7349	7349
q19	847	1161	1071	1071
q20	2046	2004	1913	1913
q21	5785	5386	5050	5050
q22	623	625	577	577
Total cold run time: 54772 ms
Total hot run time: 52644 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196866 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f95797077a3ec0e30ec9c7ab5cb1734412f0e3de, data reload: false

query1	1276	934	945	934
query2	6392	2397	2353	2353
query3	11119	4880	4919	4880
query4	33237	23620	23410	23410
query5	4598	649	443	443
query6	308	212	191	191
query7	3990	490	310	310
query8	292	239	225	225
query9	9490	2640	2618	2618
query10	478	307	246	246
query11	17735	15692	15098	15098
query12	162	110	108	108
query13	1603	517	394	394
query14	10092	6935	7557	6935
query15	257	204	195	195
query16	7930	645	485	485
query17	1529	789	612	612
query18	2094	421	324	324
query19	203	191	176	176
query20	133	124	120	120
query21	210	136	117	117
query22	4688	4642	4602	4602
query23	34492	33814	33631	33631
query24	6309	2329	2284	2284
query25	512	544	397	397
query26	792	270	151	151
query27	2172	440	334	334
query28	5846	2456	2446	2446
query29	646	552	422	422
query30	210	184	152	152
query31	972	900	858	858
query32	83	58	59	58
query33	483	354	306	306
query34	763	852	511	511
query35	854	835	768	768
query36	1014	1075	1000	1000
query37	126	105	79	79
query38	4275	4408	4369	4369
query39	1522	1455	1481	1455
query40	211	114	100	100
query41	51	45	43	43
query42	135	106	102	102
query43	537	541	522	522
query44	1313	816	818	816
query45	185	179	172	172
query46	884	1073	655	655
query47	2004	1978	1909	1909
query48	398	417	321	321
query49	709	492	416	416
query50	640	655	407	407
query51	7342	7216	7140	7140
query52	109	99	95	95
query53	226	267	188	188
query54	480	510	416	416
query55	81	89	83	83
query56	264	259	251	251
query57	1260	1252	1162	1162
query58	243	225	233	225
query59	3211	3398	3111	3111
query60	279	286	261	261
query61	109	113	107	107
query62	871	840	761	761
query63	241	196	231	196
query64	3513	1018	659	659
query65	3377	3233	3236	3233
query66	824	405	321	321
query67	16318	15920	15429	15429
query68	8130	704	500	500
query69	481	295	255	255
query70	1212	1172	1161	1161
query71	446	305	258	258
query72	6575	3859	3928	3859
query73	656	757	360	360
query74	10192	9047	8917	8917
query75	3992	3149	2697	2697
query76	3708	1205	772	772
query77	763	384	280	280
query78	10215	10057	9469	9469
query79	3441	817	580	580
query80	671	537	425	425
query81	503	272	233	233
query82	714	155	131	131
query83	157	161	142	142
query84	236	90	72	72
query85	758	355	313	313
query86	390	321	308	308
query87	4411	4404	4337	4337
query88	4880	2175	2170	2170
query89	425	334	301	301
query90	1834	188	187	187
query91	137	139	106	106
query92	65	58	56	56
query93	1919	822	519	519
query94	664	395	297	297
query95	344	282	258	258
query96	476	604	283	283
query97	2929	3010	2877	2877
query98	221	206	204	204
query99	1660	1595	1441	1441
Total cold run time: 297907 ms
Total hot run time: 196866 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10129/26048)
Line Coverage: 29.92% (85691/286397)
Region Coverage: 29.02% (43729/150665)
Branch Coverage: 25.57% (22321/87308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f95797077a3ec0e30ec9c7ab5cb1734412f0e3de_f95797077a3ec0e30ec9c7ab5cb1734412f0e3de/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f95797077a3ec0e30ec9c7ab5cb1734412f0e3de, data reload: false

query1	0.03	0.03	0.06
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.41	0.42
query6	1.21	0.66	0.65
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.59	0.50	0.51
query10	0.56	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.62	0.60
query14	2.73	2.75	2.77
query15	0.90	0.83	0.82
query16	0.39	0.39	0.36
query17	1.06	1.08	1.01
query18	0.23	0.21	0.21
query19	1.96	1.84	2.03
query20	0.01	0.02	0.01
query21	15.37	0.92	0.57
query22	0.75	0.77	0.71
query23	15.27	1.40	0.53
query24	2.79	1.28	1.10
query25	0.28	0.16	0.10
query26	0.37	0.17	0.14
query27	0.06	0.05	0.05
query28	13.40	1.61	1.06
query29	12.60	4.09	3.25
query30	0.25	0.09	0.07
query31	2.83	0.61	0.38
query32	3.23	0.55	0.46
query33	3.05	3.01	3.07
query34	16.74	5.19	4.50
query35	4.51	4.53	4.48
query36	0.63	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.59 s
Total hot run time: 31.41 s

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 6, 2025
Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by at least one committer and no changes requested.

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit af63014 into apache:master Jan 6, 2025
26 of 29 checks passed
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jan 7, 2025
… Query Processing (apache#46310)

Problem Summary:
1. different values of inverted_index_max_expansions require separate
caches.
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jan 16, 2025
… Query Processing (apache#46310)

Problem Summary:
1. different values of inverted_index_max_expansions require separate
caches.
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jan 19, 2025
… Query Processing (apache#46310)

Problem Summary:
1. different values of inverted_index_max_expansions require separate
caches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.0.x-conflict reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants