Skip to content

Navigation Menu

Sign in

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

[fix](nereids) fix sql cache bug and some tests #46443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Jan 7, 2025
Merged

[fix](nereids) fix sql cache bug and some tests #46443

merged 13 commits into from
Jan 7, 2025

Conversation

924060929
Copy link
Contributor

@924060929 924060929 commented Jan 6, 2025 •
edited
Loading

What problem does this PR solve?

  1. use retry to fix unstable test colocate_union_numbers, prune_bucket_with_bucket_shuffle_join
  2. fix failed test explain, this bug only exists in master branch, introduced by [enhancement](nereids) improve lots of values in insert into values statement #40202
  3. fix sql cache bug which use stale cache after drop table and create(table id changed), test in parse_sql_from_sql_cache, introduced by [enhancement](Nereids) Enable parse sql from sql cache #33262
  4. regression test add foreachFrontends, foreachBackends, retry function

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929 924060929 changed the title [fix](test) fix unstable test parse_sql_from_sql_cache/colocate_union_numbers/prune_bucket_with_bucket_shuffle_join/explain [fix](nereids) fix sql cache bug and some tests Jan 6, 2025
@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35e050e3180d9d54cd1ab2da4efa67c99a06145f, data reload: false

------ Round 1 ----------------------------------
q1	17613	6175	5977	5977
q2	2043	291	164	164
q3	10442	1231	704	704
q4	10215	842	424	424
q5	7493	2144	1988	1988
q6	204	175	146	146
q7	885	734	633	633
q8	9243	1335	1136	1136
q9	5237	4881	4939	4881
q10	6726	2295	1853	1853
q11	475	276	259	259
q12	336	351	213	213
q13	17751	3670	3109	3109
q14	226	234	211	211
q15	552	500	500	500
q16	633	596	593	593
q17	537	826	317	317
q18	7179	6509	6431	6431
q19	1223	960	542	542
q20	316	320	189	189
q21	2816	2179	1956	1956
q22	359	327	311	311
Total cold run time: 102504 ms
Total hot run time: 32537 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6984	6200	6205	6200
q2	231	323	230	230
q3	2251	2646	2314	2314
q4	1341	1806	1345	1345
q5	4344	4775	4752	4752
q6	187	176	143	143
q7	2071	1995	1809	1809
q8	2592	2760	2722	2722
q9	7288	7213	7240	7213
q10	3043	3309	2851	2851
q11	610	505	491	491
q12	666	758	627	627
q13	3449	3879	3278	3278
q14	283	302	287	287
q15	568	520	505	505
q16	637	696	647	647
q17	1207	1701	1249	1249
q18	7632	7340	7141	7141
q19	774	1040	1079	1040
q20	1964	1980	1818	1818
q21	5382	4999	4800	4800
q22	584	624	577	577
Total cold run time: 54088 ms
Total hot run time: 52039 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35e050e3180d9d54cd1ab2da4efa67c99a06145f, data reload: false

query1	960	398	363	363
query2	6520	2299	2329	2299
query3	6715	218	213	213
query4	34580	23532	23182	23182
query5	4383	622	447	447
query6	268	201	192	192
query7	4637	489	310	310
query8	316	246	236	236
query9	9505	2654	2641	2641
query10	464	298	247	247
query11	18333	15644	15196	15196
query12	160	104	105	104
query13	1649	516	384	384
query14	11503	7675	6833	6833
query15	235	209	188	188
query16	8080	614	462	462
query17	1553	718	552	552
query18	2046	398	319	319
query19	204	178	164	164
query20	122	116	110	110
query21	203	119	103	103
query22	4577	4389	4433	4389
query23	34631	33383	32897	32897
query24	6379	2292	2249	2249
query25	440	438	380	380
query26	899	253	150	150
query27	2043	458	329	329
query28	5449	2431	2424	2424
query29	547	541	428	428
query30	231	181	153	153
query31	1001	898	806	806
query32	77	68	60	60
query33	511	353	307	307
query34	738	863	498	498
query35	796	801	719	719
query36	1021	1051	953	953
query37	116	96	73	73
query38	4110	4125	4068	4068
query39	1503	1447	1411	1411
query40	207	110	100	100
query41	46	45	46	45
query42	115	107	102	102
query43	517	533	483	483
query44	1274	789	805	789
query45	192	171	165	165
query46	849	1029	664	664
query47	1934	1946	1890	1890
query48	365	399	327	327
query49	754	477	399	399
query50	607	640	381	381
query51	7093	7029	7169	7029
query52	97	99	91	91
query53	217	245	192	192
query54	465	471	405	405
query55	79	79	76	76
query56	257	247	238	238
query57	1204	1216	1119	1119
query58	239	222	239	222
query59	2986	3064	2939	2939
query60	278	259	248	248
query61	108	107	106	106
query62	864	788	747	747
query63	221	196	196	196
query64	4189	1081	735	735
query65	3306	3160	3192	3160
query66	1099	419	314	314
query67	16067	15963	15523	15523
query68	8991	688	500	500
query69	467	285	251	251
query70	1219	1151	1132	1132
query71	444	283	267	267
query72	6138	3817	3669	3669
query73	644	745	358	358
query74	9665	9045	8758	8758
query75	4629	3135	2646	2646
query76	4811	1169	763	763
query77	885	365	271	271
query78	10112	10224	9521	9521
query79	3094	792	625	625
query80	685	521	417	417
query81	469	271	222	222
query82	621	151	126	126
query83	191	168	140	140
query84	275	80	74	74
query85	754	344	306	306
query86	355	329	295	295
query87	4825	4591	4308	4308
query88	3155	2232	2153	2153
query89	409	333	298	298
query90	1679	187	186	186
query91	129	131	105	105
query92	62	56	55	55
query93	1262	859	512	512
query94	574	409	295	295
query95	336	260	249	249
query96	492	604	284	284
query97	2934	2956	2785	2785
query98	223	204	199	199
query99	1666	1541	1427	1427
Total cold run time: 295329 ms
Total hot run time: 188721 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35e050e3180d9d54cd1ab2da4efa67c99a06145f, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.08	0.07
query4	1.60	0.10	0.10
query5	0.40	0.41	0.40
query6	1.17	0.64	0.64
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.74	2.84	2.82
query15	0.89	0.82	0.81
query16	0.38	0.37	0.39
query17	1.07	1.06	1.06
query18	0.22	0.20	0.21
query19	1.95	1.86	2.02
query20	0.01	0.01	0.02
query21	15.35	0.94	0.60
query22	0.74	0.76	0.73
query23	15.22	1.47	0.66
query24	2.68	1.04	1.74
query25	0.11	0.21	0.17
query26	0.37	0.14	0.13
query27	0.08	0.04	0.04
query28	14.20	1.55	1.04
query29	12.60	3.93	3.27
query30	0.25	0.09	0.08
query31	2.81	0.62	0.37
query32	3.23	0.53	0.47
query33	3.06	3.09	3.07
query34	16.52	5.14	4.52
query35	4.56	4.53	4.50
query36	0.67	0.50	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.82 s
Total hot run time: 31.8 s

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddae72481fb2ed211bf60095831f9a5b897bd13f, data reload: false

------ Round 1 ----------------------------------
q1	17608	6142	6053	6053
q2	2045	296	172	172
q3	10423	1287	720	720
q4	10206	875	449	449
q5	7528	2184	1996	1996
q6	202	183	149	149
q7	891	730	636	636
q8	9221	1352	1158	1158
q9	5641	5065	5002	5002
q10	6854	2341	1856	1856
q11	479	275	270	270
q12	368	370	221	221
q13	17766	3712	3092	3092
q14	244	230	221	221
q15	566	528	514	514
q16	611	623	596	596
q17	566	862	332	332
q18	6797	6379	6508	6379
q19	1222	992	565	565
q20	310	343	190	190
q21	2850	2150	1982	1982
q22	362	333	306	306
Total cold run time: 102760 ms
Total hot run time: 32859 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6463	6203	6261	6203
q2	238	328	229	229
q3	2253	2635	2317	2317
q4	1372	1848	1375	1375
q5	4358	4739	4676	4676
q6	180	173	138	138
q7	1955	1852	1718	1718
q8	2524	2700	2600	2600
q9	6931	6944	6875	6875
q10	2994	3212	2709	2709
q11	577	504	495	495
q12	664	751	580	580
q13	3311	3775	3033	3033
q14	266	297	265	265
q15	571	501	492	492
q16	652	667	658	658
q17	1169	1692	1222	1222
q18	7366	7275	6946	6946
q19	768	1091	1065	1065
q20	1942	2003	1797	1797
q21	5619	5096	4797	4797
q22	660	610	552	552
Total cold run time: 52833 ms
Total hot run time: 50742 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190051 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddae72481fb2ed211bf60095831f9a5b897bd13f, data reload: false

query1	893	381	378	378
query2	5738	2411	2293	2293
query3	5839	225	219	219
query4	32924	23394	23862	23394
query5	4389	623	461	461
query6	269	193	180	180
query7	4456	481	304	304
query8	299	245	238	238
query9	9233	2666	2643	2643
query10	439	313	241	241
query11	18067	15355	15247	15247
query12	149	105	113	105
query13	1497	537	421	421
query14	10139	7356	7115	7115
query15	241	193	201	193
query16	6983	567	457	457
query17	1375	739	561	561
query18	2118	396	293	293
query19	204	173	148	148
query20	126	118	113	113
query21	205	120	100	100
query22	4388	4283	4321	4283
query23	34149	33141	33481	33141
query24	6202	2197	2293	2197
query25	491	438	382	382
query26	952	267	148	148
query27	2036	463	339	339
query28	5053	2458	2427	2427
query29	737	574	441	441
query30	236	184	152	152
query31	1009	895	807	807
query32	107	63	61	61
query33	509	361	302	302
query34	764	826	515	515
query35	805	821	727	727
query36	1034	1049	972	972
query37	117	95	70	70
query38	4431	4035	4272	4035
query39	1485	1441	1426	1426
query40	203	137	100	100
query41	44	44	44	44
query42	123	106	108	106
query43	510	514	493	493
query44	1273	813	792	792
query45	176	178	167	167
query46	864	1040	637	637
query47	1945	1924	1857	1857
query48	385	393	316	316
query49	761	491	391	391
query50	654	637	376	376
query51	7166	7197	7173	7173
query52	104	101	91	91
query53	219	252	187	187
query54	469	492	413	413
query55	87	79	74	74
query56	264	268	238	238
query57	1198	1167	1115	1115
query58	235	218	242	218
query59	3112	3172	2897	2897
query60	269	271	245	245
query61	108	134	133	133
query62	850	798	757	757
query63	229	197	196	196
query64	4330	1058	633	633
query65	3275	3178	3208	3178
query66	1045	412	310	310
query67	16048	15883	15576	15576
query68	9242	686	501	501
query69	507	287	258	258
query70	1203	1067	1103	1067
query71	448	285	251	251
query72	6638	3837	3968	3837
query73	705	737	357	357
query74	10273	8947	9311	8947
query75	4722	3159	2676	2676
query76	4654	1173	766	766
query77	769	363	275	275
query78	10128	10202	9442	9442
query79	2934	811	601	601
query80	601	520	427	427
query81	472	263	230	230
query82	494	152	121	121
query83	176	169	143	143
query84	273	93	69	69
query85	767	364	301	301
query86	346	316	295	295
query87	4508	4819	4566	4566
query88	3882	2191	2171	2171
query89	407	338	295	295
query90	1973	194	188	188
query91	137	148	108	108
query92	64	58	55	55
query93	998	844	521	521
query94	663	387	295	295
query95	335	267	253	253
query96	487	601	282	282
query97	2904	2959	2858	2858
query98	240	210	207	207
query99	1616	1565	1446	1446
Total cold run time: 289600 ms
Total hot run time: 190051 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddae72481fb2ed211bf60095831f9a5b897bd13f, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.08
query4	1.62	0.11	0.11
query5	0.42	0.41	0.40
query6	1.15	0.65	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.54	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.62	0.61	0.59
query14	2.84	2.68	2.82
query15	0.90	0.83	0.84
query16	0.39	0.37	0.37
query17	1.07	1.07	0.97
query18	0.22	0.21	0.22
query19	1.95	1.77	2.03
query20	0.01	0.01	0.02
query21	15.37	0.96	0.57
query22	0.75	0.82	0.55
query23	15.37	1.43	0.54
query24	3.20	0.45	0.97
query25	0.19	0.18	0.08
query26	0.34	0.14	0.13
query27	0.06	0.05	0.04
query28	13.07	1.57	1.05
query29	12.58	3.92	3.27
query30	0.25	0.09	0.06
query31	2.83	0.59	0.38
query32	3.23	0.54	0.48
query33	3.11	3.16	3.05
query34	16.94	5.11	4.48
query35	4.57	4.50	4.48
query36	0.63	0.50	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 105.97 s
Total hot run time: 30.42 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit e718c0b into apache:master Jan 7, 2025
31 of 33 checks passed
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Jan 7, 2025
@924060929
[fix](nereids) fix sql cache bug and some tests (apache#46443)
ed39efb 
1. use retry to fix unstable test `colocate_union_numbers`,
`prune_bucket_with_bucket_shuffle_join`
2. fix failed test `explain`, this bug only exists in master branch,
introduced by apache#40202
3. fix sql cache bug which use stale cache after drop table and
create(table id changed), test in `parse_sql_from_sql_cache`, introduced
by apache#33262
4. regression test add `foreachFrontends`, `foreachBackends`, `retry`
function

(cherry picked from commit e718c0b)
@924060929 924060929 mentioned this pull request Jan 7, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jan 7, 2025
@924060929
[fix](nereids) fix sql cache bug and some tests (#46443)
f507231 
1. use retry to fix unstable test `colocate_union_numbers`,
`prune_bucket_with_bucket_shuffle_join`
2. fix failed test `explain`, this bug only exists in master branch,
introduced by #40202
3. fix sql cache bug which use stale cache after drop table and
create(table id changed), test in `parse_sql_from_sql_cache`, introduced
by #33262
4. regression test add `foreachFrontends`, `foreachBackends`, `retry`
function
@github-actions github-actions bot mentioned this pull request Jan 7, 2025
Merged
924060929 pushed a commit that referenced this pull request Jan 7, 2025
@github-actions
branch-3.0: [fix](nereids) fix sql cache bug and some tests #46443 (#…
6551775 
…46506)

1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join`
2. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by #33262
3. regression test add `foreachFrontends`, `foreachBackends`, `retry` function
@924060929 924060929 mentioned this pull request Jan 7, 2025
Merged
16 tasks
924060929 added a commit that referenced this pull request Jan 7, 2025
@924060929
[fix](cache) fix mtmv cache can not use test (#46537)
1320116 
fix mtmv can not use sql cache, introduced by #46443
github-actions bot pushed a commit that referenced this pull request Jan 7, 2025
@924060929
[fix](cache) fix mtmv cache can not use test (#46537)
b05f34f 
fix mtmv can not use sql cache, introduced by #46443
924060929 pushed a commit that referenced this pull request Jan 8, 2025
@github-actions
branch-3.0: [fix](cache) fix mtmv cache can not use test #46537 (#46555)
f52b44b 
fix mtmv can not use sql cache, introduced by #46443
yiguolei pushed a commit to 924060929/incubator-doris that referenced this pull request Jan 8, 2025
@924060929 @yiguolei
[fix](nereids) fix sql cache bug and some tests (apache#46443)
a041225 
1. use retry to fix unstable test `colocate_union_numbers`,
`prune_bucket_with_bucket_shuffle_join`
2. fix failed test `explain`, this bug only exists in master branch,
introduced by apache#40202
3. fix sql cache bug which use stale cache after drop table and
create(table id changed), test in `parse_sql_from_sql_cache`, introduced
by apache#33262
4. regression test add `foreachFrontends`, `foreachBackends`, `retry`
function

(cherry picked from commit e718c0b)
yiguolei pushed a commit that referenced this pull request Jan 8, 2025
@924060929
[fix](nereids) fix sql cache bug and some tests (#46443) (#46505)
2687ded 
cherry pick from #46443
github-actions bot pushed a commit that referenced this pull request Jan 8, 2025
@924060929
[fix](cache) fix mtmv cache can not use test (#46537)
4f35c7d 
fix mtmv can not use sql cache, introduced by #46443
@924060929 924060929 deleted the fix-test branch January 8, 2025 06:49
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Open
csun5285 pushed a commit to csun5285/doris that referenced this pull request Mar 28, 2025
@morrySnow
[fix](nereids) fix sql cache bug and some tests apache#46443 apache#4…
900a7e6 
…6537 (apache#3821)

pick from apache doris' branch-3.0

- apache#46506
- apache#46555
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@924060929 @hello-stephen @doris-robot @starocean999 @morrySnow @yiguolei
Morty Proxy This is a proxified and sanitized view of the page, visit original site.