Skip to content

Navigation Menu

Sign in

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

branch-3.0: [opt](load) Increase the default value of webserver_num_workers #46304 #46593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 9, 2025
Merged

branch-3.0: [opt](load) Increase the default value of webserver_num_workers #46304 #46593

merged 1 commit into from
Jan 9, 2025

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Jan 8, 2025

Cherry-picked from #46304

@liaoxin01
[opt](load) Increase the default value of webserver_num_workers (#46304)
2d7e08a 
1. In high-concurrency stream load scenarios, there aren't enough
webserver threads available.
2. During stream load, after data reception and while waiting for
execution, the synchronous wait occupies HTTP threads.

To address the issue of insufficient HTTP threads during
high-concurrency stream loads causing slow data reception, we increased
the number of threads.
@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 8, 2025
@dataroaring dataroaring reopened this Jan 8, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d7e08ae6f704f6dd9b1cbeae9d4f5f3a7b96bae, data reload: false

------ Round 1 ----------------------------------
q1	17570	7398	7246	7246
q2	2045	175	159	159
q3	10556	1149	1156	1149
q4	10291	794	794	794
q5	8634	2891	2863	2863
q6	248	154	152	152
q7	980	631	604	604
q8	9457	1982	2083	1982
q9	6769	6538	6469	6469
q10	8304	2305	2315	2305
q11	469	265	269	265
q12	408	223	220	220
q13	17774	2992	3015	2992
q14	247	212	217	212
q15	559	510	516	510
q16	709	600	623	600
q17	977	609	545	545
q18	7360	6802	6735	6735
q19	1406	1046	971	971
q20	471	203	207	203
q21	4112	3214	3493	3214
q22	1091	992	1024	992
Total cold run time: 110437 ms
Total hot run time: 41182 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7236	7228	7192	7192
q2	328	233	227	227
q3	3171	2977	3021	2977
q4	2024	1836	1840	1836
q5	5630	5742	5719	5719
q6	222	141	147	141
q7	2201	1815	1767	1767
q8	3334	3531	3472	3472
q9	8877	8823	8862	8823
q10	3612	3603	3565	3565
q11	581	503	522	503
q12	794	600	642	600
q13	5245	3156	3091	3091
q14	284	251	246	246
q15	565	518	514	514
q16	706	668	658	658
q17	1785	1549	1547	1547
q18	7782	7439	7324	7324
q19	1634	1576	1436	1436
q20	2064	1826	1823	1823
q21	5374	5034	5181	5034
q22	1134	1005	988	988
Total cold run time: 64583 ms
Total hot run time: 59483 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193474 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d7e08ae6f704f6dd9b1cbeae9d4f5f3a7b96bae, data reload: false

query1	993	368	369	368
query2	6543	2124	2115	2115
query3	6704	225	227	225
query4	34180	23623	23639	23623
query5	4396	447	450	447
query6	279	176	178	176
query7	4619	307	322	307
query8	287	221	212	212
query9	9720	2665	2667	2665
query10	480	267	260	260
query11	18587	15402	15451	15402
query12	150	102	101	101
query13	1636	448	410	410
query14	10178	7733	7670	7670
query15	262	182	185	182
query16	8194	471	489	471
query17	1676	585	570	570
query18	2160	330	349	330
query19	379	162	158	158
query20	118	109	112	109
query21	203	108	106	106
query22	4372	4236	4386	4236
query23	34901	34141	34400	34141
query24	11274	2896	2868	2868
query25	678	415	428	415
query26	1542	171	169	169
query27	2853	353	349	349
query28	7897	2418	2408	2408
query29	953	441	453	441
query30	335	173	169	169
query31	1058	814	835	814
query32	96	58	57	57
query33	780	298	298	298
query34	942	502	523	502
query35	890	704	699	699
query36	1112	933	983	933
query37	139	73	75	73
query38	4142	3893	3826	3826
query39	1461	1437	1436	1436
query40	285	99	98	98
query41	51	48	54	48
query42	112	102	99	99
query43	539	497	492	492
query44	1380	822	814	814
query45	182	166	169	166
query46	1153	712	739	712
query47	1912	1808	1842	1808
query48	481	377	391	377
query49	1214	397	391	391
query50	821	399	411	399
query51	7335	7139	6968	6968
query52	106	94	91	91
query53	255	185	188	185
query54	1219	469	462	462
query55	77	80	80	80
query56	286	257	257	257
query57	1273	1157	1115	1115
query58	240	219	217	217
query59	3311	3060	3051	3051
query60	300	279	256	256
query61	114	112	109	109
query62	896	689	690	689
query63	225	191	192	191
query64	5346	682	660	660
query65	3277	3248	3213	3213
query66	1436	348	331	331
query67	15902	15642	15422	15422
query68	4551	580	575	575
query69	465	274	271	271
query70	1178	1156	1146	1146
query71	333	256	267	256
query72	6461	4367	4218	4218
query73	797	355	366	355
query74	10262	8977	8972	8972
query75	3442	2666	2666	2666
query76	2888	1041	1120	1041
query77	388	285	285	285
query78	10638	9596	9736	9596
query79	1134	591	601	591
query80	777	453	442	442
query81	513	247	242	242
query82	1077	120	121	120
query83	255	152	145	145
query84	233	81	81	81
query85	1100	317	296	296
query86	321	271	296	271
query87	4457	4319	4395	4319
query88	3471	2408	2369	2369
query89	407	301	301	301
query90	2017	191	184	184
query91	190	153	151	151
query92	63	52	52	52
query93	1061	551	543	543
query94	801	317	290	290
query95	356	260	260	260
query96	625	285	284	284
query97	3350	3186	3189	3186
query98	222	208	195	195
query99	1530	1334	1329	1329
Total cold run time: 302260 ms
Total hot run time: 193474 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d7e08ae6f704f6dd9b1cbeae9d4f5f3a7b96bae, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.11	0.11
query5	0.52	0.51	0.51
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.57	0.51	0.50
query10	0.56	0.55	0.54
query11	0.13	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	3.05	3.05	2.94
query15	0.90	0.83	0.83
query16	0.41	0.37	0.36
query17	1.02	1.07	1.05
query18	0.24	0.21	0.21
query19	1.89	1.90	2.01
query20	0.01	0.02	0.01
query21	15.38	0.62	0.57
query22	2.45	2.69	2.21
query23	17.05	0.95	0.82
query24	2.79	1.46	1.19
query25	0.25	0.19	0.10
query26	0.36	0.14	0.14
query27	0.04	0.04	0.04
query28	10.42	1.12	1.07
query29	12.62	3.30	3.28
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.25	0.47	0.48
query33	2.98	3.01	3.04
query34	17.10	4.50	4.54
query35	4.60	4.56	4.54
query36	0.68	0.52	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.76 s
Total hot run time: 33.81 s

@liaoxin01 liaoxin01 merged commit ade6209 into branch-3.0 Jan 9, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-46304-branch-3.0 branch January 9, 2025 07:52
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Thearas @doris-robot @liaoxin01 @dataroaring
Morty Proxy This is a proxified and sanitized view of the page, visit original site.