Skip to content

branch-3.1: [improve](glue)Refine Glue region and endpoint initialization #57365 #57479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 30, 2025
Merged

branch-3.1: [improve](glue)Refine Glue region and endpoint initialization #57365 #57479

merged 1 commit into from
Oct 30, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 30, 2025

Cherry-picked from #57365

@CalvinKirs
[improve](glue)Refine Glue region and endpoint initialization (#57365)
3cb67b3 
### Summary
- Build `glueEndpoint` automatically when only `glueRegion` is provided.
- Extract `glueRegion` from `glueEndpoint` when only endpoint is given.
- Reject unsupported endpoint formats (e.g. `.amazonaws.com.cn`).
- Add clear exception messages for invalid or missing configuration.

### Example
| Input | Output / Exception |
|--------|--------------------|
| `glue.region=us-east-1` |
`glue.endpoint=https://glue.us-east-1.amazonaws.com` |
| `glue.endpoint=https://glue.us-east-1.amazonaws.com` |
`glue.region=us-east-1` |
| `glue.endpoint=https://glue.us-east-1.amazonaws.com.cn` | `Invalid AWS
Glue endpoint format` |
| `glue.endpoint=https://glue.us-east-1.amazonaws.com.cn`
`glue.region=us-east-1` |
`glue.endpoint=https://glue.us-east-1.amazonaws.com.cn`
`glue.region=us-east-1` |
@github-actions github-actions bot requested a review from morrySnow as a code owner October 30, 2025 02:47
@Thearas
Copy link
Contributor

Thearas commented Oct 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 30, 2025
@Thearas
Copy link
Contributor

Thearas commented Oct 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-H: Total hot run time: 33460 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3cb67b3eae0d6149a542adb817dd2fc6dbef7f28, data reload: false

------ Round 1 ----------------------------------
q1	17636	5611	5381	5381
q2	2037	404	290	290
q3	12274	1259	795	795
q4	10332	883	490	490
q5	9373	2423	2207	2207
q6	198	171	138	138
q7	934	784	650	650
q8	9342	1457	1231	1231
q9	5290	5017	5003	5003
q10	6802	2298	1842	1842
q11	479	279	289	279
q12	363	370	227	227
q13	17816	3613	3070	3070
q14	239	235	213	213
q15	528	469	459	459
q16	436	448	393	393
q17	662	882	425	425
q18	7056	6466	6559	6466
q19	1217	990	567	567
q20	372	349	223	223
q21	3253	2339	2105	2105
q22	1095	1060	1006	1006
Total cold run time: 107734 ms
Total hot run time: 33460 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5559	5538	5583	5538
q2	248	329	239	239
q3	2309	2636	2353	2353
q4	1406	1811	1332	1332
q5	4495	5137	5025	5025
q6	185	169	130	130
q7	2063	1974	1897	1897
q8	2661	2869	2785	2785
q9	7281	7312	7347	7312
q10	3058	3239	2851	2851
q11	610	525	495	495
q12	699	778	607	607
q13	3433	3794	3243	3243
q14	281	301	291	291
q15	532	468	480	468
q16	450	484	455	455
q17	1278	1767	1294	1294
q18	7817	7547	7567	7547
q19	887	1231	1059	1059
q20	2059	2112	1944	1944
q21	5451	4853	4758	4758
q22	1118	1131	1067	1067
Total cold run time: 53880 ms
Total hot run time: 52690 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-DS: Total hot run time: 198789 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3cb67b3eae0d6149a542adb817dd2fc6dbef7f28, data reload: false

query1	966	419	459	419
query2	6177	2120	2117	2117
query3	8683	226	220	220
query4	33481	24312	24461	24312
query5	4605	647	477	477
query6	317	214	198	198
query7	4224	550	343	343
query8	324	274	259	259
query9	9785	2918	2899	2899
query10	519	336	277	277
query11	18097	15560	15278	15278
query12	169	120	113	113
query13	1559	603	464	464
query14	10313	7512	6989	6989
query15	257	197	185	185
query16	8119	688	552	552
query17	1585	803	627	627
query18	2168	433	328	328
query19	237	215	180	180
query20	128	128	129	128
query21	205	133	119	119
query22	4746	4761	4624	4624
query23	35411	35264	34177	34177
query24	7619	2804	2856	2804
query25	507	506	433	433
query26	1170	293	179	179
query27	2058	524	404	404
query28	5723	2481	2475	2475
query29	651	614	464	464
query30	244	193	170	170
query31	1034	940	878	878
query32	99	64	63	63
query33	508	394	333	333
query34	845	914	574	574
query35	806	823	756	756
query36	1116	1124	1098	1098
query37	115	95	75	75
query38	4062	4080	4055	4055
query39	1589	1478	1540	1478
query40	238	138	124	124
query41	54	52	51	51
query42	135	120	117	117
query43	580	585	547	547
query44	1497	986	974	974
query45	191	186	172	172
query46	967	1124	729	729
query47	2011	2014	1986	1986
query48	490	487	402	402
query49	784	539	427	427
query50	757	731	477	477
query51	7825	7826	7603	7603
query52	118	114	105	105
query53	263	283	214	214
query54	637	623	560	560
query55	92	90	90	90
query56	306	304	276	276
query57	1281	1311	1241	1241
query58	251	241	243	241
query59	3424	3505	3298	3298
query60	338	338	312	312
query61	145	143	151	143
query62	826	779	700	700
query63	247	209	210	209
query64	4737	1200	816	816
query65	3438	3336	3366	3336
query66	842	444	348	348
query67	17057	16333	16173	16173
query68	7936	936	639	639
query69	521	335	295	295
query70	1278	1277	1300	1277
query71	421	348	299	299
query72	5312	3823	3795	3795
query73	743	794	401	401
query74	10189	9350	9013	9013
query75	3324	3213	2693	2693
query76	3237	1195	804	804
query77	763	411	297	297
query78	10495	10454	9784	9784
query79	3759	942	644	644
query80	756	555	455	455
query81	489	262	230	230
query82	629	125	92	92
query83	175	171	159	159
query84	287	118	89	89
query85	816	390	321	321
query86	382	324	325	324
query87	4453	4439	4299	4299
query88	5079	2720	2691	2691
query89	437	349	308	308
query90	1793	198	201	198
query91	147	146	122	122
query92	71	61	57	57
query93	1796	950	623	623
query94	688	411	325	325
query95	365	300	302	300
query96	568	657	309	309
query97	3256	3237	3224	3224
query98	232	227	222	222
query99	1566	1432	1305	1305
Total cold run time: 301715 ms
Total hot run time: 198789 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

ClickBench: Total hot run time: 28.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cb67b3eae0d6149a542adb817dd2fc6dbef7f28, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.11
query5	0.54	0.51	0.53
query6	1.13	0.75	0.73
query7	0.03	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.50	0.52
query10	0.56	0.57	0.54
query11	0.14	0.12	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	0.79	0.79	0.81
query15	0.85	0.84	0.83
query16	0.39	0.39	0.40
query17	1.08	1.07	1.07
query18	0.25	0.24	0.24
query19	1.97	1.86	1.97
query20	0.02	0.01	0.01
query21	15.36	0.92	0.61
query22	0.74	0.72	0.66
query23	15.16	1.48	0.61
query24	3.66	0.42	0.53
query25	0.16	0.08	0.08
query26	0.33	0.16	0.16
query27	0.05	0.05	0.05
query28	12.61	1.05	0.44
query29	12.58	4.15	3.46
query30	0.27	0.09	0.07
query31	2.82	0.64	0.40
query32	3.24	0.53	0.47
query33	3.07	3.06	3.07
query34	16.60	5.28	4.56
query35	4.56	4.60	4.54
query36	0.65	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 103.48 s
Total hot run time: 28.97 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 30, 2025

FE UT Coverage Report

Increment line coverage 40.00% (4/10) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit f9f5d38 into branch-3.1 Oct 30, 2025
23 checks passed
@morrySnow morrySnow deleted the auto-pick-57365-branch-3.1 branch October 30, 2025 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @CalvinKirs