Skip to content

[refactor](be) Unify read-time column resolution#65558

Open
TsukiokaKogane wants to merge 3 commits into
apache:masterfrom
TsukiokaKogane:refactor_update_in_seg_iter
Open

[refactor](be) Unify read-time column resolution#65558
TsukiokaKogane wants to merge 3 commits into
apache:masterfrom
TsukiokaKogane:refactor_update_in_seg_iter

Conversation

@TsukiokaKogane

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@TsukiokaKogane

Copy link
Copy Markdown
Contributor Author

run buildall

@TsukiokaKogane

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28860 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b2f1cd9c2096b869a66b1d47a7261ae62c95670, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17644	3996	3973	3973
q2	2016	321	202	202
q3	10291	1406	788	788
q4	4696	467	338	338
q5	7557	845	578	578
q6	198	167	136	136
q7	764	835	609	609
q8	10160	1463	1531	1463
q9	6083	4341	4313	4313
q10	6816	1761	1519	1519
q11	514	342	304	304
q12	743	548	431	431
q13	18086	3358	2718	2718
q14	270	266	241	241
q15	q16	773	772	700	700
q17	1804	1081	835	835
q18	6879	5635	5483	5483
q19	1589	1214	1027	1027
q20	757	635	532	532
q21	5824	2570	2376	2376
q22	426	358	294	294
Total cold run time: 103890 ms
Total hot run time: 28860 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4403	4270	4445	4270
q2	283	314	221	221
q3	4605	4934	4501	4501
q4	2029	2146	1348	1348
q5	4384	4338	4269	4269
q6	221	181	125	125
q7	1725	2241	1676	1676
q8	2428	2141	2079	2079
q9	7878	7768	7779	7768
q10	4722	4658	4230	4230
q11	536	408	371	371
q12	727	873	633	633
q13	3295	3610	2942	2942
q14	290	320	294	294
q15	q16	727	717	642	642
q17	1330	1292	1320	1292
q18	8060	7386	6904	6904
q19	1093	1052	1047	1047
q20	2234	2196	1924	1924
q21	5208	4509	4385	4385
q22	519	459	389	389
Total cold run time: 56697 ms
Total hot run time: 51310 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 179600 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b2f1cd9c2096b869a66b1d47a7261ae62c95670, data reload: false

query5	4343	618	478	478
query6	454	215	197	197
query7	4976	616	324	324
query8	356	191	169	169
query9	8785	4075	4066	4066
query10	452	342	283	283
query11	5918	2373	2154	2154
query12	161	102	124	102
query13	1281	605	422	422
query14	6188	5260	4958	4958
query14_1	4331	4337	4303	4303
query15	209	204	175	175
query16	991	454	420	420
query17	1125	681	574	574
query18	2414	468	351	351
query19	206	183	150	150
query20	106	106	108	106
query21	227	156	131	131
query22	13710	13592	13430	13430
query23	17155	16464	16112	16112
query23_1	16342	16352	16229	16229
query24	7589	1752	1297	1297
query24_1	1352	1309	1327	1309
query25	566	476	399	399
query26	1324	382	219	219
query27	2634	576	386	386
query28	4520	2034	2018	2018
query29	1073	611	518	518
query30	336	266	228	228
query31	1123	1107	965	965
query32	103	61	61	61
query33	523	332	257	257
query34	1151	1144	618	618
query35	747	809	669	669
query36	1393	1389	1228	1228
query37	156	111	96	96
query38	1873	1703	1653	1653
query39	918	926	903	903
query39_1	872	867	877	867
query40	251	164	142	142
query41	73	69	69	69
query42	96	94	95	94
query43	318	326	279	279
query44	1442	785	784	784
query45	198	191	182	182
query46	1086	1203	771	771
query47	2354	2319	2229	2229
query48	381	397	301	301
query49	581	431	358	358
query50	1040	415	339	339
query51	10570	10825	10456	10456
query52	90	93	77	77
query53	261	281	201	201
query54	310	250	228	228
query55	75	70	70	70
query56	310	308	281	281
query57	1402	1404	1299	1299
query58	298	268	253	253
query59	1576	1630	1469	1469
query60	310	281	265	265
query61	180	208	149	149
query62	679	650	588	588
query63	242	202	202	202
query64	2797	1008	844	844
query65	4855	4762	4772	4762
query66	1839	513	380	380
query67	29539	29430	29174	29174
query68	3301	1596	1032	1032
query69	416	288	258	258
query70	1059	983	954	954
query71	376	317	290	290
query72	3079	2754	2317	2317
query73	824	767	456	456
query74	5090	4962	4770	4770
query75	2603	2576	2210	2210
query76	2346	1169	769	769
query77	354	381	286	286
query78	12343	12301	11889	11889
query79	1384	1178	748	748
query80	1346	554	476	476
query81	538	316	279	279
query82	615	161	121	121
query83	367	317	306	306
query84	287	206	131	131
query85	976	615	508	508
query86	423	300	273	273
query87	1826	1813	1766	1766
query88	3777	2812	2775	2775
query89	463	407	352	352
query90	1938	195	184	184
query91	196	187	160	160
query92	65	62	59	59
query93	1669	1604	1027	1027
query94	732	360	296	296
query95	759	589	443	443
query96	1088	792	374	374
query97	2669	2669	2565	2565
query98	216	204	211	204
query99	1133	1164	1029	1029
Total cold run time: 265844 ms
Total hot run time: 179600 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b2f1cd9c2096b869a66b1d47a7261ae62c95670, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.13	0.13
query4	1.62	0.14	0.14
query5	0.24	0.22	0.23
query6	1.23	1.06	1.05
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.37	0.31	0.32
query10	0.53	0.57	0.54
query11	0.19	0.14	0.13
query12	0.17	0.15	0.14
query13	0.46	0.49	0.50
query14	1.03	1.01	1.00
query15	0.61	0.59	0.59
query16	0.33	0.34	0.33
query17	1.08	1.05	1.10
query18	0.22	0.21	0.21
query19	2.04	1.87	1.89
query20	0.02	0.01	0.01
query21	15.41	0.22	0.13
query22	4.64	0.06	0.05
query23	16.11	0.31	0.12
query24	2.94	0.44	0.32
query25	0.11	0.04	0.05
query26	0.72	0.20	0.17
query27	0.04	0.04	0.04
query28	3.55	0.87	0.55
query29	12.51	4.09	3.20
query30	0.27	0.15	0.15
query31	2.77	0.59	0.31
query32	3.22	0.60	0.49
query33	3.19	3.29	3.18
query34	15.50	4.24	3.51
query35	3.48	3.58	3.47
query36	0.56	0.43	0.44
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.15 s
Total hot run time: 24.85 s

@TsukiokaKogane

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 96.88% (62/64) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.08% (23681/41486)
Line Coverage 40.80% (232116/568903)
Region Coverage 36.65% (183278/500009)
Branch Coverage 37.80% (82044/217035)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (64/64) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.80% (30269/40465)
Line Coverage 58.91% (333027/565292)
Region Coverage 55.71% (279593/501866)
Branch Coverage 56.92% (123576/217096)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants