Skip to content

[opt](filecache) schema change adaptive to file cache #57470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[opt](filecache) schema change adaptive to file cache #57470

wants to merge 1 commit into from
+276 −6

Conversation

@HaolinGuan7211
Copy link

@HaolinGuan7211 HaolinGuan7211 commented Oct 29, 2025

No description provided.

@Thearas
Copy link
Contributor

Thearas commented Oct 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou gavinchou changed the title [feat] : schema change adaptive to file cache [opt](filecache) schema change adaptive to file cache Nov 4, 2025
gavinchou
gavinchou reviewed Nov 4, 2025
View reviewed changes
be/src/cloud/cloud_index_change_compaction.cpp Outdated
DCHECK_EQ(_output_rowset->version(), _input_rowsets[0]->version());
// MUST NOT move input rowset to stale path.
cloud_tablet()->add_rowsets({_output_rowset}, true, wrlock);
cloud_tablet()->add_rowsets({_output_rowset}, true, wrlock, true);
Copy link
Contributor

@gavinchou gavinchou Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be also fixed, we cannot always warm up data when doing index compaction.

Copy link
Contributor

@gavinchou gavinchou Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in a separate PR

@HaolinGuan7211 HaolinGuan7211 marked this pull request as ready for review November 5, 2025 01:04
@HaolinGuan7211
Copy link
Author

HaolinGuan7211 commented Nov 5, 2025

run buildall

1 similar comment
@HaolinGuan7211
Copy link
Author

HaolinGuan7211 commented Nov 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 5, 2025

TPC-DS: Total hot run time: 190149 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 797459739f4c86b7a781bf72a21236a93535dcbc, data reload: false

query1	1033	430	409	409
query2	6566	1672	1754	1672
query3	6748	236	228	228
query4	26568	24025	23513	23513
query5	5715	631	480	480
query6	344	231	223	223
query7	4650	503	299	299
query8	318	264	259	259
query9	8745	2587	2596	2587
query10	552	333	300	300
query11	15684	15053	14879	14879
query12	183	122	118	118
query13	1685	568	460	460
query14	11611	9410	9447	9410
query15	211	189	175	175
query16	7689	709	546	546
query17	1630	819	685	685
query18	2117	526	384	384
query19	253	226	192	192
query20	141	140	138	138
query21	221	155	153	153
query22	4743	4725	4748	4725
query23	35008	33684	33868	33684
query24	8648	2588	2521	2521
query25	605	518	465	465
query26	1268	288	158	158
query27	2738	524	368	368
query28	4436	2233	2211	2211
query29	815	700	516	516
query30	299	236	211	211
query31	943	869	799	799
query32	81	67	69	67
query33	574	363	325	325
query34	805	849	528	528
query35	783	858	750	750
query36	962	990	894	894
query37	122	106	88	88
query38	3628	3576	3537	3537
query39	1471	1417	1414	1414
query40	220	128	116	116
query41	61	57	59	57
query42	121	109	108	108
query43	485	498	462	462
query44	1236	736	750	736
query45	181	178	172	172
query46	884	1004	641	641
query47	1739	1752	1691	1691
query48	404	415	323	323
query49	785	486	409	409
query50	639	683	410	410
query51	4003	3997	3856	3856
query52	109	106	105	105
query53	235	265	199	199
query54	305	281	271	271
query55	87	87	84	84
query56	314	333	297	297
query57	1181	1182	1106	1106
query58	303	273	275	273
query59	2575	2674	2472	2472
query60	356	353	313	313
query61	159	157	161	157
query62	800	730	703	703
query63	234	203	198	198
query64	4439	1162	873	873
query65	4059	3972	3955	3955
query66	1037	436	337	337
query67	15256	15051	14866	14866
query68	8557	952	595	595
query69	548	321	300	300
query70	1358	1290	1361	1290
query71	512	354	320	320
query72	5832	4929	4897	4897
query73	680	570	364	364
query74	8949	9168	9057	9057
query75	4165	3303	2823	2823
query76	3872	1190	751	751
query77	798	416	327	327
query78	9527	9826	8909	8909
query79	2014	793	586	586
query80	668	549	505	505
query81	475	262	224	224
query82	418	158	135	135
query83	274	258	247	247
query84	255	110	95	95
query85	875	481	442	442
query86	347	303	294	294
query87	3715	3725	3597	3597
query88	3375	2254	2218	2218
query89	396	322	300	300
query90	2024	219	218	218
query91	174	168	136	136
query92	83	65	63	63
query93	1154	979	646	646
query94	677	424	343	343
query95	404	319	321	319
query96	492	579	279	279
query97	2951	2958	2854	2854
query98	244	211	202	202
query99	1513	1428	1252	1252
Total cold run time: 280126 ms
Total hot run time: 190149 ms

@doris-robot
Copy link

doris-robot commented Nov 5, 2025

ClickBench: Total hot run time: 27.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 797459739f4c86b7a781bf72a21236a93535dcbc, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.04	0.05
query3	0.25	0.08	0.09
query4	1.61	0.12	0.12
query5	0.27	0.25	0.26
query6	1.19	0.66	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.54	0.51
query10	0.58	0.58	0.57
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.61
query14	1.02	0.99	0.99
query15	0.85	0.85	0.85
query16	0.41	0.40	0.39
query17	1.03	1.05	1.04
query18	0.22	0.21	0.20
query19	1.93	1.82	1.81
query20	0.02	0.02	0.02
query21	15.44	0.18	0.13
query22	5.09	0.08	0.05
query23	15.67	0.25	0.11
query24	2.60	0.76	1.11
query25	0.08	0.06	0.05
query26	0.14	0.12	0.12
query27	0.07	0.05	0.05
query28	5.83	1.13	0.92
query29	12.62	3.93	3.16
query30	0.28	0.15	0.11
query31	2.82	0.58	0.38
query32	3.23	0.56	0.47
query33	3.08	3.00	3.01
query34	15.90	5.15	4.62
query35	4.56	4.56	4.57
query36	0.68	0.50	0.50
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.03	0.03
Total cold run time: 99.8 s
Total hot run time: 27.81 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 5, 2025

BE UT Coverage Report

Increment line coverage 3.57% (1/28) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.76% (18189/34472)
Line Coverage 38.19% (165691/433846)
Region Coverage 33.23% (129064/388426)
Branch Coverage 33.96% (55396/163106)

gavinchou
gavinchou previously approved these changes Nov 6, 2025
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou left review comments

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dev/3.1.x dev/4.0.x

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HaolinGuan7211 @Thearas @doris-robot @hello-stephen @gavinchou