Skip to content

[fix](outfile)Preserve DateTimeV2 scale in OUTFILE CSV.#64344

Open
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:outfile_csv_format
Open

[fix](outfile)Preserve DateTimeV2 scale in OUTFILE CSV.#64344
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:outfile_csv_format

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

CSV OUTFILE serialization for DateTimeV2 was using the value-level default formatter, which emits six fractional digits whenever microseconds are present. The serializer already carries the declared DateTimeV2 scale, so pass it through and cover the local OUTFILE CSV path with a focused Groovy regression.
example :

table:
CREATE TABLE test_outfile_datetimev2 (
    id INT,
    dt3 DATETIMEV2(3) ) ...

data :
1    2026-06-06 15:54:51.442

before outfile : 
2026-06-06 15:54:51.442000


after outfile:
2026-06-06 15:54:51.442

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

CSV OUTFILE serialization for DateTimeV2 was using the value-level default formatter, which emits six fractional digits whenever microseconds are present. The serializer already carries the declared DateTimeV2 scale, so pass it through and cover the local OUTFILE CSV path with a focused Groovy regression.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

auto val = assert_cast<const ColumnDateTimeV2&, TypeCheckOnRelease::DISABLE>(*ptr).get_element(
row_num);
char buf[64];
char* pos = val.to_string(buf);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 为什么是json,不是调用csv 相关的方法?
  2. core 目录下所有的改动,需要有单测?
  3. scale, decimal,datetimev1, timestamp 这些的scale都是咋处理的?
  4. csv 格式,parquet,arrow ,json,text,csv, scale 问题是否存在?
  5. 嵌套的时候, array 里面有一个datetimev2的时候,我们能否处理?

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jun 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguolei

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.89% (21139/39227)
Line Coverage 37.60% (201295/535393)
Region Coverage 33.63% (157865/469389)
Branch Coverage 34.67% (69110/199338)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e42b209cfc4da9776b5de80fe188459f3e82905, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17619	3969	3967	3967
q2	q3	10766	1426	823	823
q4	4688	511	365	365
q5	7552	883	594	594
q6	182	168	135	135
q7	769	844	655	655
q8	9412	1893	1706	1706
q9	5939	4548	4610	4548
q10	6810	1846	1526	1526
q11	432	275	252	252
q12	627	427	287	287
q13	18151	3318	2787	2787
q14	265	259	247	247
q15	q16	827	802	717	717
q17	942	991	1033	991
q18	6846	5699	5494	5494
q19	1295	1250	1060	1060
q20	524	409	262	262
q21	6253	2796	2669	2669
q22	457	378	315	315
Total cold run time: 100356 ms
Total hot run time: 29400 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5081	4766	4787	4766
q2	q3	4830	5372	4733	4733
q4	2167	2212	1428	1428
q5	4744	4666	4868	4666
q6	236	175	127	127
q7	1884	1728	1595	1595
q8	2516	2098	2072	2072
q9	7944	7875	7656	7656
q10	4725	4676	4265	4265
q11	538	378	353	353
q12	745	738	532	532
q13	3070	3363	2818	2818
q14	272	280	249	249
q15	q16	673	691	605	605
q17	1269	1248	1242	1242
q18	7381	6932	6671	6671
q19	1083	1123	1108	1108
q20	2215	2207	1951	1951
q21	5273	4610	4381	4381
q22	538	463	414	414
Total cold run time: 57184 ms
Total hot run time: 51632 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 169785 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e42b209cfc4da9776b5de80fe188459f3e82905, data reload: false

query5	4314	631	475	475
query6	463	204	186	186
query7	4832	558	307	307
query8	364	213	202	202
query9	8773	3982	3957	3957
query10	443	333	263	263
query11	5939	2367	2200	2200
query12	162	106	98	98
query13	1285	589	437	437
query14	6465	5430	5089	5089
query14_1	4409	4402	4383	4383
query15	210	199	182	182
query16	1030	457	454	454
query17	1182	732	607	607
query18	2738	480	356	356
query19	217	191	151	151
query20	122	113	107	107
query21	224	141	119	119
query22	13794	13640	13426	13426
query23	17481	16538	16096	16096
query23_1	16336	16368	16252	16252
query24	7516	1780	1316	1316
query24_1	1330	1313	1337	1313
query25	577	482	414	414
query26	1329	341	175	175
query27	2583	543	359	359
query28	4438	2029	1982	1982
query29	1110	619	517	517
query30	315	229	199	199
query31	1116	1093	967	967
query32	107	64	63	63
query33	534	337	300	300
query34	1181	1149	649	649
query35	752	784	676	676
query36	1385	1402	1236	1236
query37	154	105	92	92
query38	3183	3138	3038	3038
query39	930	912	882	882
query39_1	884	892	864	864
query40	226	124	101	101
query41	63	61	61	61
query42	100	94	93	93
query43	316	321	277	277
query44	
query45	197	187	179	179
query46	1108	1212	721	721
query47	2353	2405	2274	2274
query48	370	427	293	293
query49	639	470	356	356
query50	955	347	251	251
query51	4389	4281	4255	4255
query52	89	92	81	81
query53	245	277	195	195
query54	276	223	213	213
query55	82	77	71	71
query56	260	251	233	233
query57	1452	1449	1349	1349
query58	276	222	219	219
query59	1613	1726	1524	1524
query60	300	260	241	241
query61	166	166	163	163
query62	720	658	573	573
query63	237	188	191	188
query64	2575	814	700	700
query65	
query66	1752	468	336	336
query67	29980	30027	29855	29855
query68	
query69	440	302	273	273
query70	1002	1007	972	972
query71	311	230	213	213
query72	3223	2695	2411	2411
query73	853	770	431	431
query74	5148	4995	4811	4811
query75	2672	2598	2244	2244
query76	2366	1164	779	779
query77	356	378	288	288
query78	12435	12676	11893	11893
query79	1520	1058	775	775
query80	590	480	387	387
query81	463	288	247	247
query82	640	156	119	119
query83	365	282	247	247
query84	
query85	927	558	446	446
query86	499	312	267	267
query87	3451	3347	3240	3240
query88	3782	2792	2756	2756
query89	430	394	340	340
query90	1974	193	182	182
query91	185	166	141	141
query92	65	64	57	57
query93	1466	1458	888	888
query94	565	382	336	336
query95	732	378	448	378
query96	1123	844	352	352
query97	2721	2720	2593	2593
query98	219	215	209	209
query99	1159	1184	1046	1046
Total cold run time: 253055 ms
Total hot run time: 169785 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.86% (28300/38315)
Line Coverage 57.92% (308327/532292)
Region Coverage 54.78% (258320/471577)
Branch Coverage 56.13% (112035/199613)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants