Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: load data test performance down #21294

Open
1 task done
heni02 opened this issue Jan 20, 2025 · 2 comments
Open
1 task done

[Bug]: load data test performance down #21294

heni02 opened this issue Jan 20, 2025 · 2 comments
Assignees
Labels
kind/bug Something isn't working phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone

Comments

@heni02
Copy link
Contributor

heni02 commented Jan 20, 2025

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

main

Commit ID

fd9d868

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

load data 性能数据:https://grafana.ci.matrixorigin.cn/d/U7ZX6G6Vk/test-nightly-tke-loaddata?orgId=1,main分支有性能下降的现象,
tpch1T load时由之前1个小时耗时增长到2个小时+
Image

二分结果:
Image

Expected Behavior

No response

Steps to Reproduce

create table table_20_columns(
clo1 tinyint,
clo2 smallint,
clo3 int,
clo4 bigint,
clo5 tinyint unsigned,
clo6 smallint unsigned,
clo7 int unsigned,
clo8 bigint unsigned,
col9 float,
col10 double,
col11 varchar(255),
col12 Date,
col13 DateTime,
col14 timestamp,
col15 bool,
col16 decimal(5,2),
col17 text,
col18 varchar(255),
col19 varchar(255),
col20 varchar(255)
);

load data url s3option {'endpoint'='http://cos.ap-guangzhou.myqcloud.com','access_key_id'='xxxx','secret_access_key'='xxx','bucket'='mo-load-guangzhou-1308875761', 'filepath'='100_columns/100000000_100_columns_load_data.csv', 'compression'='none'} into table test.table_100_columns fields terminated by ',' lines terminated by '\n' parallel 'true';

Additional information

No response

@heni02 heni02 added kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use labels Jan 20, 2025
@heni02 heni02 added this to the 2.1.0 milestone Jan 20, 2025
@ouyuanning
Copy link
Contributor

暂定不回退。会先调整一下刷s3策略,找一个平衡点吧。

如果完全回退,容易碰到很少数据也刷s3,导致小s3文件过多,后面合并压力过大的问题。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Development

No branches or pull requests

3 participants