[Bug] routine load 在非严格模式下导入 varchar 目标列时，非英文字符的长度截断不正确

[123shang60]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris-4.1.0-rc03-5960d4cea0e

What's Wrong?

我在 Doris 中创建了如下的表

CREATE TABLE test_table
(
    `env` VARCHAR(32) NOT NULL DEFAULT ''
)
ENGINE = OLAP
DUPLICATE KEY(env)
DISTRIBUTED BY RANDOM BUCKETS 1
properties(
    "compression" = "zstd",
    "enable_single_replica_compaction" = "true",
    "replication_allocation" = "tag.location.default: 1",
    "storage_format" = "V3"
);

同时开启了一个 routine load 任务

CREATE ROUTINE LOAD dwd.test_table ON test_table
COLUMNS(`env`)
PROPERTIES(
    "format"="json",
    "strict_mode"="false",
    "max_filter_ratio" = "1",
    "timezone" = "Asia/Shanghai",
    "max_error_number" = "5000000",
    "max_batch_interval" = "10",
    "max_batch_rows" = "5000000",
    "desired_concurrent_number" = "8",
    "load_to_single_tablet" = "true",
    "exec_mem_limit" = "8589934592"
)
FROM KAFKA(
    "kafka_broker_list" = "127.0.0.1:9092",
    "kafka_topic" = "test_load",
    "property.kafka_default_offsets" = "OFFSET_END",
    "property.group.id" = "test_load_doris"
);

之后，我向 kafka 中发送了一个消息：

{"env":"${jnd${upper:ı}:ldap://test.comxxxxxx}"}

这个 json 中， upper: 字符后有一个特殊的 U+0131 字符；此时 Doris 的 routine load 任务会出现如下报错：

Reason: column_name[env], the length of input is too long than schema. first 32 bytes of input str: [${jnd${upper:ı}:ldap://test.com] schema length: 32; actual length: 33; . src line []; 

同样的，导入如下的中文字符，也会出现问题：

{"env":"中123456789012345678901234567890"}

报错如下：

Reason: column_name[env], the length of input is too long than schema. first 32 bytes of input str: [中12345678901234567890123456789] schema length: 32; actual length: 33; . src line []; 

What You Expected?

能够正确的按照 varchar 定义的长度截断字符串，并成功导入数据

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_ee38da6ce046

Initial code-path analysis:

This looks like a real BE-side bug in the non-strict CHAR/VARCHAR truncation path, not a Kafka or JSON parsing issue.

The reported build hash maps to 4.1.0-rc03. In routine load, strict_mode=false is propagated into enable_insert_strict=false in be/src/load/routine_load/routine_load_task_executor.cpp, so the non-strict truncation branch in be/src/exec/sink/vtablet_block_convertor.cpp is expected to run.

The problematic path is:

1. OlapTableBlockConvertor::_internal_validate_column() checks the byte length of string columns against the schema length.
2. In non-strict mode, it tries to truncate over-length string values by executing substring(value, 1, schema_len).
3. The same function then validates the result again by byte length.

The mismatch is that the schema length is treated as bytes, while substring counts UTF-8 characters on the non-ASCII path. This is already hinted by the comment in vtablet_block_convertor.cpp: schema length is byte-based, while substring works by character units.

That explains both examples:

• 中123456789012345678901234567890 is 31 characters but 33 UTF-8 bytes. substring(..., 32) keeps the whole value, so the later byte-length check still sees 33 bytes for VARCHAR(32) and rejects it.
• The U+0131 example similarly has a 32-character prefix that is 33 UTF-8 bytes, so it fails after the attempted truncation.

So the current behavior is internally consistent with the code, but it is not the intended non-strict load behavior. strict_mode=false cannot work around it because the row is still rechecked after the character-based truncation.

Suggested next steps:

1. Fix the non-strict load truncation in OlapTableBlockConvertor to trim to a valid UTF-8 prefix whose byte length is no larger than the target CHAR/VARCHAR byte length, instead of using character-count substring.
2. Cover both TYPE_VARCHAR and TYPE_CHAR, since they share this validation branch.
3. Add a regression case with a non-strict load into VARCHAR(32) using both a 2-byte character such as U+0131 and a 3-byte Chinese character. A focused BE test around OlapTableBlockConvertor would catch the exact logic; an end-to-end stream/routine load regression would also be useful.

Temporary workaround before a fix: ensure the producer sends values already truncated to a UTF-8-safe 32-byte prefix, or increase the target VARCHAR length. Raising max_filter_ratio or keeping strict_mode=false will not make this specific row load successfully because the post-truncation byte-length validation still rejects it.