Skip to content

Support target database in write-back sink#18058

Open
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:writeback-sink-target-database
Open

Support target database in write-back sink#18058
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:writeback-sink-target-database

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Changes

  • Add sink.database / connector.database support for write-back sink data writes.
  • Validate write-back sink database names on the sink side instead of relying on generic path validation.
  • Use table-model create-database validation for table-model database names.
  • Use tree-model create-database validation semantics for tree-model database names, including root. requirement, legal tree path parsing, database-name pattern check, and length limit.
  • Support multi-level tree-model target databases such as root.target.db.
  • Reject table-model writes when the event or target mapping produces an invalid table-model database, for example tree database root.a.b mapping to table database a.b.
  • Rewrite target database for both table-model and tree-model insertion statements.
  • Add UT coverage for target database validation boundaries, target database customization/mapping, raw tablet event rejection, and statement event rejection.
  • Update tree-model write-back sink IT coverage to use a multi-level target database.

Semantics

  • sink.database=target is treated as a table-model database and mapped to tree database root.target for tree writes.
  • sink.database=root.target and sink.database=root.target.db are treated as tree-model databases.
  • A multi-level tree-model database such as root.target.db is valid for tree-model writes.
  • The corresponding table-model database name for root.target.db is target.db, which is invalid in table model and must reject table-model events.
  • Table-model events always validate their event database name with table-model create-database rules before applying a configured target database, so source values like root.a.b / a.b are rejected.
  • Tree-model events always validate their event database name with tree-model create-database rules before applying a configured target database.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode
  • mvn spotless:apply -pl integration-test -P with-integration-tests
  • git diff --check
  • mvn -pl iotdb-core/datanode -Dtest=PipeSinkTest test was attempted, but datanode compilation fails before test execution in existing generated Freemarker sources unrelated to this change, including missing IFill, IFillFilter, LinearFill, Accumulator, org.apache.commons.collections4.comparators, and IoTDBConfig#getModeMapSizeThreshold().

Comment on lines +204 to +207
String.format(
"The table-model database %s is invalid. It should not contain '%s', should match "
+ "the pattern %s, and the length should not exceed %d",
databaseName, PATH_SEPARATOR, IoTDBConfig.DATABASE_PATTERN, MAX_DATABASE_NAME_LENGTH),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i18n

Comment on lines +333 to +350
if (PathUtils.isTableModelDatabase(targetDatabase)) {
targetTableModelDatabaseName = targetDatabase.toLowerCase(Locale.ENGLISH);
targetTreeModelDatabaseName =
validateAndNormalizeTreeModelDatabaseName(
PathUtils.qualifyDatabaseName(targetTableModelDatabaseName));
return;
}

targetTreeModelDatabaseName = validateAndNormalizeTreeModelDatabaseName(targetDatabase);
final String tableModelDatabaseName =
PathUtils.unQualifyDatabaseName(targetTreeModelDatabaseName).toLowerCase(Locale.ENGLISH);
try {
TableConfigTaskVisitor.validateDatabaseName(tableModelDatabaseName);
targetTableModelDatabaseName = tableModelDatabaseName;
} catch (final Exception e) {
invalidTargetTableModelDatabaseName = tableModelDatabaseName;
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are tree db and table db set in both branches?

Comment on lines +665 to +667
if (!startsWith(deviceNodes, sourceDatabaseNodes)) {
return devicePath;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When will this happen?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants