diff --git a/.lycheeignore b/.lycheeignore index 97a8b48e871d5..fdcc00794a2e3 100644 --- a/.lycheeignore +++ b/.lycheeignore @@ -3,6 +3,9 @@ https://github\.com/.*/issues/? https://github\.com/.*/pull/? https://github\.com/.*/pull/[0-9]+ https://github\.com/.*/issues/[0-9]+ +https://github\.com/pingcap/tiflow/blob/master/dm/docs/.* +https://github\.com/\$user/(docs|docs-cn) +https://.*github.*/%7B%7B%7B%20.tidb_operator_version%20%7D%7D%7D https?://\$?\{host}/dashboard.* http://xn--\$?\{ip}-m86ht9t5l1bhz9ayu7b:3000.* http://ip:2379.* @@ -14,7 +17,6 @@ file://.*?http:/\$%7BPD_IP%7D:\$%7BPD_PORT%7D/dashboard.* http://\{grafana-ip\}:3000 http://\{pd-ip\}:2379/dashboard http://localhost:\d+/ -https://github\.com/\$user/(docs|docs-cn) https://linux\.die\.net/man.* https://dev\.mysql\.com/doc/.+/5.7/en/.* https://dev\.mysql\.com/doc/.+/8\.0/en/.* @@ -35,4 +37,4 @@ https://centminmod\.com/mydumper\.html https://docs\.pingcap\.com/tidb/v6\.6/system-variables#tidb_pessimistic_txn_aggressive_locking-new-in-v660 https://docs\.pingcap\.com/tidb/v7\.6/system-variables#tidb_ddl_version-new-in-v760 https://developers\.redhat\.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level -https://.*github.*/%7B%7B%7B%20.tidb_operator_version%20%7D%7D%7D +https://portal\.azure\.com/.* diff --git a/dm/dm-compatibility-catalog.md b/dm/dm-compatibility-catalog.md index c7a28d31ed256..08f6ec77d3bcc 100644 --- a/dm/dm-compatibility-catalog.md +++ b/dm/dm-compatibility-catalog.md @@ -14,18 +14,18 @@ DM supports migrating data from different sources to TiDB clusters. Based on the ## Data sources -| Data source | Compatibility level | Note | -| - | - | - | -| MySQL ≤ 5.5 | Not tested | | -| MySQL 5.6 | GA | | -| MySQL 5.7 | GA | | -| MySQL 8.0 | GA | Does not support binlog transaction compression [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html). | -| MySQL 8.1 ~ 8.3 | Not tested | | -| MySQL 8.4 | Incompatible | For more information, see [DM Issue #11020](https://github.com/pingcap/tiflow/issues/11020). | -| MySQL 9.x | Not tested | | -| MariaDB < 10.1.2 | Incompatible | Incompatible with binlog of the time type. | -| MariaDB 10.1.2 ~ 10.5.10 | Experimental | | -| MariaDB > 10.5.10 | Not tested | Expected to work in most cases after bypassing the [precheck](/dm/dm-precheck.md). See [MariaDB notes](#mariadb-notes). | +| Data source | Compatibility level | Note | +| ------------------------ | ------------------- | ---- | +| MySQL ≤ 5.5 | Not tested | | +| MySQL 5.6 | GA | | +| MySQL 5.7 | GA | | +| MySQL 8.0 | GA | Does not support binlog transaction compression [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html). | +| MySQL 8.1 ~ 8.3 | Not tested | | +| MySQL 8.4 | Incompatible | For more information, see [DM Issue #11020](https://github.com/pingcap/tiflow/issues/11020). | +| MySQL 9.x | Not tested | | +| MariaDB < 10.1.2 | Incompatible | Incompatible with binlog of the time type. | +| MariaDB 10.1.2 ~ 10.5.10 | Experimental | | +| MariaDB > 10.5.10 | Not tested | Expected to work in most cases after bypassing the [precheck](/dm/dm-precheck.md). See [MariaDB notes](#mariadb-notes). | ### Incompatibility with foreign key CASCADE operations diff --git a/dm/dm-precheck.md b/dm/dm-precheck.md index 48b41f16a6e09..280a7cacecfab 100644 --- a/dm/dm-precheck.md +++ b/dm/dm-precheck.md @@ -52,7 +52,7 @@ Regardless of the migration mode you choose, the precheck always includes the fo - Compatibility of the upstream MySQL table schema - - Check whether the upstream tables have foreign keys, which are not supported by TiDB. A warning is returned if a foreign key is found in the precheck. + - Check whether the upstream tables have foreign keys, which are not fully supported by TiDB. A warning is returned if a foreign key is found in the precheck. - Check whether the upstream tables use character sets that are incompatible with TiDB. For more information, see [TiDB Supported Character Sets](/character-set-and-collation.md). - Check whether the upstream tables have primary key constraints or unique key constraints (introduced from v1.0.7). @@ -132,6 +132,7 @@ For the incremental data migration mode (`task-mode: incremental`), in addition - Check whether binlog is enabled (required by DM). - Check whether `binlog_format=ROW` is configured (DM only supports the migration of binlog in the ROW format). - Check whether `binlog_row_image=FULL` is configured (DM only supports `binlog_row_image=FULL`). + - Check whether `binlog_transaction_compression=OFF` is configured (DM does not support transaction compression). - If `binlog_do_db` or `binlog_ignore_db` is configured, check whether the database tables to be migrated meet the conditions of `binlog_do_db` and `binlog_ignore_db`. * (Mandatory) Check if the upstream database is in an [Online-DDL](/dm/feature-online-ddl.md) process (in which the `ghost` table is created but the `rename` phase is not executed yet). If the upstream is in the online-DDL process, the precheck returns an error. In this case, wait until the DDL to complete and retry. @@ -144,21 +145,21 @@ For the full and incremental data migration mode (`task-mode: all`), in addition Prechecks can find potential risks in your environments. It is not recommended to ignore check items. If your data migration task has special needs, you can use the [`ignore-checking-items` configuration item](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) to skip some check items. -| Check item | Description | -| :---------- | :------------ | -| `dump_privilege` | Checks the dump privilege of the user in the upstream MySQL instance. | -| `replication_privilege` | Checks the replication privilege of the user in the upstream MySQL instance. | -| `version` | Checks the version of the upstream database. | -| `server_id` | Checks whether server_id is configured in the upstream database. | -| `binlog_enable` | Checks whether binlog is enabled in the upstream database. | -| `table_schema` | Checks the compatibility of the table schemas in the upstream MySQL tables. | -| `schema_of_shard_tables`| Checks the consistency of the table schemas in the upstream MySQL multi-instance shards. | -| `auto_increment_ID` | Checks whether the auto-increment primary key conflicts in the upstream MySQL multi-instance shards. | -|`online_ddl`| Checks whether the upstream is in the process of [online-DDL](/dm/feature-online-ddl.md). | -| `empty_region` | Checks the number of empty Regions in the downstream database for physical import. | -| `region_distribution` | Checks the distribution of Regions in the downstream database for physical import. | -| `downstream_version` | Checks the versions of TiDB, PD, and TiKV in the downstream database. | -| `free_space` | Checks the free space of the downstream database. | +| Check item | Description | +| :-------------------------- | :------------ | +| `dump_privilege` | Checks the dump privilege of the user in the upstream MySQL instance. | +| `replication_privilege` | Checks the replication privilege of the user in the upstream MySQL instance. | +| `version` | Checks the version of the upstream database. | +| `server_id` | Checks whether server_id is configured in the upstream database. | +| `binlog_enable` | Checks whether binlog is enabled in the upstream database. | +| `table_schema` | Checks the compatibility of the table schemas in the upstream MySQL tables. | +| `schema_of_shard_tables` | Checks the consistency of the table schemas in the upstream MySQL multi-instance shards. | +| `auto_increment_ID` | Checks whether the auto-increment primary key conflicts in the upstream MySQL multi-instance shards. | +| `online_ddl` | Checks whether the upstream is in the process of [online-DDL](/dm/feature-online-ddl.md). | +| `empty_region` | Checks the number of empty Regions in the downstream database for physical import. | +| `region_distribution` | Checks the distribution of Regions in the downstream database for physical import. | +| `downstream_version` | Checks the versions of TiDB, PD, and TiKV in the downstream database. | +| `free_space` | Checks the free space of the downstream database. | | `downstream_mutex_features` | Checks whether the downstream database is running tasks that are incompatible with physical import. | > **Note:** diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index d708402cb6502..0a6be421edf23 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -127,12 +127,12 @@ For {{{ .essential }}}, the Data Migration feature supports the following data s | Data source | Supported versions | |:-------------------------------------------------|:-------------------| -| Self-managed MySQL (on-premises or public cloud) | 8.0, 5.7 | -| Amazon Aurora MySQL | 8.0, 5.7 | +| Self-managed MySQL (on-premises or public cloud) | 8.0, 5.7 | +| Amazon Aurora MySQL | 8.0, 5.7 | | Amazon RDS MySQL | 8.0, 5.7 | | Alibaba Cloud RDS MySQL | 8.0, 5.7 | -| Azure Database for MySQL - Flexible Server | 8.0, 5.7 | -| Google Cloud SQL for MySQL | 8.0, 5.7 | +| Azure Database for MySQL - Flexible Server | 8.0, 5.7 | +| Google Cloud SQL for MySQL | 8.0, 5.7 | @@ -140,12 +140,13 @@ For {{{ .essential }}}, the Data Migration feature supports the following data s To continuously replicate incremental changes from the source MySQL-compatible database to the TiDB Cloud target cluster using DM, you need the following configurations to enable binary logs in the source database: -| Configuration | Required value | Why | -|:--------------|:---------------|:----| -| `log_bin` | `ON` | Enables binary logging, which DM uses to replicate changes to TiDB | -| `binlog_format` | `ROW` | Captures all data changes accurately (other formats miss edge cases) | -| `binlog_row_image` | `FULL` | Includes all column values in events for safe conflict resolution | -| `binlog_expire_logs_seconds` | ≥ `86400` (1 day), `604800` (7 days, recommended) | Ensures DM can access consecutive logs during migration | +| Configuration | Required value | Why | +|:---------------------------------|:---------------|:----| +| `log_bin` | `ON` | Enables binary logging, which DM uses to replicate changes to TiDB | +| `binlog_format` | `ROW` | Captures all data changes accurately (other formats miss edge cases) | +| `binlog_row_image` | `FULL` | Includes all column values in events for safe conflict resolution | +| `binlog_expire_logs_seconds` | ≥ `86400` (1 day), `604800` (7 days, recommended) | Ensures DM can access consecutive logs during migration | +| `binlog_transaction_compression` | `OFF` | DM does not support transaction compression | #### Check current values and configure the source MySQL instance @@ -154,7 +155,7 @@ To check the current configurations, connect to the source MySQL instance and ex ```sql SHOW VARIABLES WHERE Variable_name IN ('log_bin','server_id','binlog_format','binlog_row_image', -'binlog_expire_logs_seconds','expire_logs_days'); +'binlog_expire_logs_seconds','expire_logs_days','binlog_transaction_compression'); ``` If necessary, change the source MySQL instance configurations to match the required values. @@ -170,6 +171,7 @@ If necessary, change the source MySQL instance configurations to match the requi binlog_format = ROW binlog_row_image = FULL binlog_expire_logs_seconds = 604800 # 7 days retention + binlog_transaction_compression = OFF ``` 2. Restart the MySQL service to apply the changes: