Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .lycheeignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ https://github\.com/.*/issues/?
https://github\.com/.*/pull/?
https://github\.com/.*/pull/[0-9]+
https://github\.com/.*/issues/[0-9]+
https://github\.com/pingcap/tiflow/blob/master/dm/docs/.*
https://github\.com/\$user/(docs|docs-cn)
https://.*github.*/%7B%7B%7B%20.tidb_operator_version%20%7D%7D%7D
https?://\$?\{host}/dashboard.*
http://xn--\$?\{ip}-m86ht9t5l1bhz9ayu7b:3000.*
http://ip:2379.*
Expand All @@ -14,7 +17,6 @@ file://.*?http:/\$%7BPD_IP%7D:\$%7BPD_PORT%7D/dashboard.*
http://\{grafana-ip\}:3000
http://\{pd-ip\}:2379/dashboard
http://localhost:\d+/
https://github\.com/\$user/(docs|docs-cn)
https://linux\.die\.net/man.*
https://dev\.mysql\.com/doc/.+/5.7/en/.*
https://dev\.mysql\.com/doc/.+/8\.0/en/.*
Expand All @@ -35,4 +37,4 @@ https://centminmod\.com/mydumper\.html
https://docs\.pingcap\.com/tidb/v6\.6/system-variables#tidb_pessimistic_txn_aggressive_locking-new-in-v660
https://docs\.pingcap\.com/tidb/v7\.6/system-variables#tidb_ddl_version-new-in-v760
https://developers\.redhat\.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level
https://.*github.*/%7B%7B%7B%20.tidb_operator_version%20%7D%7D%7D
https://portal\.azure\.com/.*
24 changes: 12 additions & 12 deletions dm/dm-compatibility-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,18 @@ DM supports migrating data from different sources to TiDB clusters. Based on the

## Data sources

| Data source | Compatibility level | Note |
| - | - | - |
| MySQL ≤ 5.5 | Not tested | |
| MySQL 5.6 | GA | |
| MySQL 5.7 | GA | |
| MySQL 8.0 | GA | Does not support binlog transaction compression [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html). |
| MySQL 8.1 ~ 8.3 | Not tested | |
| MySQL 8.4 | Incompatible | For more information, see [DM Issue #11020](https://github.com/pingcap/tiflow/issues/11020). |
| MySQL 9.x | Not tested | |
| MariaDB < 10.1.2 | Incompatible | Incompatible with binlog of the time type. |
| MariaDB 10.1.2 ~ 10.5.10 | Experimental | |
| MariaDB > 10.5.10 | Not tested | Expected to work in most cases after bypassing the [precheck](/dm/dm-precheck.md). See [MariaDB notes](#mariadb-notes). |
| Data source | Compatibility level | Note |
| ------------------------ | ------------------- | ---- |
| MySQL ≤ 5.5 | Not tested | |
| MySQL 5.6 | GA | |
| MySQL 5.7 | GA | |
| MySQL 8.0 | GA | Does not support binlog transaction compression [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html). |
| MySQL 8.1 ~ 8.3 | Not tested | |
| MySQL 8.4 | Incompatible | For more information, see [DM Issue #11020](https://github.com/pingcap/tiflow/issues/11020). |
| MySQL 9.x | Not tested | |
| MariaDB < 10.1.2 | Incompatible | Incompatible with binlog of the time type. |
| MariaDB 10.1.2 ~ 10.5.10 | Experimental | |
| MariaDB > 10.5.10 | Not tested | Expected to work in most cases after bypassing the [precheck](/dm/dm-precheck.md). See [MariaDB notes](#mariadb-notes). |

### Incompatibility with foreign key CASCADE operations

Expand Down
33 changes: 17 additions & 16 deletions dm/dm-precheck.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Regardless of the migration mode you choose, the precheck always includes the fo

- Compatibility of the upstream MySQL table schema

- Check whether the upstream tables have foreign keys, which are not supported by TiDB. A warning is returned if a foreign key is found in the precheck.
- Check whether the upstream tables have foreign keys, which are not fully supported by TiDB. A warning is returned if a foreign key is found in the precheck.
- Check whether the upstream tables use character sets that are incompatible with TiDB. For more information, see [TiDB Supported Character Sets](/character-set-and-collation.md).
- Check whether the upstream tables have primary key constraints or unique key constraints (introduced from v1.0.7).

Expand Down Expand Up @@ -132,6 +132,7 @@ For the incremental data migration mode (`task-mode: incremental`), in addition
- Check whether binlog is enabled (required by DM).
- Check whether `binlog_format=ROW` is configured (DM only supports the migration of binlog in the ROW format).
- Check whether `binlog_row_image=FULL` is configured (DM only supports `binlog_row_image=FULL`).
- Check whether `binlog_transaction_compression=OFF` is configured (DM does not support transaction compression).
- If `binlog_do_db` or `binlog_ignore_db` is configured, check whether the database tables to be migrated meet the conditions of `binlog_do_db` and `binlog_ignore_db`.

* (Mandatory) Check if the upstream database is in an [Online-DDL](/dm/feature-online-ddl.md) process (in which the `ghost` table is created but the `rename` phase is not executed yet). If the upstream is in the online-DDL process, the precheck returns an error. In this case, wait until the DDL to complete and retry.
Expand All @@ -144,21 +145,21 @@ For the full and incremental data migration mode (`task-mode: all`), in addition

Prechecks can find potential risks in your environments. It is not recommended to ignore check items. If your data migration task has special needs, you can use the [`ignore-checking-items` configuration item](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) to skip some check items.

| Check item | Description |
| :---------- | :------------ |
| `dump_privilege` | Checks the dump privilege of the user in the upstream MySQL instance. |
| `replication_privilege` | Checks the replication privilege of the user in the upstream MySQL instance. |
| `version` | Checks the version of the upstream database. |
| `server_id` | Checks whether server_id is configured in the upstream database. |
| `binlog_enable` | Checks whether binlog is enabled in the upstream database. |
| `table_schema` | Checks the compatibility of the table schemas in the upstream MySQL tables. |
| `schema_of_shard_tables`| Checks the consistency of the table schemas in the upstream MySQL multi-instance shards. |
| `auto_increment_ID` | Checks whether the auto-increment primary key conflicts in the upstream MySQL multi-instance shards. |
|`online_ddl`| Checks whether the upstream is in the process of [online-DDL](/dm/feature-online-ddl.md). |
| `empty_region` | Checks the number of empty Regions in the downstream database for physical import. |
| `region_distribution` | Checks the distribution of Regions in the downstream database for physical import. |
| `downstream_version` | Checks the versions of TiDB, PD, and TiKV in the downstream database. |
| `free_space` | Checks the free space of the downstream database. |
| Check item | Description |
| :-------------------------- | :------------ |
| `dump_privilege` | Checks the dump privilege of the user in the upstream MySQL instance. |
| `replication_privilege` | Checks the replication privilege of the user in the upstream MySQL instance. |
| `version` | Checks the version of the upstream database. |
| `server_id` | Checks whether server_id is configured in the upstream database. |
| `binlog_enable` | Checks whether binlog is enabled in the upstream database. |
| `table_schema` | Checks the compatibility of the table schemas in the upstream MySQL tables. |
| `schema_of_shard_tables` | Checks the consistency of the table schemas in the upstream MySQL multi-instance shards. |
| `auto_increment_ID` | Checks whether the auto-increment primary key conflicts in the upstream MySQL multi-instance shards. |
| `online_ddl` | Checks whether the upstream is in the process of [online-DDL](/dm/feature-online-ddl.md). |
| `empty_region` | Checks the number of empty Regions in the downstream database for physical import. |
| `region_distribution` | Checks the distribution of Regions in the downstream database for physical import. |
| `downstream_version` | Checks the versions of TiDB, PD, and TiKV in the downstream database. |
| `free_space` | Checks the free space of the downstream database. |
| `downstream_mutex_features` | Checks whether the downstream database is running tasks that are incompatible with physical import. |

> **Note:**
Expand Down
24 changes: 13 additions & 11 deletions tidb-cloud/migrate-from-mysql-using-data-migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,25 +127,26 @@ For {{{ .essential }}}, the Data Migration feature supports the following data s

| Data source | Supported versions |
|:-------------------------------------------------|:-------------------|
| Self-managed MySQL (on-premises or public cloud) | 8.0, 5.7 |
| Amazon Aurora MySQL | 8.0, 5.7 |
| Self-managed MySQL (on-premises or public cloud) | 8.0, 5.7 |
| Amazon Aurora MySQL | 8.0, 5.7 |
| Amazon RDS MySQL | 8.0, 5.7 |
| Alibaba Cloud RDS MySQL | 8.0, 5.7 |
| Azure Database for MySQL - Flexible Server | 8.0, 5.7 |
| Google Cloud SQL for MySQL | 8.0, 5.7 |
| Azure Database for MySQL - Flexible Server | 8.0, 5.7 |
| Google Cloud SQL for MySQL | 8.0, 5.7 |

</CustomContent>

### Enable binary logs in the source MySQL-compatible database for replication

To continuously replicate incremental changes from the source MySQL-compatible database to the TiDB Cloud target cluster using DM, you need the following configurations to enable binary logs in the source database:

| Configuration | Required value | Why |
|:--------------|:---------------|:----|
| `log_bin` | `ON` | Enables binary logging, which DM uses to replicate changes to TiDB |
| `binlog_format` | `ROW` | Captures all data changes accurately (other formats miss edge cases) |
| `binlog_row_image` | `FULL` | Includes all column values in events for safe conflict resolution |
| `binlog_expire_logs_seconds` | ≥ `86400` (1 day), `604800` (7 days, recommended) | Ensures DM can access consecutive logs during migration |
| Configuration | Required value | Why |
|:---------------------------------|:---------------|:----|
| `log_bin` | `ON` | Enables binary logging, which DM uses to replicate changes to TiDB |
| `binlog_format` | `ROW` | Captures all data changes accurately (other formats miss edge cases) |
| `binlog_row_image` | `FULL` | Includes all column values in events for safe conflict resolution |
| `binlog_expire_logs_seconds` | ≥ `86400` (1 day), `604800` (7 days, recommended) | Ensures DM can access consecutive logs during migration |
| `binlog_transaction_compression` | `OFF` | DM does not support transaction compression |

#### Check current values and configure the source MySQL instance

Expand All @@ -154,7 +155,7 @@ To check the current configurations, connect to the source MySQL instance and ex
```sql
SHOW VARIABLES WHERE Variable_name IN
('log_bin','server_id','binlog_format','binlog_row_image',
'binlog_expire_logs_seconds','expire_logs_days');
'binlog_expire_logs_seconds','expire_logs_days','binlog_transaction_compression');
```

If necessary, change the source MySQL instance configurations to match the required values.
Expand All @@ -170,6 +171,7 @@ If necessary, change the source MySQL instance configurations to match the requi
binlog_format = ROW
binlog_row_image = FULL
binlog_expire_logs_seconds = 604800 # 7 days retention
binlog_transaction_compression = OFF
```

2. Restart the MySQL service to apply the changes:
Expand Down