릴리스 노트
Synchronization Instructions
포커스 모드
폰트 크기
Scenarios
This document introduces the process of using Data Transfer Service (DTS) to synchronize data from MongoDB to TencentDB for MongoDB.
Preparations
1. Read the Usage Instructions carefully to learn about feature constraints and must-knows.
2. Establish the access channel between DTS and the database in advance according to the access type you want to use. For details, see Network Preparations.
IDC self-built databases/Databases of other cloud vendors: The access method can be set to Public Network, Direct Connect (DC), VPN Access, and Cloud Connect Network (CCN).
Self-built databases on Cloud Virtual Machine (CVM) instances: The access method can be set to Self-Built on CVM.
Tencent Cloud database instances: The access method can be set to Cloud Database.
3. It is recommended to create a read-only account in the source database for synchronization. The reference method for creating a read-only account is as follows.
# Syntax example when the source database is a replica set or a single node.use admindb.createUser({user: "username",pwd: "password",roles:[{role: "readAnyDatabase", db: "admin"},{role: "read", db: "local"}]})# Syntax example when the source database is a sharded cluster.use admindb.createUser({user: "username",pwd: "password",roles:[{role: "readAnyDatabase", db: "admin"},{role: "read", db: "local"},{role: "read", db: "config"}]})
4. When the target database is a Tencent Cloud database instance, you can use mongouser for synchronization or create an account yourself. The reference method for creating your own account is as follows.
db.createUser({user:"username",pwd:"password",roles:[{role:"readWriteAnyDatabase",db:"admin"}]})
Operation Steps
Step 1: Creating a Synchronization Task
1. Log in to the DTS console, choose Data Sync in the left-side bar, and then click Create Sync Task to go to the synchronization task creation page.
2. On the synchronization task creation page, select the source instance type and region as well as the target instance type, region, and specification, and then click Buy Now.
Parameter | Description |
Creating Mode | New task: Create a task. Create a similar task: Quickly create a task with the same configuration as the historical task. The default pre-filled options, such as the database type and billing mode, of the new task are consistent with those of the historical task. Users can also modify them as needed. |
Billing Mode | Yearly/Monthly subscription and pay-as-you-go modes are supported. |
Source Instance Type | Select MongoDB. It cannot be modified after purchase. |
Source Instance Region | Select the source instance region. It cannot be modified after purchase. |
Target Instance Type | Select MongoDB. It cannot be modified after purchase. |
Target Instance Region | Select the target instance region. It cannot be modified after purchase. |
Specification | Select specifications according to your business requirements. Higher specifications deliver better performance. For details, see Billing Overview. |
Tag | Set tag keys and values for synchronization tasks to facilitate efficient management of multiple tasks through tags. |
Task Name | Set it to name the task after creation. The task name is the same as the task ID by default. After the synchronization task is created, the task name can be reset. Set it to name the task now. Enter the task name in the input box below. |
Terms of Service |
3. After the purchase is completed, return to the data synchronization task list to see the newly created data synchronization task. It requires configuration before usage.
Step 2: Setting the Synchronization Source and the Target Database
1. In the data synchronization task list, click Configure in the Operation column to go to the synchronization task configuration page.
2. On the Set source and target databases tab, enter the information about source and target databases, and then click Test Connectivity. After connectivity testing, click Next.
Task Configuration
Parameter | Description |
Task Name | A task name is automatically generated by DTS. It is recommended that users modify it to a meaningful name to facilitate task identification. |
Running Mode | Immediate execution: The task is started immediately after the pre-check passes. Scheduled execution: Set a start time for the task execution. After the pre-check passes, the task is not started immediately, but is started at the scheduled time. |
Source Database Settings
Parameter | Description |
Source Instance Type | Source instance type selected during purchase. It cannot be modified. |
Source Instance Region | Source instance region selected during purchase. It cannot be modified. |
Access Type | Select the access type according to your scenario. For preparations related to different access types, see Network Preparation Overview. If the source database is an IDC self-built database or a database of other cloud vendors, you can set the access type to Public Network, DC, VPN Access, or CCN. Public network: The source database is accessible through a public IP address. DC: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC. VPN access: The source database can connect to Tencent Cloud VPC through VPN access. CCN: The source database can connect to Tencent Cloud VPC through CCN. VPC: Both the source and target databases are deployed on Tencent Cloud with VPC configured. When the source database is a self-built database on the CVM instance, set the access type to Self-Built on CVM. When the source database is a Tencent Cloud database instance, set the access type to "Cloud Database". |
Architecture | This parameter is displayed when the Access Type is set to Public Network/Self-Built on CVM/DC/VPN Access/CCN/VPC. Select an access type according to the actual situation. Replica set: refers to an architecture in which the source database is a replica set, composed of one primary node and one or more secondary nodes. Cluster migration: refers to an architecture in which the source database is a sharded cluster, composed of components such as mongos nodes, config servers, and shard nodes. Single node: refers to an architecture in which the source database cluster is a single node that performs read and write operations. Note: After an architecture type is selected and the connectivity test is passed, changing to a different architecture type is not allowed; otherwise, an error is reported for the task. |
Migration Method | When the Architecture is set to Cluster Migration and the migration method is set to oplog, the following parameters are required to be configured. Node - mongod: Enter the IP address and port, or the domain name and port of the mongod node. For multiple nodes, enter each on a new line. Specifying only one mongod node per shard is required. Example: 1xx.xx.55.77:6xx9. Node - mongos: Enter the IP address and port, or the domain name and port of the mongos node. Node - Config Server: Enter the IP address and port, or the domain name and port of the Config Server node. When the Architecture is set to Cluster Migration and the migration method is set to Change Stream, the following parameters are required to be configured. Connection string type: Regular or SRV connection strings are supported. Node - mongos: Enter the IP address and port, or the domain name and port of the mongos node. Note: The following DDL events are supported in the Change Stream migration method: createIndexes, dropIndexes, modify, drop, create, shardCollection, reshardCollection, and refineCollectionShardKey Among them, createIndexes, dropIndex, create, and modify are supported only in 6.0 and later versions; refineCollectionShardKey is supported only in 6.10 and later versions. |
Public Network | When the Access Type is set to "public network", the following parameters are required to be configured. Host address: IP address or domain name of the source database. Port: port used by the source database. |
Self-Built on CVM | When the Access Type is set to Self-Built on CVM, the following parameters are required to be configured. CVM instance: CVM instance ID. Port: port used by the source database. |
Direct Connect | When the Access Type is set to "Direct Connect", the following parameters are required to be configured. VPC Direct Connect gateway: Only VPC Direct Connect gateway is supported for DC. Confirm the network type associated with the gateway. VPC: Select a VPC network and a subnet. Host address: IP address or domain name of the source database. Port: port used by the source database. |
VPN Access | When the Access Type is set to VPN Access, the following parameters are required to be configured. VPN gateway: Select a VPN gateway ID. VPC: Select the VPC network and the subnet associated with the VPN gateway. Host address: IP address or domain name of the source database. Port: port used by the source database. |
Database | When the Access Type is set to "Database", the following parameters are required to be configured. Whether cross-account synchronization is required: Same account: The Tencent Cloud root account for both the source and target database instances is the same. Cross-account: The Tencent Cloud root accounts for the source and target database instances are different. For guidance on cross-account synchronization operations, see Synchronization Between Cross-Account Cloud Database Instances. Cloud database instance: instance ID of the source database. |
CCN | When the Access Type is set to "CCN", the following parameters are required to be configured. Both CCN access under the same account and access across accounts are supported. Host network environment: Select the network environment according to the actual situation. For example, if the source database is a Tencent Cloud database instance, select "Tencent Cloud"; if the source database is an IDC self-built database, select "Self-built IDC"; if the source database is a database of other cloud vendors, select the corresponding network. Host address: IP address or domain name of the source database. Port: port used by the source database. CCN instance account My account: The Tencent Cloud root account for both the CCN resources and the target database is the same. Other accounts: The Tencent Cloud root accounts for CCN resources and for the target database are different. VPC CCN: CCN instance name. VPC network and subnet associated with CCN: "CCN-associated VPC" refers to the VPC network that is connected to the synchronization link within CCN. Select VPC networks associated with CCN, excluding the VPC network to which the source database belongs. VPC region: The VPC access region is required to be consistent with the source database region selected during task purchase. If they are inconsistent, the selected source database region will be changed to the VPC access region automatically. |
VPC | When the Access Type is set to "VPC", the following parameters are required to be configured. VPC: Select the VPC network and the subnet associated with the VPN gateway. Host address: IP address or domain name of the source database. Port: port used by the source database. |
Authentication Required | Whether security authentication is required for the username and password of the source database. If Yes is selected, the following parameters are required. Database authentication: Name of the database that requires authentication, which is the name of the database to which the account executing the synchronization task belongs. Only "admin" is supported. Authentication mechanism: Currently, only SCRAM-SHA-1 is supported. Account and password selection Same account and password: Select this parameter if the mongod, mongos, and Config Server roles use the same account and password. Enter the unified account and password. Different accounts and passwords: Select this parameter if the mongod, mongos, and Config Server roles use different accounts and passwords. Enter the usernames and passwords for each role respectively. |
Account | Account of the source database. |
Password | Password of the source database account. |
Connection method | Secure Sockets Layer (SSL) connection refers to the use of SSL to establish a secure connection between DTS and the database, encrypting the transmission link. Selecting the SSL connection may increase the database connection response time. Generally, the private network link of Tencent Cloud is relatively secure, and the SSL connection is not required to be enabled. However, for scenarios where transmission methods such as public networks and DC are used, and data security requirements are high, the SSL connection is required to be enabled. Before the SSL connection is selected, SSL encryption is required to be enabled for the database first. |
Target Database Settings
The parameter settings of the target database are similar to those of the source database. The access type should be selected according to the actual situation, which is not repeated here.
Step 3: Setting the Synchronization Option and the Synchronization Object
On the Set sync options and objects tab, set the data initialization, data synchronization, and synchronization object options. After the settings are completed, click Save and Go Next.
Note:
If the user plans to perform a rename operation on a table during synchronization (for example, renaming table A to table B), the synchronization object is required to include the entire database (or the entire instance) where table A is located. If only table A is selected, data from table B will not be synchronized to the target database after the rename operation.
Settings | Parameter | Description |
Data Initialization Option | Initialization Type | Full data initialization: When a synchronization task is executed, the data of the source instance is initialized to the target instance first. |
| If Target Already Exists | Precheck and error reporting: If tables with the same name exist, an error will be reported, and the process will be stopped. Ignore and execute: Full and incremental data are directly appended to the tables in the target instance. |
Data Sync Option | Primary Key Conflict Resolution | Conflict overwriting: When table primary key conflicts are detected during synchronization, the primary key record of the source database is used to overwrite that of the target database. |
| SQL Type | Supported operations include Insert, Update, Delete, and DDL. Currently, "custom DDL operations" are not supported. |
Sync Object Option | Sync Object | Entire instance: Synchronize the entire instance. Specified object: Synchronize the specified object, and basic databases and tables are supported. |
Advanced Options | Limit Transfer Speed | Generally, no configuration is required here. Adjustment is only required when the database configuration is low and cannot handle the default concurrent number of threads and RPS for DTS. For details, see Speed Limit. |
Step 4: Setting the Consistency Check
Data consistency check involves comparing table data between the source and target databases through DTS during data synchronization, providing comparison results and inconsistency details to help users quickly verify synchronization results.
On the Set consistency check page, check Enable data consistency verification, configure the parameters, and then click Next.
Configuration Item | Parameter | Description |
Check Option | Verification Scope | Database information: Check indexes, database and table information, and shard keys in the source and target databases. When both the source and target databases are sharded clusters, it is supported to select shard keys for checking. Full data validation: Compare all data in the target database before it is fully synchronized with the source database. Continuous incremental verification: Compare the incremental data generated after the initiation of the continuous incremental check task. |
| Database Information | Checking indexes, shard keys, and database and table information is supported. When both the source and target databases are sharded clusters, it is supported to select shard keys for checking. |
| Data Check | Content check: Perform a content check on the selected check objects. Row number check: Perform a comparison of the row number on the selected check objects. |
| Verification Benchmark | Source: Use data from the source as the check benchmark. |
VerificationParameters | ThreadCount | The value range is 1–8. Select an appropriate value according to the actual situation. Increasing the number of threads can accelerate the consistency check speed, but it will also increase the load on the source and target databases. |
VerifiedObject option | Check Object | All synchronization objects: The check scope includes all objects selected for the synchronization task. |
Step 6: Checking and Starting a Task
1. On the check task page, click Immediate start after the check is completed, and all verification items are passed.
2. If the check task fails, you can see Troubleshooting Approach to fix the issue and reinitiate a check task.
Failed: indicates that a check item fails and the task is blocked. It is required to fix the issue and execute the check task again.
Warning: indicates that a check item does not completely meet the requirements, and the task can be continued, but the business may be affected. Users are required to assess whether to ignore the warning or fix the issue before continuing to execute the task based on the prompt.
3. In the pop-up dialog box, click OK.
4. Return to the data synchronization task list, and the task status is changed to Running.
Note:
Choose More > End in the Operation column to disable the synchronization task. Ensure that data synchronization is completed before disabling the task.
5. (Optional) You can click the task name to go to the task details page and view the task initialization status and monitoring data.
피드백