DTS is a data transfer service with data migration, sync, and subscription features. DTS for MongoDB helps migrate your database to the cloud without interrupting your business. In its full + incremental data migration mode, historical data in the source database written before migration and incremental data written during migration can be migrated together.
Use Cases
DTS supports data migration for the following source and target databases:
Source
Target
Notes
Self-built MongoDB database in IDC and CVM
TencentDB for MongoDB
-
Third-party MongoDB database
TencentDB for MongoDB
-
TencentDB for MongoDB
TencentDB for MongoDB
Migration scenarios between TencentDB for MongoDB instances include:
Migration between TencentDB instances in the same or different regions.
Migration between TencentDB instances under the same or different accounts.
Migration between TencentDB for MongoDB instances on different versions.
Migration between TencentDB for MongoDB replica set and sharded clusters.
Use Limits
During migration, do not perform the following operations; otherwise, the migration task will fail:
Do not modify or delete user information (including username, password, and permissions) in the source and target databases and port numbers.
Do not clear oplogs in the source database.
Do not delete the target database TencetDTSData during data migration.
Manipulate data in the target database with caution during data migration; otherwise, data inconsistency may occur.
As DTS will filter out the DDL operations of the sharded cluster, do not perform DDL operations other than transactions on the source database during shard migration; otherwise, data inconsistency may occur.
If you only perform full data migration, do not write new data into the source instance during migration; otherwise, the data in the source and target databases will be inconsistent. In scenarios with data writes, to ensure the data consistency in real time, we recommend that you select full + incremental data migration.
If the source is a TencentDB for MongoDB 3.2 sharded cluster, all shard keys will be treated as hash shard keys during migration by default. If you want to use range shard keys in the target, create them in the target first before data migration.
We recommend that you clean up the orphaned documents in the source cluster in advance before migrating the sharded cluster. Otherwise, it may cause data inconsistency after migration. For more information on the operation, see cleanupOrphaned.
During incremental migration of a shard, do not enable data sharding on the source database; otherwise data of the target and source database will be inconsistent; if data sharding is enabled, check the status of the shard on the target database and re-enable it. For more information on enabling a shard, see Shard a Collection.
For migration with DTS, only replica set migration supports DDL operations, while shard migration will filter out DDL operations (except transactions).
When DTS performs full data migration, it will occupy some resources in the source instance, which may increase the load of the source instance and the database pressure. If your database has low configurations, we recommend that you migrate the data during off-peak hours.
To migrate instances over the public network, make sure that the source instance is accessible from the public network.
Incremental migration is not supported for self-built single-node instances as they have no oplogs.
Preparations
Before migration with DTS, check the source and target environments as follows:
Type
Environment Requirements
Requirements for the source database
The server where the source database resides must have enough outbound bandwidth; otherwise, the migration speed will be affected.
The user provided by the source database must have the permission to read the database.
The source database cannot have a database named TencetDTSData.
If the source database is in cluster mode, the balancer needs to be disabled before incremental sync.
Oplogs can be obtained from the source database during full + incremental migration.
Requirements for the target database
The target database space must be at least 1.3 times the space of the collections to be migrated in the source database.
The user provided by the target database must have the root permission.
The target database cannot have a database named TencetDTSData.
The target database cannot have collections with the same name as those in the source database.
If the source database is a sharded database, you need to correctly enter the mongos, ConfigServer, and mongod node information.
The target database cannot have active businesses; otherwise, an alarm will be reported.
The shard keys of the source and target databases must be the same; otherwise, an alarm will be reported.
Make sure that the access types of the source and target network environments are interoperable. For more information, see Overview.
Get the accounts and passwords for accessing the source and target databases.
We recommend that you create a read-only account in the source instance for the migration in the following way:
use admin
db.createUser({
user: "username",
pwd: "password",
roles:
[
{role: "readAnyDatabase", db: "admin"},
{role: "read", db: "local"}
]
})
Directions
1. Log in to the DTS console, select Data Migration on the left sidebar, and click Create Migration Task to enter the DTS page.
2. On the DTS page, select the types, regions, and specifications of the source and target instances as follows and click Buy Now.
Parameter
Description
Parameter Configuration
Service Type
Select the service type that DTS needs to provide, including data migration, data sync, and data subscription.
Select Data Migration.
Creation Mode
Select the mode for creating the task.
Create Task: Create a new task.
Create similar task: Quickly create a similar migration task based on an existing task.
Select an option as needed.
Task ID
This parameter will be displayed if Creation Mode is Create similar task. It configures the ID of the existing task.
To create a similar task, enter the ID of the existing task in the Task ID input box.
Billing Mode
Select the billing mode of the task. For pay-as-you-go billing details, see Billing Overview.
-
Source Instance Type
Select the source database type.
Select MongoDB, which cannot be modified after purchase.
Source Instance Region
Select the source database region.
If the source database is a self-built one, select a region nearest to it.
Target Instance Type
Select the target database type.
Select MongoDB, which cannot be modified after purchase.
Target Instance Region
Select the target database region.
-
Specification
Select the specification of the migration link.
It is specified as Medium.
Tag
Set tags for the migration task to facilitate task management.
Click Add and select specific tag keys and values in the Tag Key and Tag Value drop-down lists.
Task Name
Set the task name.
Select Name after Creation. The task name is the same as the task ID by default. After the migration task is created, you can rename the task.
Select Name Now and enter the task name in the input box below.
Any special symbols other than hyphen are not supported.
Letters and digits are supported.
The length limit is as displayed in the console.
We recommend that you set an easily recognizable name.
Terms of Service
Learn about the terms of service for data migration.
Select I have read and agreed to TENCENT CLOUD TERMS OF SERVICE.
3. You will be automatically redirected to the migration task list. Find the newly created migration task, view the task status/progress, and wait for the task to be created. Then, click Configure in the Operation column to enter the Modify Migration Task wizard. On the Set source and target databases tab, set the source and target databases.
Note:
We recommend that you create a read-only account in the source instance for the migration; otherwise, an alarm will be reported during the pre-verification. You can ignore the alarm as needed.
Configuration Type
Parameter
Description
Configuration Method
Task Configuration
Task Name
Set the name of the data migration task.
If you have already set the task name when creating the data migration task, this parameter will be the specified name by default.
If you haven't set the task name when creating the data migration task, enter an easily recognizable task name in the input box.
Any special symbols other than hyphen are not supported.
Letters and digits are supported.
The length limit is as displayed in the console.
Running Mode
Configure the running time of the data migration task.
Immediate execution: The task will be started immediately after the task verification passes.
Scheduled execution: You can set a specific time to start migration.
After the migration task is configured, you can modify the scheduled execution time before it comes.
After the scheduled execution time is set for the migration task, if you want to run the task immediately before the scheduled time, click Immediate start in the Operation column in the migration task list.
Select an option as needed. If Scheduled execution is selected, select the specific time in the Execution Time input box below.
After the migration task is configured, you can modify the scheduled execution time before it comes.
After the scheduled execution time is set for the migration task, if you want to run the task immediately before the scheduled time, click Immediate start in the Operation column in the migration task list.
Source Database Settings
Source Database Type
The source database type selected during purchase.
Check whether the source database type is correct, which cannot be modified. If you want to modify it, you can only terminate the current task and create another one.
Region
The source database region selected during purchase.
Check whether the source database region is correct, which cannot be modified.
Access Type
Select the network type for the source database to access the migration task.
Public Network: The source database can be accessed through a public IP.
Self-Build on CVM: The source database is deployed in a CVM instance.
Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
VPN Access: The source database can be interconnected with VPCs through VPN Connections.
Database: The source database is a TencentDB instance.
CCN: The source database can be interconnected with VPCs through CCN.
VPC: The source and target databases can be interconnected through VPC.
Select an option based on your actual network environment. Network parameters vary by access network type.
Source Database Network Environment
This parameter will be displayed if Access Type is CCN. It configures whether the source database is a TencentDB database.
Select an option based on the source database.
Architecture
This parameter will be displayed if Access Type is Public Network, Self-Build on CVM, Direct Connect, VPN Access, or CCN. It configures the architecture of the source database.
Replica Set: It means that the source database is a replica set, which consists of one primary node and one or more secondary nodes.
Cluster Migration: It means that the source database is a sharded cluster, which consists of mongos nodes, ConfigServer nodes, shard nodes, and other components.
Single-Node: It means that the source database has a single node for reads/writes.
Select an option based on the cluster architecture of the source database. After the connectivity test is performed, the architecture type cannot be modified.
VPC-based Direct Connect Gateway
This parameter will be displayed if Access Type is Direct Connect. It sets the Direct Connect gateway ID of the VPC.
Select the VPC-based Direct Connect gateway ID in the drop-down list. Only VPC-based direct connect gateway is supported. Confirm the network type associated with the gateway.
VPC
This parameter will be displayed if Access Type is Direct Connect or VPN Access. It configures the information of the VPC the source database accesses through Direct Connect or VPN.
Select the VPC ID and subnet in the drop-down lists.
VPN Type
This parameter will be displayed if Access Type is VPN Access. It indicates the VPN type.
It is specified as Cloud VPN Service.
Host Address
This parameter will be displayed if Access Type is Public Network, Direct Connect, VPN Access, or CCN. It sets the host IP address or domain name of the source database.
Enter the IP address or domain name of the source database in the input box, such as 10.20.3.21.
Port
This parameter will be displayed if Access Type is Public Network, Direct Connect, VPN Access, or CCN. It configures the network port of the source database.
Enter the port number in the input box, such as 6379.
CVM Instance
This parameter will be displayed if Access Type is Self-Build on CVM. It configures the information of the CVM instance.
Select the CVM instance ID in the drop-down list.
Authentication Required
This parameter will be displayed if Access Type is Public Network, Self-Build on CVM, Direct Connect, VPN Access, or CCN. It configures whether to require authentication for the access account of the source database.
Select an option based on your data security requirement.
Yes: The source database requires authentication. In this case, you need to configure both Authentication Database and Authentication Mechanism.
No: The source database can be accessed without authentication.
Authentication Database
This parameter will be displayed if Authentication Required is Yes. It configures the name of the authentication database for accessing the source database.
Enter the authentication database name in the input box. By default, the admin database is used as the authentication database for login authentication.
Authentication Mechanism
This parameter will be displayed if Authentication Required is Yes. It configures the authentication method for the source database.
Currently, only SCRAM-SHA-1 is supported.
Cross-/Intra-Account
This parameter will be displayed if Access Type is Database. It configures whether the source TencentDB instance requires cross-account access.
Cross-Account: The source database and the current account belong to different Tencent Cloud accounts.
Intra-Account: The source database and the current account belong to the same Tencent Cloud account.
Database Instance
This parameter will be displayed if Access Type is Database. It configures the information of the TencentDB instance.
Select the source TencentDB instance ID in the drop-down list.
VPC-based CCN Instance
This parameter will be displayed if Access Type is CCN. It configures the information of the CCN instance.
Select the CCN instance ID in the drop-down list.
Node - mongod
This parameter will be displayed if Architecture is Cluster Migration. It configures the mongod node information of the source MongoDB database.
Enter the IP and port of the mongod node in the input box and separate multiple nodes by line break, such as 10.3.55.77:6379.
Node - mongos
This parameter will be displayed if Architecture is Cluster Migration. It configures the mongos node information.
Enter the IP and port of the mongos node in the input box.
Node - ConfigServer
This parameter will be displayed if Architecture is Cluster Migration. It configures the IP address and port of the ConfigServer node.
Enter the IP and port of the ConfigServer node in the input box.
Account & Password Selection
This parameter will be displayed if Architecture is Cluster Migration and Authentication Required is Yes. It configures whether the accounts of the mongod, mongos, and ConfigServer nodes of the source database are the same.
Same account & password: The accounts and passwords of the mongod, mongos, and ConfigServer nodes are the same.
Different accounts & passwords: The accounts and passwords of the mongod, mongos, and ConfigServer nodes are different and need to be configured separately. Enter their account and password information in the input boxes below.
Account
This parameter will be displayed if Authentication Required is Yes. It configures the account information that needs to be authenticated.
Enter the account information for accessing the source database.
Password
This parameter will be displayed if Authentication Required is Yes. It configures the password corresponding to the authentication account.
Enter the password corresponding to the account to access the source database.
Target Database Settings
Target Database Type
The target database type selected during purchase.
Check whether the target database type is correct, which cannot be modified. If you want to modify it, you can only terminate the current task and create another one.
Region
The target database region selected during purchase.
Check whether the target database region is correct, which cannot be modified.
Access Type
Configure the network access type of the target database.
It is specified as Database, indicating that the target database is a TencentDB instance.
Database Instance
Select the instance ID of the target database.
Select the instance ID of the target database.
Account
Account of the target database.
Enter the account information in the input box.
Password
Password of the target database.
Enter the password of the account in the input box.
4. After configuring the source and target databases, click Test Connectivity to test their network connectivity respectively. If the connectivity test fails, fix the problem as instructed in Failed Connectivity Test.
5. After the connectivity test passes, click Save. On the Set migration options and select migration objects page, configure the migration options and objects and click Save.
Parameter
Description
Configuration Method
Migration Type
Select an option as needed.
Full migration: The entire database will be migrated. The migrated data will only be existing content of the source database when the task is initiated but not include the incremental data written to the source database after the task is initiated.
Full + Incremental migration: The migrated data will include the existing content of the source database when the task is initiated as well as the incremental data written to the source database after the task is initiated.
Select whether to migrate incremental data based on the actual situation of the migrated data.
Data Consistency Check
Configure whether to perform data consistency check after full migration is completed.
Full check: A detailed comparison of the data in the source and target databases will be performed after migration. When the migration task enters the "incremental sync" stage, the source-target database data gap is 0 MB, and the source-target database time lag is 0 seconds, the migration task will automatically trigger a data consistency check task.
No check: This indicates not to perform data check. After the task is created, you can manually trigger the consistency check when the task enters the "incremental sync" stage. For more information, see Creating Data Consistency Check Task.
If Migration Type is Full migration, Data Consistency Check is specified as No check.
If Migration Type is Full + Incremental migration, you can select Full check or No check.
Full check: A detailed comparison of the data in the source and target databases will be performed after migration. When the migration task enters the "incremental sync" stage, the source-target database data gap is 0 MB, and the source-target database time lag is 0 seconds, the migration task will automatically trigger a data consistency check task.
No check: This indicates not to perform data check. After the task is created, you can manually trigger the consistency check when the task enters the "incremental sync" stage. For more information, see Creating Data Consistency Check Task.
Data Check
It is specified as Content check, i.e., checking the data in the database.
-
Migration Object
Configure the specific scope of the migration task.
Entire instance: Migrate the entire database instance, including the metadata definitions of roles and users but excluding system databases such as system objects in PostgreSQL.
Specified objects: Migrate specified objects.
If Migration Object is Specify object, select the specific collections to be migrated in Source Database Object below. Then, confirm the selected collections in Selected Object.
6. On the Verify task tab, wait for the pre-verification to complete and click Start Task.
If the verification fails, fix the problem as instructed in Check Item Overview and initiate the verification again.
If the verification result is Failed, a check item fails and the task is blocked. Click View Details to view the cause of the failure. You need to fix the problem and run the verification task again.
If the verification result is Alarm, a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. Click View Details to view the alarm. You need to assess whether to ignore the alarm.
7. Return to the migration task list and wait for the task to complete.
If Full migration is selected for Migration Type, once completed, the task will be stopped automatically.
If Full + Incremental migration is selected for Migration Type, after full migration is completed, the migration task will automatically enter the incremental data sync stage, which will keep running and will not stop automatically. To stop the task, make sure that the migration stage is incremental sync, the source-target database time lag is 0 seconds, and there is no delay, stop writing to the source database for a few minutes, and click Complete* in the Operation** column of the migration task.
More Operations
(Optional) If you want to view, delete, or perform other operations on a task, click the task and select the target operation in the Operation column. For more information, see Viewing Task.
After the migration task status becomes Task successful, you can formally cut over the business. For more information, see Cutover Description.
Yes
No
Was this page helpful?