Synchronization Instructions
Scenarios
This document describes how to use Data Transfer Service (DTS) to synchronize data from a PostgreSQL database to TencentDB for PostgreSQL.
Preparations
1. Read the Use Instructions carefully to learn about feature constraints and must-knows.
2. Establish the access channel between DTS and the database in advance according to the access type you want to use. For details, see Network Preparations.
Internet Data Center (IDC) self-built databases/Databases of other cloud vendors: The access method can be set to Public Network/DC/VPN Access/CCN.
When instances are synchronized via the public network, ensure that the service of the source database instance is accessible in the public network environment and maintain the stability of the public network connection. Network fluctuations or failures may cause synchronization failure. Once synchronization fails, you need to re-initiate the synchronization task.
Self-built databases on Cloud Virtual Machine (CVM) instances: The access method can be set to Self-Built on CVM.
TencentDB instances: The access method can be set to Cloud Database.
3. The account that executes the DTS synchronization task must have the following permissions for the source database: Event Trigger, Replication, CREATE, DROP, ALTER, SELECT, UPDATE, INSERT, and DELETE.
Note:
When the source database is AWS RDS PostgreSQL, the synchronization account must also have the
rds_superuser role permission. Use the master user to execute the following statement for authorization:GRANT rds_superuser TO <synchronization account>;
4. The account that executes the DTS synchronization task must have the following permissions for the target database: CREATE, DROP, ALTER, SELECT, UPDATE, INSERT, and DELETE.
Note:
When the target database is AWS RDS PostgreSQL, the synchronization account must also have the
rds_superuser role permission. Use the master user to execute the following statement for authorization:GRANT rds_superuser TO <synchronization account>;
Operation Steps
Step 1: Creating a Synchronization Task
1. Log in to the DTS console, select Data Sync in the left sidebar, and then click Create Sync Task to go to the synchronization task creation page.
2. On the synchronization task creation page, select the source database instance type and region as well as the target instance type, region, and specification for synchronization, and then click Buy Now.
Parameter | Description |
Creating Mode | New task: Create a task. Create a similar task: Quickly create a task with the same configuration as the historical task. The settings of the new task, such as the database type and billing mode, are pre-populated to match the historical task. Users can also modify these settings as needed. |
Billing Mode | The yearly/monthly subscription and pay-as-you-go modes are supported. |
Source Instance Type | Select PostgreSQL. It cannot be modified after purchase. |
Source Instance Region | Select the source instance region. It cannot be modified after purchase. |
Target Instance Type | Select PostgreSQL. It cannot be modified after purchase. |
Target Instance Region | Select the target instance region. It cannot be modified after purchase. |
Specification | Select specifications according to your business requirements. Higher specifications deliver better performance. For details, see Billing Overview. |
Tag | Set tag keys and values for synchronization tasks to facilitate efficient management of multiple tasks through tags. |
Task Name | Set it to Name After Creation. The task name is the same as the task ID by default. After the synchronization task is created, the task name can be reset. Set it to Name Now. Enter the task name in the input box below. |
Terms of Service |
3. After the purchase is completed, return to the data synchronization task list to view the newly created data synchronization task. It requires configurations before it can be used.
Step 2: Setting Source and Target Databases for Synchronization
1. In the data synchronization task list, click Configure in the Operation column to go to the synchronization task configuration page.
2. On the Set source and target databases tab, enter the information about source and target databases, and then click Test Connectivity. After connectivity testing, click Next.
There are many scenarios where the deployment forms and access types of the source database are combined in different ways. Considering that the operation steps for synchronization are similar in various scenarios, only configuration examples for typical scenarios are provided below. See the following examples for configurations in other scenarios.
Example 1: Data is synchronized from a local self-built database to a TencentDB instance via DC/VPN.
Configuration Item | Parameter | Description |
Task Configuration | Task Name | DTS automatically generates a task name. You can customize it based on your actual needs. |
| Running Mode | Support immediate execution and scheduled execution. |
Source Database Settings | Source Instance Type | Source instance type selected during purchase. It cannot be modified. |
| Source Instance Region | Source instance region selected during purchase. It cannot be modified. |
| Access Type | Select the access type based on your scenario. For this scenario, select DC or VPN Access. This scenario requires configuring the connection between VPN and IDC. For preparations of other access types, see Preparations Overview. Public Network: The source database is accessible through a public IP address. Self-Build on CVM: The source database is deployed on Tencent CVM. Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC. VPN Access: The source database can connect to Tencent Cloud VPC through VPN Access. Database: The source database is a TencentDB instance. CCN: The source database can connect to Tencent Cloud VPC through CCN. |
| VPC-Based Direct Connect Gateway/VPN Gateway | Only the VPC-based Direct Connect gateway is supported for DC. Confirm the network type associated with the gateway. For the VPN gateway, select the VPN gateway instance for access. |
| VPC | Select the VPC network and subnet associated with the VPC-based Direct Connect gateway and VPN gateway. |
| Host Address | IP address or domain name for accessing the source database instance. |
| Port | Port for accessing the source database instance. |
| Account | Source instance account. The account permissions should meet the requirements. |
| Password | Password of the source instance account. |
Target Database Settings | Target Instance Type | Target instance type selected during purchase. It cannot be modified. |
| Target Instance Region | Target instance region selected during purchase. It cannot be modified. |
| Access Type | Select the access type based on your scenario. For this scenario, set it to Cloud Database. |
| Instance ID | Select the target instance ID. |
| Account | Target instance account. The account permissions should meet the requirements. |
| Password | Password of the target instance account. |
Example 2: Data is synchronized from a TencentDB instance to another TencentDB instance.
Configuration Item | Parameter | Description |
Task Configuration | Task Name | DTS automatically generates a task name. You can customize it based on your actual needs. |
| Running Mode | Support immediate execution and scheduled execution. |
Source Database Settings | Source Instance Type | Source instance type selected during purchase. It cannot be modified. |
| Source Instance Region | Source instance region selected during purchase. It cannot be modified. |
| Access Type | Select the access type according to your scenario. Set it to Cloud Database for this scenario. For preparations related to different access types, see Preparation Overview. Public Network: The source database is accessible through a public IP address. Self-Build on CVM: The source database is deployed on Tencent CVM. Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC. VPN Access: The source database can connect to Tencent Cloud VPC through VPN Access. Database: The source database is a TencentDB instance. CCN: The source database can connect to Tencent Cloud VPC through CCN. |
| Cross-/Intra-Account | Same account: The Tencent Cloud root account for both the source and target database instances is the same. Cross-account: The Tencent Cloud root accounts for the source and target database instances are different. For guidance on cross-account synchronization operations, see Synchronization Between Cross-Account Cloud Database Instances. |
| Instance ID | ID of the source database instance. |
| Account | Source instance account. The account permissions should meet the requirements. |
| Password | Password of the source instance account. |
Target Database Settings | Target Instance Type | Target instance type selected during purchase. It cannot be modified. |
| Target Instance Region | Target instance region selected during purchase. It cannot be modified. |
| Access Type | Select the access type based on your scenario. For this scenario, set it to Cloud Database. |
| Instance ID | Select the target instance ID. |
| Account | Target instance account. The account permissions should meet the requirements. |
| Password | Password of the target instance account. |
Example 3: Data is synchronized from a PostgreSQL instance of a third-party cloud vendor to a TencentDB instance via the public network.
Configuration Item | Parameter | Description |
Task Configuration | Task Name | DTS automatically generates a task name. You can customize it based on your actual needs. |
| Running Mode | Support immediate execution and scheduled execution. |
| Automatic Retry | When it is enabled, if a synchronization task is temporarily interrupted due to network exceptions or other issues, DTS will automatically retry and resume the task within the specified time range without requiring manual intervention from users. The supported time range for automatic retries is 5–720 minutes. |
Source Database Settings | Source Instance Type | Source instance type selected during purchase. It cannot be modified. |
| Source Instance Region | Source instance region selected during purchase. It cannot be modified. |
| Access Type | For databases from a third-party cloud vendor, you can generally choose to access them via the public network, VPN, DC, or CCN based on actual network conditions. Select Public Network for this scenario. For preparation steps for different access types, see Preparation Overview. Public Network: The source database is accessible through a public IP address. Self-Build on CVM: The source database is deployed on Tencent CVM. Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC. VPN Access: The source database can connect to Tencent Cloud VPC through VPN Access. Database: The source database is a TencentDB instance. CCN: The source database can connect to Tencent Cloud VPC through CCN. |
| Host Address | IP address or domain name for accessing the source database instance. |
| Port | Port of the source database instance. |
| Account | Source instance account. The account permissions should meet the requirements. |
| Password | Password of the source instance account. |
Target Database Settings | Target Instance Type | Target instance type selected during purchase. It cannot be modified. |
| Target Instance Region | Target instance region selected during purchase. It cannot be modified. |
| Access Type | Select the access type based on your scenario. For this scenario, set it to Cloud Database. |
| Instance ID | Select the target instance ID. |
| Account | Target instance account. The account permissions should meet the requirements. |
| Password | Password of the target instance account. |
Step 3: Setting the Synchronization Option and the Synchronization Object
On the Set sync options and objects tab, set the data initialization, data synchronization, and synchronization object options. After the settings are completed, click Save and Go Next.
Note:
When Initialization Type is set to Full data initialization only, the system assumes that the table structure has already been created in the target database. Table structure synchronization will not be performed, and no check will be conducted for existing tables with the same name in the source and target databases. Therefore, if Precheck and report error is selected for If Target Already Exists, the pre-check and error reporting feature will not take effect.
If the user plans to perform a rename operation on a table during synchronization (for example, renaming table A to table B), the Sync Object is required to include the entire database (or the entire instance) where table A is located. If only table A is selected, data from table B will not be synchronized to the target database after the rename operation.
Configuration Item | Parameter | Description |
Data Initialization Option | Initialization Type | Structure Initialization: When a synchronization task is executed, the table structures of the source instance are initialized to the target instance first. Full data initialization: When a synchronization task is executed, the data of the source instance is initialized to the target instance first. For scenarios where only full data initialization is selected, users need to create the table structures in the target database in advance. Both are selected by default and can be deselected based on actual requirements. |
| If Target Already Exists | Precheck and report error: If tables with the same name exist, an error will be reported, and the process will be stopped. Ignore and execute: Full and incremental data are directly appended to the tables in the target instance. |
Data Sync Option | Primary Key Conflict Resolution | Report : If a primary key conflict in the table is detected during synchronization, an error is reported, and the data synchronization task is paused. Ignore: If a primary key conflict in the table is detected during synchronization, the primary key record in the target database is retained. Overwrite: When table primary key conflicts are detected during synchronization, the primary key record of the source database is used to overwrite that of the target database. |
| SQL Type | Supported operations: Insert, Update, Delete, and DDL. Currently, custom DDL operations are not supported. |
Sync Object Option | Sync Object | Select the objects to be synchronized. Basic objects such as databases, tables, and views are supported. For advanced objects, only SEQUENCE is supported. |
Advanced Options | Limit Transfer Speed | Generally, no configuration is required here. Adjustment is only required when the database has configurations too low to handle the default concurrent number of threads and RPS of DTS. For details, see Speed Limit. |
Step 4: Processing Data
During synchronization task configuration, DTS supports setting Where condition filters for individual tables or in batches. Only data meeting the specified rules will be synchronized to the target database, thus allowing users to flexibly perform data splitting.
1. Select Enable Where Condition Filtering.
2. Select objects.
From the range of synchronization objects selected in the previous step, select the objects for which you need to set the Where clause.
3. Set filter conditions and confirm.
UPDATE Data Condition: You can choose Satisfied before and after the update, Satisfied before the update, or Satisfied after the update.
Conditional Expression: Enter the where clause. After setting the where condition, click Check Conditions to verify the input validity, and then click Generate Filtering Rule.
4. Confirm the filtering rule. If it is correct, click Save and Go Next.
Step 5: Setting the Consistency Check
Data consistency check compares table data between the source and target databases through DTS during data synchronization, providing comparison results and inconsistency details to help users quickly verify synchronization results.
On the Set consistency check page, select Enable data consistency verification (enabled by default). After configuring the parameters, click Next.
Configuration Item | Parameter | Description |
check option | Verification Scope | full data validation: All data in the source and target databases is compared when a check task is initiated. Structure Verification: Only the most basic table structures in the source and target databases are checked. Others, such as indexes, constraints, and comments on the tables, are not checked. Continuous incremental verification: Data comparison is performed on the incremental data generated in the source database when the check task is initiated. Tables without primary keys will not be checked. |
| Verification Benchmark | Source: Use data from the source as the check benchmark. |
Verification Parameters | Thread Count | The value range is 1–8. Select an appropriate value according to the actual situation. Increasing the number of threads can accelerate the consistency check speed, but it will also increase the load on the source and target databases. |
Verified Object option | Check Object | All sync objects: The check scope includes all objects selected for the synchronization task. Custom: Select specific objects from the selected synchronization objects for checking. |
Step 6: Checking and Starting a Task
1. On the task check page, after completing the check and passing all check items, click Immediate Start.
If the check task fails, fix the issue as instructed in Check Failure Handling Methods and re-initiate the check task.
Failed: indicates that a check item fails and the task is blocked. Fix the issue and execute the check task again.
Warning: indicates that a check item does not completely meet the requirements, and the task can be continued, but the business may be affected. Users are required to assess whether to ignore the warning or fix the issue before continuing to execute the task based on the prompt.
Note:
During plugin detection in the pre-check phase, after the first mismatched plugin is detected and an alarm is triggered, the system stops detecting the remaining plugins.
2. In the pop-up dialog box, click OK.
3. Return to the data synchronization task list, and the task status is changed to Running.
Note:
Choose More > Stop in the Operation column to disable the synchronization task. Ensure that data synchronization is completed before disabling the task.
4. (Optional) You can click the Task ID / Name to go to the task details page and view the task initialization status and monitoring data.
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback