This document provides guidance on how to improve the DTS transmission rate in various scenes.
Factors Affecting Transmission Performance
1. Source/Target Specifications: During DTS transmission, the load on both the source and target ends will increase. If the load is already high, this can lead to a decline in transmission performance. In scenes where the source or target end has lower specifications, it is recommended to reduce the DTS transmission rate accordingly.
2. DTS Transmission Parameter Configuration: The RPS (Requests Per Second) and concurrency settings for DTS tasks can be flexibly configured. In most cases, the default parameters do not require modification. However, if these settings are not configured properly, they may limit performance.
3. Network Bandwidth and Latency Between Source and Target: The network between the source and target should be stable, with no latency or packet loss, to ensure optimal performance.
When the Public Network is selected as an access method, the bandwidth cannot be guaranteed due to the public network environment, which may become a transmission bottleneck. It is recommended to select Direct Connect or VPN Access to improve network quality.
When the CCN is selected as an access method, note that it only provides free bandwidth of up to 10 Kbps between all regions. If you are using DTS to transmit official business data, you will need to pay for higher bandwidth.
Cross-region transmission latency is typically higher than within the same region. The region of the DTS task instance should match the region of the target instance to avoid increased latency. Additionally, for self-hosted databases, it is recommended to select a DTS region that is geographically close to the source for optimal transmission.
4. Source Data Issues: If the data on the source end does not comply with DTS specifications (e.g., long-running SQL queries, tables without primary keys), it may cause the transmission to stall. In such cases, it is recommended to optimize the indexes and queries on the source end. Additionally, large transactions or frequent DDL operations on the source end can result in slower transmission rates during the incremental phase.
DTS Performance Optimization Methods
1. Adjust the DTS task transmission parameters.
Select the specified task, then choose More > Limit Speed in the Operation column to adjust the parameters.
Adjustable Parameters in the Full Data Phase: The number of concurrent threads and RPS for both the source and target database full data import can be adjusted.
Adjustable Parameters in the Incremental Data Phase: The number of concurrent threads for the target database incremental import can be adjusted.
2. Adjust the DTS task specifications.
Select the specified task, then choose More > Adjust Specification in the Operation column to adjust the parameters.
When you purchase a DTS task, different specifications correspond to different RPS limits during the incremental phase; for example, in data synchronization tasks, if the write load on the source database is too high and the purchased specification is insufficient, you will need to upgrade the DTS task specifications. For more details, see the Specification Description. |
micro | 1000 |
small | 2000 |
medium | 5000 |
large | > 5000 |
3. Pause and then resume the task.
Select the specified task, then choose More > Pause in the Operation column.
If a temporary spike in source database activity causes a high load on the target end, you can pause the task and resume synchronization after the peak has subsided.
General Principles
When you encounter performance issues, first check the network and configuration factors before adjusting the DTS rate configuration parameters.
DTS transmission performance is influenced by factors such as source/target configurations, DTS task specifications, network bandwidth, and latency. Therefore, simply increasing the number of concurrent threads or RPS may not be effective in improving performance. It is essential to analyze where the performance bottleneck lies and address the specific issue accordingly.
Performance Optimization Steps
1. Monitor the information.
Select the specified task, click Task ID to enter the task details page. Then, switch to the relevant tab and click Monitoring Data to view the corresponding metric data.
2. If the monitoring data shows high latency, first check for network issues. Latency greater than 3 ms within the same region is considered high, and cross-region latency may be slightly higher.
2.1 When the Public Network is selected as an access method, the bandwidth cannot be guaranteed due to the public network environment, which may become a transmission bottleneck. It is recommended to select Direct Connect or VPN Access to improve network quality.
2.2 When the CCN is selected as an access method, note that it only provides free bandwidth of up to 10 Kbps between all regions. For official business data transmission using DTS, you will need to pay for higher bandwidth.
2.3 The DTS task instance should be located in the same region as the target instance; otherwise, latency will increase. For example, if you intend to synchronize data to the Guangzhou region, you should create the synchronization task in the Guangzhou region.
2.4 If you are using a self-hosted database, it is important to select a DTS region that is geographically close to the database for transmission; otherwise, latency will increase. For example, if your self-hosted database is in Beijing and you need to synchronize it to a Tencent Cloud database in the Shanghai region, you should select the region as Beijing -> Shanghai when creating the DTS link. Choosing Shanghai -> Shanghai or Shenzhen -> Shanghai would increase latency.
3. During the Full Data Phase, adjust the DTS rate.
Note:
During the Full Export Phase of data migration tasks (limited to structure export or export of tables without primary keys) or data synchronization tasks, or when the primary key conflict strategy is set to Error on Conflict, adjusting the rate parameters may cause the task to restart from the beginning. Therefore, carefully evaluate before proceeding with the adjustments in sections 3.1 and 3.2.
3.1 Normal export rate: Full export BPS (Bytes Per Second) is higher than 20-50 MB/s, with network latency to the source instance less than 1 ms during full export.
In such cases, the adjustment is generally not necessary. However, if the user needs to increase the rate, it should be done after analyzing the load on the source end.
If the load on the source end is high (e.g., CPU usage greater than 90%), it is not recommended to increase the rate further.
If the load on the source end is low (e.g., CPU usage less than 50%), you can consider increasing the number of concurrent export threads on the source end. In the full data phase, the default RPS is significantly high, so adjusting the RPS is usually unnecessary when trying to improve transmission speed. If increasing the number of export threads does not produce significant results, proceed to step 5.
3.2 Limited export rate: Full export BPS is less than 20 MB/s, and network latency to the source instance during full export is greater than 3 ms.
Analysis should be conducted in conjunction with the load on the source end.
If the load on the source end is high (e.g., CPU usage greater than 90%), it indicates that the source's configuration is relatively low, and the added load from DTS transmission is further limiting performance. It is recommended to reduce the number of concurrent export threads and RPS on the source end. If this does not significantly improve performance, it is recommended to use a replica database for synchronization.
If the load on the source end is low, you can consider increasing the number of concurrent export threads. If this adjustment does not produce significant results, proceed to step 5.
3.3 Normal import rate: Full import BPS is higher than 20-50 MB/s, with network latency to the target instance less than 1 ms during full import.
In such cases, the adjustment is generally not necessary. However, if the user needs to increase the rate, it should be done after considering the resource usage on the target end.
If the load on the target end is high (e.g., CPU usage greater than 90%), it is not recommended to increase the rate further.
If the load on the target end is low (e.g., CPU usage less than 50%), you can consider increasing the number of concurrent import threads on the target end. If this adjustment does not produce significant results, proceed to step 5.
3.4 Limited import rate: Full import BPS is less than 20 MB/s, and network latency to the target instance during full import is greater than 3 ms.
Analysis should be conducted in conjunction with the load on the target end.
If the load on the target end is high (e.g., CPU usage greater than 90%), it indicates that the target's configuration is relatively low, and the added load from DTS transmission is further limiting performance. In this case, it is recommended to reduce the number of concurrent import threads and RPS on the target end.
If the adjustment does not yield significant results, you can use the pause feature to temporarily stop the task transmission. It is recommended not to pause the task for more than 3 days, as it may not be able to resume transmission afterward.
If the load on the target end is low, you can consider increasing the number of concurrent import threads. If this adjustment does not produce significant results, proceed to step 5.
4. During the Incremental Phase, adjust the DTS rate.
4.1 Check the monitoring data to see if the incremental export RPS from the source instance or the incremental import RPS to the target instance has reached the specified limit. Different task specifications have different RPS limits.
If the incremental RPS has reached the limit, first upgrade the DTS task specifications (done through the Adjust Specification operation).
If the incremental RPS is less than 60% of the task specification limit, performance may be restricted. Proceed to step 4.2.
4.2 Check the load on the target end.
If the target end is at its limit (e.g., CPU usage greater than 95%), proceed to step 4.3.
If the target end load is normal (e.g., CPU usage is between 60 and 85%), adjustments are not necessary.
If the target end load is low (e.g., CPU usage is less than 50%), you can consider increasing the DTS rate parameters.
If the target end is a database, you can consider increasing the number of concurrent write threads on the target end. If this adjustment does not produce significant results, proceed to step 5.
If the target end is Kafka, increasing the number of concurrent write threads may not significantly improve performance. Instead, first increase the bandwidth configuration and the number of partitions for the target CKafka (multiple partitions can be written concurrently, improving the write rate to the target end).
4.3 Check if there are large transactions or frequent DDL operations on the source end.
4.3.1 The source end has large transactions or frequent DDL operations.
This can cause a temporary load on the target end. You can pause the task and resume it after the large transaction or DDL operation has completed. However, it is recommended not to pause the task for more than 3 days, as this may lead to task failure.
4.3.2 The source end has no large transactions or frequent DDL operations.
In this case, the low configuration of the target end, combined with the added load from DTS transmission, is limiting performance. It is recommended to first reduce the number of concurrent import threads on the target end.
If the target end is a database, you can consider reducing the number of concurrent write threads. If this adjustment does not improve performance, you can consider downgrading the task configuration.
If the target end is Kafka, you can first increase the bandwidth configuration and the number of partitions on the target CKafka (multiple partitions can be written concurrently, improving the write rate to the target end). Then, reduce the number of concurrent write threads. If these adjustments do not improve performance, you can consider downgrading the task configuration.
5. Check the data content on the source end.
5.1 Check if there are long-running SQL queries on the source end.
If there are slow SQL queries running on the source end, it can cause the synchronization rate to slow down. Check for any slow SQL queries on the source end, address them as needed, and consider optimizing indexes to prevent slow SQL from running.
5.2 Check if there is any data on the source end that does not comply with DTS specifications.
If there are tables without primary keys on the source end, large-scale queries on these tables may cause the synchronization process to slow down. It is recommended to add primary keys to these tables or avoid synchronizing tables without primary keys.