tencent cloud

Feedback

Migration with DTS

Last updated: 2022-07-11 15:57:31

    Background Overview

    Tencent Cloud Data Transmission Service (DTS) is a data transmission service that integrates such features as data migration, sync, and subscription. It helps you migrate your databases to the cloud without interrupting your business and build a high-availability database disaster recovery architecture through real-time sync channels. Its data subscription feature meets your requirements for commercial data mining and async business decoupling.

    DTS for Redis currently supports the data migration feature for you to migrate data to TencentDB in a non-stop manner at a time. In addition, in its full + incremental data migration mode, historical data in the source database written before migration and incremental data written during migration can be migrated together.

    Use Cases

    DTS for Redis data migration supports source and target databases in the following deployment modes:

    Source Database Target Database Description
    Self-built Redis database, such as self-built databases in IDC and CVM TencentDB for Redis -
    Third-party Redis TencentDB for Redis The third-party cloud vendor should grant the SYNC or PSYNC command permission.
    TencentDB for Redis TencentDB for Redis Data migration, version upgrade, and cross-region migration are supported between database instances under the same Tencent Cloud account.

    Migration Support Description

    Note:

    For compatibility issues with migration from Standalone Edition to Memory Edition (Cluster Architecture), see Check on Migration from Standard Architecture to Cluster Architecture.

    Supported versions

    • DTS supports Redis 2.8, 3.0, 3.2, 4.0, 5.0, and 6.0. To use it for v6.0, submit a ticket for application. We recommend you migrate from an earlier version to a later version; otherwise, compatibility problems may occur.
    • Standard architecture and cluster architecture instances can be migrated to each other; however, such heterogeneous migration may has compatibility problems.
    • Supported architectures include single-node, Redis cluster, Codis, and twemproxy.
    • Migration permission requirements: To migrate data through DTS, the source instance must support SYNC or PSYNC commands.

    Supported networks

    DTS supports data migration based on the public network, CVM instances, Direct Connect gateways, VPN gateways, and CCN.

    • Public Network: The source database can be accessed through a public IP.
    • Self-Build on CVM: The source database is deployed in a CVM instance.
    • Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
    • VPN Access: The source database can be interconnected with VPCs through VPN Connections.
    • CCN: The source database can be interconnected with VPCs through CCN.

    Limitations

    • To ensure migration efficiency, cross-region migration of CVM-based self-built instances is not supported.
    • To migrate instances over a public network, make sure that the source instance can be accessed over the public network.
    • Only instances that are running normally can be migrated, while instances with no password initialized or with ongoing tasks cannot.
    • The target instance must be empty. During the migration process, the target instance will be read-only and cannot be written to.
    • After the successfully migrated data is verified by your business, you can disconnect the source instance and connect to the target instance.

    Environment Requirements

    System check

    Note:

    The DTS system will check the following environment requirements before starting a migration task and report an error if a requirement is not met. You can also check them in advance. For more information on error handling, see Check Item Overview.

    TypeEnvironment Requirement
    Requirements for source database
  • The source and target databases can be connected.
  • The number of databases in the source database must be less than or equal to that in the target database.
  • The source database must be on v2.2.6 or later.
  • The source database must be a replica node.
  • Requirements for target database
  • The target database version should be later than or equal to the source database version; otherwise, an alarm will be triggered for compatibility problems during verification.
  • The target database must have the latest proxy.
  • The space of the target database must be at least 1.5 times the volume of the data to be migrated in the source database.
  • The target database must be empty.
  • Manual check

    You should check and make sure that the following items are passed before the migration; otherwise, the migration may fail.

    Big keys in the source database

    Before the migration, check whether there are big keys in the source database. They may cause the buffer (client-output-buffer-limit) to overflow during the migration, leading to a migration failure.

    • For TencentDB databases, you can use the performance optimization feature of TencentDB for DBbrain to quickly analyze big keys. For more information, see Memory Analysis.
    • For non-TencentDB databases, use RDBTools to analyze big keys in Redis.

    Evaluate large keys for splitting or cleaning. If you need to retain them, set the source buffer size (client-output-buffer-limit) to infinite.

    config set client-output-buffer-limit 'slave 0 0 0' 
    

    Limit on the number of TCP connections in the source Linux kernel

    If the number of concurrent business requests is high, check the limit on the number of connections in the Linux kernel before the migration. If this value is exceeded, the Linux server will actively disconnect from DTS.

    echo "net.ipv4.tcp_max_syn_backlog=4096" >> /etc/sysctl.conf
    echo "net.core.somaxconn=4096" >> /etc/sysctl.conf
    echo "net.ipv4.tcp_abort_on_overflow=0" /etc/sysctl.conf
    sysctl -p
    

    Access permission of the source RDB file directory

    Before the migration, check and make sure that the directory where RDB files are stored in the source database is readable; otherwise, the migration will fail.

    If the RDB file directory is not readable, run the following command in the source database to set "diskless replication". Then, RDB files will be directly sent to DTS for storage, with no need to be stored in the source database first and then sent.

    config set repl-diskless-sync yes
    

    Command compatibility (for migration from standard architecture to cluster architecture)

    The most challenging problem in migrating data from Standard Edition to Memory Edition (Cluster Architecture) is the command compatibility with usage specifications of Memory Edition (Cluster Architecture).

    • Multi-key operations
      TencentDB for Redis Memory Edition (Cluster Architecture) only supports cross-slot multi-key access for mget, mset, del, and exists commands. In the source database, keys that need to engage in multi-key computing can be aggregated into the same slot through a hash tag. For more information on how to use hash tags, see Redis cluster specification.
    • Transactional operations
      Memory Edition (Cluster Architecture) supports transactions, but cross-slot access to keys in transactions is not supported.

    Perform static and dynamic evaluations as instructed in Check on Migration from Standard Architecture to Cluster Architecture.

    Migration Directions

    1. Create a migration task

    (1) Log in to the DTS console and click Create Migration Task on the data migration page.
    (2) On the Create Migration Task page, select the types and regions of the source and target databases and click Buy Now.

    2. Set the source and target databases

    Configure the Source Database Settings and Target Database Settings and click Test Connectivity. After the test is passed, click Save to proceed to the next step.

    Setting TypeConfiguration ItemDescription
    Task Configuration Task Name Set a meaningful name for easy task identification.
    Running Mode You can set **Immediate execution** or **Scheduled execution**.
    • If a scheduled task is modified and passes verification, you need to click Scheduled start again to make the task start at the scheduled time.
    • If the specified time has passed, the task will be started immediately. You can also click Immediate start to start the task immediately.
    Tag Tags are used to manage resources by category in different dimensions. If the existing tags do not meet your requirements, go to the console to create more.
    Source Database Settings Source Database TypeThe source database type selected during purchase, which cannot be changed.
    RegionThe region selected during purchase, which cannot be changed.
    Access TypeFor a third-party cloud database, you can select **Public Network** generally or select **VPN Access**, **Direct Connect**, or **CCN** based on your actual network conditions.
    In this scenario, select **Direct Connect** or **VPN Access**. You need to configure VPN-IDC interconnection in this scenario. For the preparations for different access types, see Overview.
    • Public Network: The source database can be accessed through a public IP.
    • Self-Build on CVM: The source database is deployed in a CVM instance.
    • Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
    • VPN Access: The source database can be interconnected with VPCs through VPN Connections.
    • Database: The source database is a TencentDB instance.
    • CCN: The source database can be interconnected with VPCs through CCN.
    VPC-based Direct Connect GatewayOnly VPC-based Direct Connect gateway is supported. Confirm the network type associated with the gateway.
    VPCSelect a VPC and subnet associated with the VPC-based Direct Connect gateway.
    Node Type**Single-Node Migration** and **Cluster Migration** are supported. **Cluster Migration** is used as an example here.
    Currently, there are no limits on the number of shards and replicas in migration from Cluster Edition Redis to Cluster Edition Redis.
    Node InfoEnter the addresses and passwords (IP:port:password or IP:port) of all shards of the source database cluster and separate the information of different nodes with line breaks.
    We strongly recommend you migrate data from a replica node of the source database to avoid any impact on business access to the source database.
    Target Database Settings Target Database TypeThe target database type selected during purchase, which cannot be changed.
    RegionThe target database region selected during purchase, which cannot be changed.
    Access TypeSelect a type based on your scenario. In this scenario, select **Database**.
    Database InstanceSelect the instance ID of the target database.

    3. Verify and start the task

    On the task verification page, verify the task. After the verification is passed, click Start Task.

    • Failed: It indicates that a check item failed and the task is blocked. You need to fix the problem and run the verification task again.
    • Alarm: It indicates that a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. You need to assess whether to ignore the alarm or fix the problem and continue the task based on the alarm message.

    Return to the data migration task list, and you can see that the task has entered the Preparing status. After 1–2 minutes, the data migration task will be started.

    4. Complete the migration task

    (1) (Optional) If you want to view, delete, or perform other operations on a task, click the task and select the target operation in the Operation column. For more information, see Viewing Task.
    (2) If the keys of the source and target databases are the same, click Done in the Operation column to stop the data migration task.
    (3) After the migration task status becomes Task successful, verify the data in the target database. If the verification is passed, you can formally cut over the business. For more information, see Cutover Description.

    Event Alarming and Metric Monitoring

    (1) DTS can automatically report event alarms triggered upon migration interruption to keep you informed of any exceptions. For detailed directions, see Configuring Alarm Policy for Data Migration.
    (2) DTS allows you to view the monitoring data of various metrics during migration to understand the performance metrics of the system. For more information, see Viewing Monitoring Metric.

    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support