tencent cloud

Feedback

Configuring a DTS Connector

Last updated: 2022-06-13 11:49:13

    Overview

    Data Transmission Service (DTS) supports data subscription for MySQL, MariaDB, PostgreSQL, Redis, MongoDB, and other relational databases and NoSQL databases. It gets the data changes of key database businesses and provides them for downstream entities to subscribe to, get, and consume. This facilitates data sync between TencentDB databases and heterogeneous systems, such as cache update, real-time ETL (data warehousing technology) sync, and async business decoupling.

    In EventBridge, you can configure a DTS connector to pull incremental binlogs from the source instance in real time based on DTS data subscription to consume and process logs and deliver them to different targets. This document introduces how to create a DTS connector and the structure of events generated by the DTS connector.

    Feature Description

    For more information, see Data Subscription (for Kafka) Overview.

    Supported databases

    Source Database Type Source Database Version Subscribable Data Types
    TencentDB for MySQL TencentDB for MySQL 5.5.x, 5.6.x, 5.7.x, and 8.0.x
  • Data update
  • Structure update
  • Full instance
  • TencentDB for MariaDB TencentDB for MariaDB 10.0.10 and 10.1.9
  • Data update
  • Structure update
  • Full instance
  • TencentDB for MariaDB (Percona) TencentDB for MariaDB (Percona) 5.6.x and 5.7.x
  • Data update
  • Structure update
  • Full instance
  • TDSQL for MySQL TDSQL for MySQL 5.6.x, 5.7.x, and 8.0.18
  • Data update
  • Structure update
  • Full instance
  • TDSQL-C MySQL TDSQL-C MySQL 5.7.x and 8.0.x
  • Data update
  • Structure update
  • Full instance
  • TDSQL for PostgreSQL TDSQL for PostgreSQL Data update

    Supported subscription operations

    DTS allows you to subscribe to databases and tables. Specifically, the following three subscription types are supported:

    • Data update: Subscription to DML operations.
    • Structure update: Subscription to DDL operations.
    • Full instance: Subscription to the DML and DDL operations of all tables.

    Use limits

    • EventBridge has a limit on the event size. An upstream log greater than 1 MB cannot be successfully delivered to EventBridge for consumption. Pay attention to your log size.
    • Under the current scheme, EventBridge can consume data in multiple partitions concurrently and cannot guarantee the consumption sequence.
    • To ensure normal data consumption, you need to exercise caution when updating the consumer group account password after creating a connector. If the password is updated, you need to bind the connector again. Otherwise, data consumption may fail.
    • DTS supports batch delivery. Batch changes will be delivered in array format in the same event.

    Prerequisites

    1. You have activated the DTS data subscription service and created an instance.
    2. Your sub-account has obtained EventBridge and DTS related operation permissions by using the root account.

    Directions

    1. Log in to the EventBridge console and select Event Bus on the left sidebar.
    2. In the Event Bus list, select the event bus for which you want to configure a DTS connector.
    3. On the event bus details page, click Add in the event connector section, and set the connector type to DTS data subscriptions.
    4. Select the data subscription instance to consume as instructed, and enter information such as the consumer group name, account, and password as needed. If no consumer object is bound, complete related configuration in the DTS console.
    5. Click OK. Then you can view the information of the bound connector on the event bus page.
    6. On the event bus details page, click Manage Event Rules.
    7. Click Create Event Rule and enter the relevant information as prompted.
      You need to set Tencent Cloud service to DTS data subscriptions and configure data conversion and complete trigger target binding as needed.
    8. Click OK.

    Event Format

    The format of processed events received by a DTS connector is as follows:

    DDL operation example

    {
      "id":"38cecd93-a9c2-11ec-b952-******d8da53:16",
      "type":"dts:MYSQL:INSERT",
      "specversion":"1.0",
      "source":"dts.cloud.tencent",
      "subject":"cdb-xxxx",
      "time":1648109734,
      "region":"ap-guangzhou",
      "datacontenttype":"application/json;charset=utf-8",
      "tags":null,
      "data":{
         "topic":"topic-subs-xxxx-cdb-xxxx",
         "partition":3,
         "offset":16005,
         "partition_seq":16006,
         "event":{
            "dmlEvent":{
               "columns":[
                  {
                     "name":"string",
                     "originalType":"text"
                  },
                  {
                     "name":"int",
                     "originalType":"tinyint(4)"
                  },
                  {
                     "name":"time",
                     "originalType":"time"
                  },
                  {
                     "name":"double",
                     "originalType":"double"
                  },
                  {
                     "name":"id",
                     "originalType":"int(11)",
                     "isKey":true
                  },
                  {
                     "name":"float",
                     "originalType":"float"
                  },
                  {
                     "name":"longtext",
                     "originalType":"longtext"
                  }
               ],
               "rows":[
                  {
                     "newColumns":[
                        {
                           "dataType":13,
                           "charset":"utf8",
                           "bv":"dG1w"
                        },
                        {
                           
                        },
                        {
                           
                        },
                        {
                           "dataType":10,
                           "sv":"1"
                        },
                        {
                           "dataType":3,
                           "sv":"3"
                        },
                        {
                           
                        },
                        {
                           
                        }
                     ]
                  }
               ]
            }
         },
         "header":{
            "sourceType":1,
            "messageType":2,
            "timestamp":1648109734,
            "serverId":109741,
            "fileName":"mysql-bin.000005",
            "position":2234587,
            "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:16",
            "schemaName":"dts",
            "tableName":"dts_mysql",
            "seqId":16017,
            "isLast":true
         },
         "eb_consumer_time":"2022-03-24 16:15:34.287359965 +0800 CST m=+1120.357657669",
         "eb_connector":""
      }
    }
    

    DML operation example

    {
      "id":"38cecd93-a9c2-11ec-b952-******8da53:19",
      "type":"dts:MYSQL:DDL",
      "specversion":"1.0",
      "source":"dts.cloud.tencent",
      "subject":"cdb-xxxx",
      "time":1648110060,
      "region":"ap-guangzhou",
      "datacontenttype":"application/json;charset=utf-8",
      "tags":null,
      "data":{
         "topic":"topic-subs-aniwxeewm4-cdb-xxxx",
         "partition":0,
         "offset":16065,
         "partition_seq":16066,
         "event":{
            "ddlEvent":{
               "schemaName":"dts",
               "sql":"ALTER TABLE `dts_mysql` ADD COLUMN `t` tinyint (0) NULL , ADD UNIQUE `t` USING btree (`t`)"
            }
         },
         "header":{
            "sourceType":1,
            "messageType":3,
            "timestamp":1648110060,
            "serverId":109741,
            "fileName":"mysql-bin.000005",
            "position":2235430,
            "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:19",
            "seqId":16087,
            "isLast":true
         },
         "eb_consumer_time":"2022-03-24 16:21:01.19682088 +0800 CST m=+1447.267118604",
         "eb_connector":""
      }
    }
    

    Parameter description

    Parameter Description
    id Event ID, which is automatically generated by EventBridge. It is the unique ID of an event in EventBridge.
    type Event type in three-segment format. For a DTS connector, the format is dts:${Database type}:${Operation type}.
    specversion CloudEvents version, which is 1.0 by default. The value of this parameter is automatically generated by EventBridge.
    source Event source. For a DTS connector, the value is dts.cloud.tencent.
    subject Instance on which an event is generated. For a DTS connector, the value of this parameter is the ID of the database instance bound to the data subscription.
    time Time when an event is delivered to EventBridge.
    region Region where an event is generated.
    tags Resource tags.
    data Binlog content of the database.
    data.topic Information of the data subscription instance.
    data.partition Consumer partition.
    data.offset Consumer offset.
    data.event There are two types: dmlEvent and ddlEvent. dmlEvent describes the data table schema format and updates. ddlEvent describes specific SQL operations.
    data.header Operation log header information, including the database name, table name, and update timestamp.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support