Consuming TDSQL for PostgreSQL Data

Recent Pages

Consuming TDSQL for PostgreSQL Data

Last updated: 2023-11-21 20:17:30

Overview
In data subscription (Kafka Edition, where the current Kafka server version is v2.6.0), you can consume the subscribed data through Kafka 0.11 or later available at DOWNLOAD. This document provides client consumption demos for you to quickly test the process of data consumption and understand the method of data parsing.
Note
The demo only prints out the consumed data and contains no instructions. You need to write your own data processing logic based on the demo. You can also use Kafka clients in other languages to consume and parse data.
Currently, data subscription to Kafka for consumption can be implemented over the Tencent Cloud private network but not the public network. In addition, the subscribed database instance and the data consumer must be in the same region.
In scenarios where only the specified databases and tables (part of the source instance objects) are subscribed and single-partition Kafka topics are used, only the data of the subscribed objects will be written to Kafka topics after DTS parses the incremental data. The data of non-subscription objects will be converted into empty transactions and then written to Kafka topics. Therefore, there are empty transactions during message consumption. The BEGIN and COMMIT messages in the empty transactions contain the GTID information, which can ensure the GTID continuity and integrity.
To ensure that data can be rewritten ‍from where the task is paused, DTS adopts the checkpoint mechanism for data subscription. Specifically, when messages are written to Kafka topics, a checkpoint message is inserted every 10 seconds to mark the data sync offset. When the task is resumed after being interrupted, data can be rewritten from the checkpoint message. The consumer commits a consumption offset every time it encounters a checkpoint message so that the consumption offset can be updated timely.
Downloading Consumption Demos
Currently, TDSQL for PostgreSQL data subscription only supports the Protobuf format. The following demos already contain the ProtoBuf protocol file, so you don't need to download it separately. If you download it on your own, use Protobuf 3.X for code generation to ensure that data structures are compatible.
The logic description of the consumption demo for TDSQL for PostgreSQL is similar to that for MySQL. For more information, see Demo Description.
Demo Language
Protobuf Demo Download
Go
Address
Java
Address
Python
Address
Directions for the Java Demo
Compiling environment: The package management tool Maven or Gradle, and JDK8. You can choose a desired package management tool. The following takes Maven as an example.
Runtime environment: Tencent Cloud CVM (which can access the private network address of the Kafka server only if it is in the same region as the subscribed instance). Install JRE 8.
Follow the steps below:
1. Create a data subscription task (NewDTS) as instructed in Creating TDSQL for PostgreSQL Data Subscription.
2. Create one or multiple consumer groups as instructed in Adding Consumer Group.
3. Download the Java demo and decompress it.
4. Access the decompressed directory. Maven model and pom.xml files are placed in the directory for your convenience.
Package with Maven by running mvn clean package.
5. Run the demo.
After packaging the project with Maven, go to the target folder target and run the following code: java -jar consumerDemo-avro-1.0-SNAPSHOT.jar --brokers xxx --topic xxx --group xxx --user xxx --password xxx --trans2sql
brokers is the private network access address for data subscription to Kafka, and topic is the subscription topic, which can be viewed on the Subscription details page as instructed in Viewing Subscription Details.
group, user, and password are the name, account, and password of the consumer group, which can be viewed on the Consumption Management page as instructed in Managing Consumer Group.
trans2sql indicates whether to enable conversion to SQL statement. In Java code, if this parameter is carried, the conversion will be enabled.
Note
If trans2sql is carried, javax.xml.bind.DatatypeConverter.printHexBinary() will be used to convert byte values to hex values. You should use JDK 1.8 or later to avoid incompatibility. If you don't need SQL conversion, comment this parameter out.
6. Observe the consumption.
﻿
Directions for the Golang Demo
Compiling environment: Go 1.12 or later, with the Go module environment configured.
Runtime environment: Tencent Cloud CVM (which can access the private network address of the Kafka server only if it is in the same region as the subscribed instance).
Follow the steps below:
1. Create a data subscription task (NewDTS) as instructed in Creating MySQL or TDSQL for MySQL Data Subscription.
2. Create one or multiple consumer groups as instructed in Adding Consumer Group.
3. Download the Go demo and decompress it.
4. Access the decompressed directory and run go build -o subscribe ./main/main.go to generate the executable file subscribe.
5. Run ./subscribe --brokers=xxx --topic=xxx --group=xxx --user=xxx --password=xxx --trans2sql=true.
brokers is the private network access address for data subscription to Kafka, and topic is the subscription topic, which can be viewed on the Subscription details page as instructed in Viewing Subscription Details.
group, user, and password are the name, account, and password of the consumer group, which can be viewed on the Consumption Management page as instructed in Managing Consumer Group.
trans2sql indicates whether to enable conversion to SQL statement.
6. Observe the consumption.
﻿
Directions for the Python3 Demo
Compiling and runtime environment: Tencent Cloud CVM (which can access the private network address of the Kafka server only if it is in the same region as the subscribed instance). Install Python 3 and pip3 (for dependency package installation).
Use pip3 to install the dependency package:
pip install flag
pip install kafka-python
pip install avro
Follow the steps below:
1. Create a data subscription task (NewDTS) as instructed in Creating MySQL or TDSQL for MySQL Data Subscription.
2. Create one or multiple consumer groups as instructed in Adding Consumer Group.
3. Download Python3 demo and decompress it.
4. Run python main.py --brokers=xxx --topic=xxx --group=xxx --user=xxx --password=xxx --trans2sql=1.
brokers is the private network access address for data subscription to Kafka, and topic is the subscription topic, which can be viewed on the Subscription details page as instructed in Viewing Subscription Details.
group, user, and password are the name, account, and password of the consumer group, which can be viewed on the Consumption Management page as instructed in Managing Consumer Group.
trans2sql indicates whether to enable conversion to SQL statement.
5. Observe the consumption.
﻿