This document describes how to create a data subscription task in DTS for TencentDB for MongoDB.
Version description
Data subscription is supported for TencentDB for MongoDB 3.6, 4.0, 4.2, and 4.4.
TencentDB for MongoDB 3.6 only supports collection-level subscription.
Prerequisites
You have prepared a TencentDB instance to be subscribed to, and the database version meets the requirements. For more information, see Databases Supported by Data Subscription.
We recommend that you create a read-only account in the source instance by referring to the following syntax. For operations in the console, see Account Management.
# Create an instance-level read-only account
use admin
db.createUser({
user: "username",
pwd: "password",
roles:[
{role: "readAnyDatabase",db: "admin"}
]
})
# Create a database-specific read-only account
use admin
db.createUser({
user: "username",
pwd: "password",
roles:[
{role: "read",db: "Name of the specified database"}
]
})
Restrictions
Currently, the subscribed message content is retained for 1 day by default. Once expired, the data will be cleared. Therefore, you need to consume the data promptly.
The region where the data is consumed should be the same as that of the subscribed instance.
The Kafka built in DTS has a certain upper limit for the size of processed individual messages. When a single row of data in the source database exceeds 10 MB, this row may be discarded in the consumer.
If the subscribed database or collection specified in the data subscription task is deleted from the source database, the subscription data (change stream) of the database or collection will be invalidated. Even if the database or collection is rebuilt in the source database, the subscription data cannot be resubscribed. In this case, you need to reset the subscription task and select the subscription object again.
1. Log in to the DTS console, select Data Subscription on the left sidebar, and click Create Subscription.
2. On the DTS page, configure parameters as follows and click Buy Now.
Parameter
Description
Configuration Method
Service Type
Select the service type. This document describes the Data Subscription service.
Select Data Subscription.
Billing Mode
Select the billing mode of the service. For billing details, see Billing Overview.
Pay-as-you-go is supported. For more information, see
Billing Overview.
Region
Select the region where the subscription service resides.
It must be the same as that of the database instance to be subscribed to.
Database
Select the type of the database for the data subscription service.
Select MongoDB.
Version
Data can be directly consumed through the Kafka client.
Select Kafka Edition.
Tag
Specify the tag for the data subscription service.
Click Add and select Tag Key and Tag Value in the drop-down list.
Subscribed Instance Name
Specify how to name the data subscription service.
Name after Creation: Set the name after the data subscription service is created, which is name-subscription ID by default. The subscription ID is randomly assigned by the system.
Name Now: Enter the name of the data subscription service directly in the input box below.
Quantity
Select the number of tasks to be purchased.
You can purchase up to 10 tasks at a time.
3. After successful purchase, return to the data subscription list, select the purchased subscription task, and click Configure Subscription in the Operation column.
4. On the Select Instance tab of the Configure data subscription wizard, configure the database information for the data subscription task and perform the connectivity test.
Parameter
Description
Configuration Method
Subscription ID / Name
ID and name of the subscription task. The task name is name-subscription ID by default.
Confirm the ID and name of the data subscription task.
Instance Type
It is MongoDB by default.
-
Region
The region where the subscription service resides.
Confirm the region.
Access Type
Select the type of source database access to the data subscription service.
Currently, only Database is supported, i.e., a TencentDB instance.
Instance Name
Select the specific MongoDB instance for the data subscription service.
Select a specific instance ID in the drop-down list.
Account
Set the access account information of the MongoDB instance.
Enter the prepared read-only account information in the input box.
Password
Set the password of the access account of the MongoDB instance.
Enter the password of the read-only account in the input box. Password-free access is not supported.
Kafka Partition Count
Select the number of Kafka partitions for the data subscription task. In Kafka, a consumer can get data by subscribing to one or more topics and then consuming data from one or more partitions of each topic.
You can select 1, 4, or 8.
A single partition can guarantee the order of messages, while multiple partitions cannot. If you have strict requirements for the order of messages during consumption, set this value to 1.
Increasing the number of partitions can improve the throughput and parallelism of the Kafka cluster, because multiple consumers can consume different partitions at the same time. However, doing so will also increase the management and maintenance costs of the Kafka cluster and may cause data imbalance or delays.
Test Connectivity
Test the connectivity between the data subscription service and the TencentDB for MongoDB instance.
Click Test Connectivity and wait for the test result.
If the test fails, troubleshoot as prompted. Then, click Test Again to test the connectivity again. After the test passes, proceed to the next step.
5. Click Next, configure the parameters as follows on the Subscription Type and Object page, and click Save.
Parameter
Description
Configuration Method
Subscription ID / Name
ID and name of the subscription task.
Check whether the subscription task information is correct.
Subscribed Instance
Instance ID of the subscribed MongoDB database.
Check whether the instance information is correct.
Data Subscription Type
The type of data that the subscriber can choose to subscribe to. MongoDB uses the change stream feature to monitor data changes and implement data subscription.
It is Change Stream by default and cannot be modified.
Subscription Object Level
Level of the subscribed data, including Full instance, Database, and Collection.
Full instance: Subscribe to the data in the entire instance.
Database: Subscribe to the data in the specified database. After selecting this option, you can select only one database in Task Configuration.
Collection: Subscribe to the data in the specified collection. After selecting this option, you can select only one collection in Task Configuration.
Select the level of data subscription as needed. System databases admin, local, and config cannot be selected.
Task Configuration
When Subscription Level is Database or Collection, this parameter will be displayed for you to specify the database or collection.
In the Select databases and tables section, select the database or collection to be subscribed.
In the Selected section, check whether the selected database or collection is correct.
Output Aggregation Settings
This parameter configures whether to enable aggregation for the subscribed data before sending it to the subscriber.
Enable: Click
to enable aggregation. Click Add, select an operator in the Aggregation Operator drop-down list, and enter an expression in the Aggregation Expression input box.
Click Add to add multiple aggregation expressions. The aggregation pipeline will be executed based on the order of added aggregation operations. For more information, see Modify Change Stream Output.
Kafka Partitioning Policy
If Kafka Partition Count is not 1 in the previous step, you need to set the partition policy.
By Collection Name: Partitions the subscribed data from the source database by collection name. With this policy, data with the same collection name is written to the same Kafka partition.
Custom Partitioning Policy: Database and collection names of the subscribed data are matched through a regex first. Then, matched data is partitioned by collection name or collection name + objectid.
Select Custom Partitioning Policy, click Add in Custom Partitioning Policy below, set the matching mode of database name or collection name in the form of regular expression in the Database Name Match or Table Name Match input box below, and select By Collection Name or By Collection Name + ObjectId in the Partitioning Policy drop-down list.
When you enable the custom partitioning policy option, your custom partitioning policies will be applied first, followed by the Kafka partitioning policies.
The data in a collection that cannot be partitioned using the custom partitioning policies will be routed to Kafka partitions by default policy By Collection Name.
Custom Partitioning Policy
This parameter will be displayed if Custom Partitioning Policy is selected in Kafka Partitioning Policy. It sets the custom partitioning policy.
Policy Combo Result
This parameter will be displayed if Custom Partitioning Policy is selected in Kafka Partitioning Policy. It indicates the combo result of the custom partitioning policy.
6. On the Pre-verification page, a pre-verification task will run for 2–3 minutes. After the pre-verification is passed, click Start to complete data subscription task configuration.
Note:
If the verification fails, fix the problem as instructed in Database Connection Check and initiate the verification again.
7. The subscription task will be initialized, which will take 3–4 minutes. After successful initialization, the task will enter the Running status, and data consumption will start.
Subsequent operations
1. Adding Consumer Group.
The consumption in data subscription (Kafka Edition) depends on the consumer groups of Kafka; therefore, you must create a consumer group first before data can be consumed. Data subscription (Kafka Edition) allows you to create multiple consumer groups for multi-point consumption.
Yes
No
Was this page helpful?