Release Notes
Announcements
Cross-Cluster Message Replication
Scenarios
By configuring inter-cluster message replication tasks, TDMQ for RocketMQ enables customers to synchronize messages between two clusters (within the same region or across different regions). Message replication tasks can be unidirectional replication (all messages from Cluster A flow to Cluster B) or bidirectional replication (messages between Cluster A and Cluster B are mutually synchronized, with the server automatically preventing circular replication). After a task is created, the console allows you to start or stop replication tasks at any time, and supports monitoring the progress and health status of replication tasks.
Note:
After a cluster-level message replication task is configured, such as replicating all message data from Cluster A to Cluster B, all metadata (Topics, Groups, roles, and so on) of Cluster A will be synchronized to Cluster B. Additionally, any subsequent metadata operations in Cluster A—including additions, modifications, and deletions—will be synchronized to Cluster B, and vice versa. If a bidirectional replication task is configured, any operation performed in either cluster will be synchronized to the other cluster.
The traffic for cross-cluster replication is counted towards the cluster's TPS specification. Therefore, before creating a message replication task, you need to properly plan the cluster's TPS capacity. When the cluster specification is exceeded, Traffic throttling will occur for replicated messages. You can monitor relevant metrics and promptly upgrade the cluster if throttling occurs.
Currently, only Tencent Cloud RocketMQ 4.x general clusters are supported. Support for RocketMQ 5.x clusters will be available soon.
Billing Rules
The cross-cluster message replication feature is currently free of charge. Before charging starts, Tencent Cloud will notify customers multiple times a month in advance through the message center, short messages, and emails.
Operation Steps
Creating a Task
Go to the TDMQ for RocketMQ console, click Cross-Cluster Replication > New Task in the left sidebar, and fill in the following fields as required:
Task Type: Select Cluster Cross-Cluster Replication.
Task Name: It must be up to 200 characters in length and can contain only Chinese characters, digits, letters, hyphens (-), and underscores (_).
Replication Direction: Select unidirectional replication or bidirectional replication based on actual scenarios and requirements;
Source Cluster: Fill in the relevant information of the source cluster.
Cluster Type: If you select Tencent Cloud, you can select the region and cluster from the drop-down lists in sequence (currently only 4.x general clusters are supported; support for 5.x clusters will be available soon). If you select the self-built cluster type, you need to fill in the following fields:
Network Connection Type: This applies only to self-built clusters. Tencent Cloud clusters establish connectivity via VPC networks by default for migration. For self-built clusters, choose between public network and VPC network based on your actual scenario:
If you select the VPC network for message replication, you need to create a VPC network in the corresponding Tencent Cloud region first. This VPC network will be used to establish network connectivity. After selecting the VPC network, accurately fill in the following information:
Region: Select the region where the VPC is pre-created on Tencent Cloud.
VPC: Select a pre-created VPC network and subnet. If no suitable VPC network is available, you can go to the VPC console to create a new one.
Resource Type: TDMQ for RocketMQ supports message replication between self-built RocketMQ instances from Tencent Cloud and other cloud providers. Select based on the actual location of the RocketMQ instance. To enable message exchange with self-built clusters on Tencent Cloud, select from Tencent Cloud. In subsequent steps, you only need to fill in the Cluster ID, Name Server address, and ACL configuration. Tencent Cloud TDMQ for RocketMQ will automatically establish VPC network connectivity on Tencent Cloud. If you select from other cloud providers, in addition to the Cluster ID, Name Server address, and ACL configuration, you need to select the dedicated line connection type and the corresponding resource ID to establish network connectivity.
Cluster ID: Fill in the accurate cluster ID.
Dedicated Line Connection Type: This field is required only when the cluster resources are from other cloud providers. If you need to establish connectivity via VPC networks with other cloud providers, currently only CCN and Direct Connect Gateway are supported.
Dedicated Connect Gateway ID/CCN ID: Enter the currently effective dedicated connect gateway ID/CCN ID. You can find the specific ID in the Direct Connect Gateway console or CCN console. Entering an incorrect or invalid ID will result in failure in establishing network connectivity.
Name Server address: You can enter a fixed public IP address or a domain name accessible via the public network. If there are multiple IP addresses, separate them with ";".
Enable ACL for Source Cluster: If ACL is enabled, you need to accurately fill in the AK and SK with Admin privileges to ensure successful message replication.
If you select the public network for message replication, you must ensure that both the source and target clusters have enabled public network access points and have sufficient bandwidth to support the message traffic for replication.
After the public network is selected, please accurately fill in the following information:
Name Server address: You can enter a fixed public IP address or a domain name accessible via the public network. If there are multiple IP addresses, separate them with ";".
Cluster ID: Fill in the accurate cluster ID.
Enable ACL for Source Cluster: If ACL is enabled, you need to accurately fill in the AK and SK with Admin privileges to ensure successful message replication.
Target cluster: The filling rules are the same as those for the "source cluster". For inter-cluster replication tasks, at least one of the source cluster and target cluster must be Tencent Cloud TDMQ for RocketMQ.
Replication Start Offset: Currently, only replication starting from the latest offset is supported.
After you click Create Task, the Task List page is displayed. After the task is initialized, it has been created.
Managing Tasks
After the inter-cluster replication task is created, you can perform basic task status viewing and task management on the Task List page.
By default, a created task is in the "Running" state. For tasks in the "Running" state, editing, deletion, and resetting the consumer offset are not allowed. To perform these operations, you need to pause the task first.
Start: Restart a task in the "Stopped" state to continue message synchronization between clusters.
Edit: Modify task information. Currently, only the task name can be modified. To ensure stable operation and avoid frequent changes, other task configurations cannot be modified at this time.
Delete: Delete the current task. After deletion, the replication task will be suspended, and the relevant configurations will be removed. Please exercise caution. Once the task is deleted, modifications to metadata (such as Topics, Groups, and roles) made in the source cluster will no longer be replicated to the target cluster.
Reset Consumer Offset: After pausing the task, you can choose to restart message replication from a new timestamp.
Viewing Task Details
You can click the task name to go to the Task Details page and view the detailed configurations of the task.
In the monitoring section, you can view real-time monitoring of the current message replication task, such as the total number of source messages consumed, the number of message replication failures, and message synchronization delay.
In the metadata synchronization section, you can view the metadata information synchronized from the source cluster to the target cluster.
Handling Exceptions
Normally, the status bar displays Running or Paused. If the status is Start Failed, you need to check the task running status and whether the detailed task configurations are correct, such as whether the SQL expression is valid. Hovering the cursor over the failed status will show specific failure reasons.
If the task status is failed, you can click Edit in the operation column, or go to the Task Details page and click Edit in the upper-right corner of the Basic Information section to correct the task information.
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback