- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- RocketMQ 5.x
- RocketMQ 4.x
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- SDK Documentation
- Migration to Cloud
- API Documentation
- FAQs
- TDMQ FOR ROCKETMQ SERVICE LEVEL AGREEMENT
- Contact Us
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- RocketMQ 5.x
- RocketMQ 4.x
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- SDK Documentation
- Migration to Cloud
- API Documentation
- FAQs
- TDMQ FOR ROCKETMQ SERVICE LEVEL AGREEMENT
- Contact Us
Message Service Data Flow Migration
Last updated: 2024-01-18 10:07:12
Overview
In a metadata migration task, you can sync the metadata of a self-built RocketMQ cluster to TDMQ for RocketMQ. After the sync, you need to modify the access information of the producer and consumer clusters in order to migrate them from the self-built cluster to TDMQ for RocketMQ for message sending and receiving.
Note:
Currently, the message migration service migrates only message production and consumption linkages but not message data in the old RocketMQ cluster. It applies only to exclusive clusters as the destination and will be supported for virtual clusters after the beta test ends.
Migration Directions
This document describes the double-read double-write and batch release schemes of the message migration service. During migration, the producer and consumer clusters can produce or consume messages in the old RocketMQ and new TDMQ for RocketMQ clusters in parallel. Data will not be heaped because of the migration, so the business can transition smoothly.
The detailed directions are as follows:
1. Create a TDMQ for RocketMQ cluster, migrate the metadata, and get the required client information in the console, such as the access point of the new cluster,
AccessKey, and SecretKey.2. Modify the access information of certain nodes in the consumer cluster to connect corresponding consumers to the new TDMQ for RocketMQ cluster. They will consume messages in the new cluster, while the rest will continue to consume messages in the old cluster.
3. Modify the access information of certain nodes in the producer cluster to connect corresponding producers to the new TDMQ for RocketMQ cluster. They will send messages to the new cluster, while the rest will continue to send messages to the old cluster. To avoid message repetition or loss, implement the idempotency logic for message consumption in advance.
4. Connect the remaining producers to the new TDMQ for RocketMQ cluster. Then, all messages will be sent to the new cluster.
5. Check whether there are heaped messages that are not consumed in the old RocketMQ cluster, and if not, connect the remaining consumers to the new TDMQ for RocketMQ cluster. At this point, the migration is completed.
Note:
If you don't follow the above steps strictly, for example, if you switch producers first and then consumers, message loss may occur.
Before switching the remaining consumers, make sure that all messages in the old RocketMQ cluster have been consumed; otherwise, some messages may not be consumed. You can view the number of heaped messages in the old cluster to check whether consumption has completed.
Migration process diagram
Possible Issues
Sequence
Message sequence cannot be guaranteed due to cluster switch.
Message repetition
Message repetition occurs only in extreme cases. For example, if a consumer consumes a message but doesn't send an acknowledgment to the server (the old RocketMQ cluster), the message will be put into the retry queue, causing repeated consumption. Implementing the idempotency logic can avoid this issue.
Consumption delay
During the read switch, partitions are reallocated with rebalancing between the queue and consumer client. This may cause a short consumption delay, but you don't need to handle it because it won’t persist after the switch.
Yes
No
Was this page helpful?