Through a metadata migration task, you can synchronize the metadata of your self-built RocketMQ cluster to TDMQ for RocketMQ. After completing the metadata migration, you need to modify the access information of the producer and consumer clusters and migrate them from the self-built RocketMQ cluster to TDMQ for RocketMQ. Ultimately, this enables all message sending and receiving operations to be conducted on the new RocketMQ cluster.
Note
In message service migration, only the message production and consumption linkages are migrated, and the message data on the original RocketMQ cluster is not migrated. Only migration to exclusive clusters is supported. Migration to shared clusters will be supported after the open beta ends.
Migration Steps
This document mainly introduces the method of migrating message services using the dual-read dual-write and phased release approaches. During the migration process, both producer and consumer clusters can simultaneously produce and consume messages on the original RocketMQ cluster and the new RocketMQ cluster. This ensures no data backlog occurs due to migration, enabling a smooth business transition. The following diagram shows the solution and operation steps.
The operation steps are as follows:
1. Create a TDMQ for RocketMQ cluster, migrate the metadata, and obtain the required client information in the console, such as the access point, AccessKey, and SecretKey of the new cluster.
2. Modify the access information of certain nodes in the consumer cluster to connect corresponding consumers to the new TDMQ for RocketMQ cluster. They will consume messages in the new cluster, while the remaining consumers will continue to consume messages in the original RocketMQ cluster.
3. Modify the access information of certain nodes in the producer cluster to connect corresponding producers to the new TDMQ for RocketMQ cluster. They will send messages to the new cluster, while the remaining producers will continue to send messages to the original RocketMQ cluster. To prevent message duplication or loss, implement the idempotency logic for message consumption in advance.
4. Connect the remaining producers to the new TDMQ for RocketMQ cluster. Then, all messages will be sent to the new cluster.
5. Check whether there are backlogged messages that are not consumed in the original RocketMQ cluster, and if not, connect the remaining consumers to the new TDMQ for RocketMQ cluster. At this point, the migration is completed.
Note
If you do not follow the above steps strictly, for example, if you switch producers first and then consumers, message loss may occur.
Before you switch the remaining consumers, make sure that all messages in the original RocketMQ cluster have been consumed. Otherwise, message loss may occur. You can view the message backlog in the original RocketMQ cluster to check whether message consumption has completed.
Migration Process Diagram
Possible Issues
Out-of-Order
Due to the cluster switch, the order of messages cannot be guaranteed during the switch process. The switch may cause partial out-of-order.
Message Duplication
Theoretically, duplication should not occur, but it may happen in extreme cases. For example, during the switch, a consumer consumes a message but has not yet sent an acknowledgment to the server (the original RocketMQ cluster). This causes the message to enter the retry queue, leading to duplicate consumption. Implementing idempotency processing logic for messages can avoid this issue.
Consumption Delay
During the read-write switch process, the reassignment of partitions requires rebalance between queues and consumer clients, which may cause temporary consumption delays. No additional action is required if this occurs. Normal consumption will resume after the switch is complete.
