tencent cloud

Migration Scheme Overview
Last updated: 2025-09-29 15:39:05
Migration Scheme Overview
Last updated: 2025-09-29 15:39:05

Overview

This document introduces feasible solutions to migrate self-built or other cloud vendors' Kafka clusters to Tencent Cloud CKafka clusters. You can choose an appropriate scheme according to your own business scenarios.
Currently, two migration solutions are mainly provided:
1. Use a connector: Use the connector functionality provided by CKafka to achieve migration, supporting various scenarios such as cloud, on-premises (cross-cloud, hybrid cloud). For a detailed introduction, refer to Configure data sync between Kafka clusters.
2. Use open-source tools: Use open-source tools to complete migration. The following context will focus on the specific operation steps of this solution.

Open-Source Tool Migration Scheme Description

Scheme 1. Single-Producer dual-consumer

This scheme is simple, clear, and easy to implement, with no data heap for smooth transition.



Scheme ideas:
1. Migrate the topic metadata.
2. Keep the original consumer in the self-built Kafka cluster intact.
3. Start a new consumer in CKafka and configure the bootstrap-server of the new CKafka cluster for consumption.
4. Wait for all consumers to listen on the new CKafka cluster.
5. Switch the production of the self-built cluster to the new CKafka cluster (by configuring the bootstrap-server of the new CKafka cluster).
6. Deactivate the original consumer in the self-built Kafka cluster after it finishes consuming the remaining data in the cluster.
Pros and cons of the scheme:
Pros: the overall migration process is simple, clear, and easy to implement, with no data heap for smooth transition.
Cons: an additional consumer is required.

Scheme 2. Single-Producer single-consumer

This scheme is simple, clear, and easy to implement.



Scheme ideas:
1. Migrate the topic metadata.
2. Switch the production of the self-built Kafka cluster to the new CKafka cluster (by configuring the bootstrap-server of the new CKafka cluster).
3. Wait for the consumer in the self-built cluster to consume the remaining data.
4. Switch the old consumer to the new CKafka cluster for consumption (by configuring the bootstrap-server of the new CKafka cluster).
Pros and cons of the scheme:
Pros: the overall migration process is simple, clear, and easy to implement for smooth transition.
Cons: after the production is switched to the CKafka cluster, before the old consumer is switched to the CKafka cluster, there will be a certain amount of data heap in the CKafka cluster.

Scheme 3. Migration with MirrorMaker

This scheme migrates the existing data in the self-built Kafka cluster to CKafka.



Scheme ideas:
1. Migrate the topic metadata.
2. Keep the original consumer in the self-built Kafka cluster intact.
3. Start the data sync feature of MirrorMaker.
4. Wait for the data sync to complete, modify the consumer configuration, and switch the consumer.
5. Wait for the data sync to complete, modify the producer configuration, and switch the producer.
6. The migration is completed.
Pros and cons of the scheme:
Pros: the overall migration process is simple, clear, and easy to implement, where historical data can be synced to the CKafka cluster.
Cons: the consumer needs to consume data from the beginning after switch to the destination cluster, so consumption idempotency must be ensured.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback