Object Storage Migration
Object storage migration is a cloud data migration service and tool provided by Tencent Cloud. It enables you to easily migrate data from third-party data sources to Tencent Cloud COS buckets or migrate data between different COS buckets.
Supported Data Sources
Alibaba Cloud OSS, AWS S3 (international), Azure Blob, HTTP URL list, storage systems supporting S3 protocol (such as Ceph, Minio, GCP), and Tencent Cloud COS.
Migration Mode
Fully-Managed Migration
In the fully-managed migration mode, you do not need to manually deploy an agent. The migration is performed automatically after you create a task in the console. Object storage migration will pull data from the origin server of third-party data sources via the public network and store the data in Tencent Cloud COS buckets. If it is migration between COS storage buckets, pull the source data through the private network to transmit. For the task creation process, see the specific instructions on the data source of each service provider.
Semi-managed Migration
In the semi-managed migration mode, you need to create a task in the console and manually deploy a migration agent on your own server. The agent first obtains the task you have created using a fixed network request, pulls source data over the network, and uploads the data to Tencent Cloud COS buckets. At the same time, the agent updates the progress in real time in the console, allowing you to monitor the task progress just like in fully-managed migration mode. For the task creation process, see the instructions on the data source of each service provider. For the deployment and usage of the agent, see Using Semi-Managed Migration Agent. Notes
In the fully-managed migration mode, public network outbound traffic fees are incurred for third-party data sources. For specific fees, see the pricing of the storage service provider. To migrate between Tencent Cloud COS buckets, see Cross-Bucket Replication(supports replicating incremental data only). You can also contact us to design a migration plan. In the semi-managed migration mode, corresponding traffic fees may be incurred depending on the server where the agent is deployed. Traffic fees will not be charged if a direct connection is established between the data source and Tencent Cloud COS. Therefore, we recommend that you use this mode if a direct connection is already established.
In the semi-managed migration mode, both computing and network resources are dedicated. You can accurately estimate the migration speed and resource usage. This mode applies to scenarios where the data volume is large, the time for migration is urgent, or a direct connection has been established.
The resources used in the fully-managed migration mode are shared by all migration tasks, and the migration speed and resource usage cannot be accurately estimated. This mode applies to scenarios where the data volume is small or the time for migration is sufficient.
Data Verification
All data undergoes a simple verification to check if the data sizes are consistent.
If the header of the source file carries a CRC-64 or Content-MD5 checksum value, it will be checked on both the source and destination files for the consistency.
If no such checksum value is used, the source and destination files will be sampled for double check.
Transmission Security
Supported SSL/TLS-based HTTPS encrypted transfer to prevent security risks during data migration.
File Migration Time Estimation
The overall migration speed is subject to the lowest speed during each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. Factors are as follows:
|
Read speed of the migration source | The read speed of the data source varies depending on the service provider. Generally: The speed ranges from 50 Mbps to 20,000 Mbps. The file read QPS ranges from 500 to 5,000. (The transmission of a large number of small files is affected by the concurrency level.) |
Transmission speed of Cloud Migration | Cloud Migration provides a migration bandwidth of up to 2,000 Mbps in the fully-managed migration mode. In the semi-managed migration mode, the transmission speed depends on the migration resource specifications provided by users. |
Write speed of the migration destination | The default write speed of Tencent Cloud COS buckets is 15,000 Mbps, and the write QPS ranges from 3,000 to 30,000. |