Cloud Migration

COS Migration Guide

Product Documentation

Copyright Notice

©2013-2025 Tencent Cloud. All rights reserved.

Copyright in this document is exclusively owned by Tencent Cloud. You must not reproduce, modify, copy or distribute in any way, in whole or in part, the contents of this document without Tencent Cloud's the prior written consent.

Trademark Notice

All trademarks associated with Tencent Cloud and its services are owned by the Tencent corporate group, including its parent, subsidiaries and affiliated companies, as the case may be. Trademarks of third parties referred to in this document are owned by their respective proprietors.

Service Statement

This document is intended to provide users with general information about Tencent Cloud's products and services only and does not form part of Tencent Cloud's terms and conditions. Tencent Cloud's products or services are subject to change. Specific products and services and the standards applicable to them are exclusively provided for in Tencent Cloud's applicable terms and conditions.

Contents

COS Migration Guide

Overview

Last updated：2024-10-22 18:09:27

Object Storage Migration

Object storage migration is a cloud data migration service and tool provided by Tencent Cloud. It enables you to easily migrate data from third-party data sources to Tencent Cloud COS buckets or migrate data between different COS buckets.

Supported Data Sources

Alibaba Cloud OSS, AWS S3 (Global), Azure Blob, HTTP URL list, self-built storage services supporting S3 protocol, and Tencent Cloud COS.

Migration Mode

Fully-Managed Migration

In the fully-managed migration mode, you do not need to manually deploy an agent. The migration is performed automatically after you create a task in the console. Object storage migration will pull data from the origin server of third-party data sources via the public network and store the data in Tencent Cloud COS buckets. For the task creation process, see the specific instructions on the data source of each service provider.

Semi-managed Migration

In the semi-managed migration mode, you need to create a task in the console and manually deploy a migration agent on your own server. The agent first obtains the task you have created using a fixed network request, pulls source data over the network, and uploads the data to Tencent Cloud COS buckets. At the same time, the agent updates the progress in real time in the console, allowing you to monitor the task progress just like in fully-managed migration mode. For the task creation process, see the instructions on the data source of each service provider. For the deployment and usage of the agent, see Using Semi-Managed Migration Agent.

Notes

1. In the fully-managed migration mode, public network outbound traffic fees are incurred for third-party data sources. For specific fees, see the pricing of the storage service provider. To migrate between Tencent Cloud COS buckets, see Cross-Bucket Replication(supports replicating incremental data only). You can also contact us to design a migration plan.
2. In the semi-managed migration mode, corresponding traffic fees may be incurred depending on the server where the agent is deployed. Traffic fees will not be charged if a direct connection is established between the data source and Tencent Cloud COS. Therefore, we recommend that you use this mode if a direct connection is already established.
3. In the semi-managed migration mode, both computing and network resources are dedicated. You can accurately estimate the migration speed and resource usage. This mode applies to scenarios where the data volume is large, the time for migration is urgent, or a direct connection has been established.
4. The resources used in the fully-managed migration mode are shared by all migration tasks, and the migration speed and resource usage cannot be accurately estimated. This mode applies to scenarios where the data volume is small or the time for migration is sufficient.

Data Verification

1. All data undergoes a simple verification to check if the data sizes are consistent.
2. If the header of the source file carries a CRC-64 or Content-MD5 checksum value, it will be checked on both the source and destination files for the consistency.
3. If no such checksum value is used, the source and destination files will be sampled for double check.

File Migration Time Estimation

The overall migration speed is subject to the lowest speed during each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. Factors are as follows:
Affecting Factor
Description
Read speed of the migration source
The read speed of the data source varies depending on the service provider. Generally:
The speed ranges from 50 Mbps to 20,000 Mbps.
The file read QPS ranges from 500 to 5,000. (The transmission of a large number of small files is affected by the concurrency level.)
Transmission speed of Cloud Migration
Cloud Migration provides a migration bandwidth of up to 2,000 Mbps in the fully-managed migration mode.
In the semi-managed migration mode, the transmission speed depends on the migration resource specifications provided by users.
Write speed of the migration destination
The default write speed of Tencent Cloud COS buckets is 15,000 Mbps, and the write QPS ranges from 3,000 to 30,000.




Using Semi-Managed Migration Agent

Last updated：2024-10-22 18:09:27

Overview

msp-agent is a tool for migrating data to COS. You can deploy it on a server in your data center or in the cloud. It will execute the semi-managed migration tasks created in the Cloud Migration console, facilitating the easy migration of cloud storage data to COS.

Support Features

Migration from all the source vendors supported in the console.
Easy-to-deploy distributed master-worker architecture for efficient migration of a high volume of data.
Supports Breakpoint Resume.
Supports Traffic Control.
Supports all features that can be entered in the console, in line with fully managed service features.
Supports both push and pull modes. Please contact the Storage Architect to design the migration plan.
Push mode: Deploy the msp-agent on the host nearest to the data source to push data to the COS target bucket.
Pull mode: Deploy the msp-agent on the host nearest to the Tencent Cloud COS target bucket to pull data from the data source and write it into the COS target bucket.
Supports both Dedicated Line and Public Internet networking modes, see Network Traffic.
Supports Private Network Global Acceleration. Enabling the global acceleration feature will incur COS global acceleration traffic costs, which are charged by COS. For more details, see Global Acceleration Overview.

Running environment

Suitable for Linux systems.

System Deployment Method

Installation

Download the msp-agent installation package msp-agent, then decompress it. Here is the directory structure after decompression:


Note:
msp-agent adopts the distributed master-worker architecture, where one master can connect to one or multiple workers.
The 'master' directory is for the master, and the 'worker' directory is for the workers.
If you need to deploy multiple workers, copy the entire 'worker' directory, then modify the relevant parameters before starting them using the method below.
You can start multiple worker processes on a single server. However, you need to modify relevant parameters as detailed below to avoid port conflicts.

Startup

Start the master:
  cd {path-to-msp-agent}/master && ./bin/start.sh
Start the worker:
cd {path-to-msp-agent}/worker && ./bin/start.sh

Configuration Parameter Description

Both the master and worker directories contain the same configs structure:


Among them, pl_config.yaml configures the main parameters for process execution; app_logger_config.yaml configures the application runtime log format; query_logger_config.yaml configures the master-slave RPC communication log format.

Log configuration

You can basically use the default log parameter settings.
Note:
Log Rolling Configuration: If the disk space is limited and the task size is large, you need to adjust the log configuration to save disk space (Migration uses the disk only for storing logs, actual file migration does not use the disk).

Master configuration

Parameter
Definition
Description
"....gRPCPort"
gRPC port for listening on the master
This parameter specifies the port used to receive the information reported by workers. This port must be opened to worker servers on the master server.
failFilePartSize
Part size of a failed file
This parameter specifies the part size of a failed file, which is 10,485,760 bytes by default. The product of this value multiplied by 10,000 is the maximum size of failed files, i.e., 100 GB. If there are both high numbers of files to be migrated and failed files, for example, more than 200 million files, you can increase this value appropriately.
fragMaxSize
Fragment size of an assigned task
To reduce the pressure on master-worker communication, the master packages multiple file paths into a fragment and assigns it to a worker as a subtask. fragMaxSize is the maximum size of files in each fragment in bytes.
If the value is too small, the pressure on master-worker communication will get higher, and server resources will be wasted; if it is too large, workers will become too slow to report the completion of fragment migration and have imbalanced loads.
The default value is 10737418240, i.e., 100 GB.
When fragMaxSize or fragMaxNum reaches the limit during packaging, the system will stop adding more files.
fragMaxNum
Maximum number of files of an assigned task fragment
To reduce the pressure on master-worker communication, the master packages multiple file paths into a fragment and assigns it to a worker as a subtask. fragMaxNum is the maximum number of files in each fragment.
If the value is too small, the pressure on master-worker communication will get higher, and server resources will be wasted; if it is too large, workers will become too slow to report the completion of fragment migration and have imbalanced loads.
The default value is 1,000.
When fragMaxSize or fragMaxNum reaches the limit during packaging, the system will stop adding more files.
secretId
The SecretId used to request TencentCloud APIs for Cloud Migration
As the master process needs to request TencentCloud APIs for Cloud Migration to get the tasks created in the console, you need to enter the secretId of your key.
Note that the key here refers to your key used to create Cloud Migration tasks, not the keys of source and target buckets.

secretKey
The SecretKey used to request TencentCloud APIs for Cloud Migration
As the master process needs to request TencentCloud APIs for Cloud Migration to get the tasks created in the console, you need to enter the secretKey of your key. Note that the key here refers to your key used to create Cloud Migration tasks, not the keys of source and target buckets.
listerIp
Private IP of the server where the master process is deployed
You may create multiple tasks, and want them to run on different clusters. Therefore, you need to enter the Private IP Address of the server where the Master process is deployed. This way, the Master will only run tasks that are assigned to this server's IP during task creation in the Console.
When creating tasks in the Console, enter the same IP in the form for the Master Node Private IP.



useAccelerateDomain
Access the internal COS bucket holding the Failed File List using the Global Acceleration Domain
To avoid the failure of saving the "Failed File List" causing the retry of migration tasks to fail, the Failed File List of tasks is uniformly saved to a bucket in domestic regions. When the msp-agent is deployed overseas, to ensure the task runs without errors, it is recommended to set this configuration to true in the configuration file.

Worker configuration

Parameter
Definition
Description
"....gRPCPort"
gRPC port for listening on the worker
This parameter specifies the port used to receive the scheduling information from the master. This port must be opened to the master server on the worker server. If multiple worker processes run on a single server, you need to change this parameter to a value different from that of other workers, so as to avoid process startup failure due to port conflicts.
fileMigrateTryTimes
Number of retries
This parameter specifies the maximum number of retries after a file fails.
goroutineConcurrentNum
Number of concurrent coroutines
This parameter specifies the number of concurrent coroutines, i.e., the number of files to be migrated concurrently. The value is subject to two factors: server configuration and average file size. The more the server CPU cores, the greater the value can be set. The larger the average file size, the smaller the value can be set. As a small number of concurrent coroutines can lead to a high bandwidth usage, if the average file size is small, you should add more concurrent coroutines to increase the overall bandwidth.
baseWorkerMaxConcurrentFileNum
Queue of cached files to be migrated
To improve the assignment efficiency of distributed tasks, each worker caches a certain number of file fragments to be migrated. If the files to be migrated are small ( that is, the migration QPS is high), you can increase this value to make workers less hungry during consumption. However, the larger the cache, the more the status data that the master needs to store, which will increase the master load and instability. Therefore, you should choose an appropriate value.
partSize
Part size
This parameter specifies the default part size for multipart upload during large file migration.
downloadPartTimeout
Download timeout period
This parameter specifies the file download timeout period in seconds.
uploadPartTimeout
Upload timeout period
This parameter specifies the file upload timeout period in seconds.
perHostMaxIdle
HTTP client concurrency
This parameter specifies the connection pool size per host and generally should be set to the same value as goroutineConcurrentNum.
addr
Private network communication address of the master
This parameter specifies the master communication address of workers, so that workers will register with the master to form a cluster. For example, if the master's listerIp is 10.0.0.1, and the gRPC port to be listened on in the master configuration is 22011, then set addr to 10.0.0.1:22011.
sample
Whether to perform spot check
The overview section describes the cases of data consistency check after migration. If source files don't have a Content-MD5 or CRC-64 checksum, they cannot be directly checked for data consistency. In this case, you can only perform spot check to reduce their probability of inconsistency. If this parameter is set to true, spot check will be performed on such files.
sampleTimes
Number of sampled fragments per file
This parameter specifies the number of sampled fragments of a single file. Each sampled fragment will add one download request and increase the download traffic usage.
sampleByte
Sampled fragment size in bytes
This parameter specifies the size of each sampled fragment. The greater the value, the higher the sampling bandwidth.
useInternalAccDomain
Using the COS private network global acceleration domain
Enabling this configuration as true will use the private network global acceleration domain when uploading files to the target COS bucket, enhancing the performance of data upload to COS; if the source bucket is also a COS bucket, the private network global acceleration domain will be used for downloading as well. Before enabling this configuration, please ensure that both the target COS bucket and the source COS bucket (if the migration source is a COS bucket) have global acceleration enabled.
Note:
Enabling this configuration will incur COS global acceleration traffic costs, which are charged by COS. For more details, see Global Acceleration Overview.

Network traffic

If the data source is a competitor:
Managed mode
Dedicated Line or Public network
Network traffic
Push Mode
Dedicated Line
Through Dedicated Line, migration performance is guaranteed; Dedicated Line is connected with COS, please contact the Storage Architect.
Public network
Migration performance is limited by public network bandwidth.
Pull Mode
Dedicated Line
As above, peer providers will incur outbound external network traffic.
Public network
As above, peer providers will incur outbound external network traffic.
If the data source is Tencent Cloud COS, migrating buckets within the same region uses the private network; migrating buckets across regions uses the public network.


Migrating from AWS S3 (International)

Last updated：2024-10-22 18:09:27

Overview

This document describes how to configure an Agent semi-managed migration task to migrate data from AWS S3 (International) to Tencent Cloud COS.
Note:
You need to log in to the AWS Management Console (International) for migration from AWS S3 (International).

Preparations

AWS S3

Create an AWS IAM account and grant the relevant permissions. Below is a simple guide; for detailed instructions, please refer to AWS S3 Product Documentation.
1. Log in to the AWS Console.
2. In the navigation pane, select Users and then Add User. Type in the username for the new user.
3. Select the access permission type that this group of users will have. Choose Programmatic access and AWS Management Console access.
4. Click Next: Permissions(Next: Permissions). On the Set permissions page, specify how you want to assign permissions to the new user. Grant IAM account storage space read-write permissions.
5. Click Create user. To view the user's access keys (Access Key ID and Secret Access Key), select show next to each password and access key you wish to view. To save the access keys, select download .csv, to obtain AccessKeyID and AccessKeySecret.

Tencent Cloud COS

1. Create target bucket: Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2. Create a Sub-user for migration and grant the necessary permissions. For a brief guide, refer to Create and authorize the sub-account.
2.1 Log in to the CAM console.
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user, and check the Programming access and Tencent Cloud console access options.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete the sub-user creation and save the username, login password, SecretId, and SecretKey.
Note:
You can also log in to Cloud Migration with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.

Directions

Logging in to Cloud Migration

1. Log in to Cloud Migration console.
2. Click Object Storage Migration in the left sidebar to enter the object storage migration page.

Creating Migration Task

1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.


Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.



4. Set the file source for migration.



Service Provider: Select AWS S3 (International) for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Upon entering the key, click the Refresh button to the right of the "Bucket name" dropdown menu to retrieve a list of source object storage buckets. Alternatively, you may opt to manually input the name of the source bucket.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the target location to migrate to.



Service Provider: The default value is Tencent Cloud COS. SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths. For example: For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration. If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration pattern.



Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.



Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and start to start the migration task.




Viewing Migration Status and Progress

You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.

Estimating File Migration Time

The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.

Migrating by HTTP URL List

Last updated：2024-10-22 18:09:27

Operation scenarios

This article will detail how to implement data migration through URL list configuration and Agent in a semi-managed approach.

Preparations

Tencent Cloud COS

1. Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2. Create a Sub-user for migration and grant the necessary permissions. For a brief guide, refer to Create and authorize the sub-account.
2.1 Log in to the CAM console.
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user and select Programming access and Tencent Cloud console access.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete creating the sub-user and save the sub-username, login password, SecretId, and SecretKey.
Note:
You can also log in to Cloud Migration with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.

Directions

Logging in to Cloud Migration

1. log in to Cloud Migration console.
2. Click COS Migration in the left sidebar to enter the COS Migration page.

Creating Migration Task

1. On the COS Migration page, click Create New Task to enter the COS migration task configuration page and set the migration parameters.
2. Set migration task name.

Task Name: The character length should be between 1 and 60 characters, and allowed characters include Chinese, English, 0-9, _, and -. The name set here will be used to check migration status and progress in the task list.
3. Estimate the task size. Please accurately fill in the task size to better prepare the related resources, though this is not mandatory.


4. Set the file source for migration.



Service provider: Select URL listfor this field.
Upload URL list file: First, paste the URLs of the files to be migrated into a text file (e.g., 1.txt), then upload this text file to a location accessible via the HTTP protocol (such as a Tencent Cloud COS bucket). Next, set the text file to public read permission and obtain a public read access link. Finally, enter the link into the input box. Cloud Migration will batch upload the files listed in the text file to COS by reading the file. After the migration task is completed, delete the text file.
Note:
Each line of the text file should contain only one URL of the file to be migrated.
The storage service of both the files to be migrated and the text file should support HTTP partial downloads and public read.
If the HTTP URL of the file to be migrated contains special characters, it is recommended to perform URL encode encoding.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the destination for migration.



Service Provider: The default value is Tencent Cloud COS. SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Bucket Path Setting:
Use the path in the URL for the bucket: The relative path in the URL, excluding the domain name, is used as the COS Key. For instance, migrating http://test.com/game/1.apk to COS will result in it being stored as cos://game/1.apk.
Ignore path in the URL: All files are placed directly in the root directory of the bucket. Files with identical names will be overwritten. For example, migrating http://test.com/game/1.apk to COS will result in it being saved as cos://1.apk.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths. For example: For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration. If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration mode.



Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.



Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and start to start the migration task.



Viewing Migration Status and Progress

You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.

Estimating File Migration Time

The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.


Migrating Between Tencent Cloud COS Buckets

Last updated：2024-10-22 18:09:27

Operation scenarios

The following will detail how to configure fully managed and semi-managed migration tasks for migrating between Tencent Cloud COS buckets, facilitating data migration.

Preparations

Tencent Cloud COS

1. Create a target storage space to store the migrated data, for more details, see Creating Bucket.
2. Create a sub-user for migration and grant the required permissions:
2.1 Log in to the Tencent Cloud CAM Console.
2.2 Select User > User List in the left sidebar to enter the user list page.
2.3 Create a Sub-user, selecting Programming Access and Tencent Cloud Console Access.
2.4 Search for and select the `QcloudMSPFullAccess` and `QcloudCOSAccessForMSPRole` policies.
2.5 Complete creating the sub-user and save the sub-username, login password, `SecretId`, and `SecretKey`.
Note:
You can also log in to Cloud Migration with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.

Directions

Logging in to Cloud Migration

1. log in to Cloud Migration console.
2. Click COS Migration in the left sidebar to enter the COS Migration page.

Creating Migration Task

1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.


Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.



4. Set the file source for migration.



Service Provider: Select Tencent Cloud COS for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Upon entering the key, click the Refresh button to the right of the "Bucket name" dropdown menu to retrieve a list of source object storage buckets. Alternatively, you may opt to manually input the name of the source bucket.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the target location to migrate to.



Service Provider: The default value is Tencent Cloud COS. SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths. For example: For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration. If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration mode.



Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.



Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and start to start the migration task.




Viewing Migration Status and Progress

You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.

Estimating File Migration Time

The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.



Migrating from Alibaba Cloud OSS

Last updated：2024-10-22 18:09:27

Overview

The following describes how to configure fully managed and semi-managed migration tasks to migrate data from Alibaba Cloud OSS to Tencent Cloud COS.

Preparations

Alibaba Cloud OSS

Create a RAM sub-account and grant the required permissions:
1. Log in to the RAM console.
2. Select Personnel Management > Users > Create User.
3. Select console password login and programming access, and then enter the user account information.
4. Save the generated account, password, AccessKeyID, and AccessKeySecret.
5. Select the user log in name, click Add Permissions to grant the sub-account storage space read and write permissions (AliyunOSSFullAccess).

Tencent Cloud COS

1. Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2. Create a sub-user for migration and grant the required permissions:
2.1 Log in to the CAM console.
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user, and check the Programming access and Tencent Cloud console access options.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete the sub-user creation and save the username, login password, SecretId, and SecretKey.
Note:
You can also log in to Cloud Migration CMG with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.

Directions

Logging in to Cloud Migration

1. Log in to Cloud Migration console.
2. Click Object Storage Migration in the left sidebar to enter the object storage migration page.

Creating Migration Task

1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.


Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.



4. Set the file source for migration.




Service Provider: Select AlibabaCloud OSS for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Upon entering the key, click the Refresh button to the right of the "Bucket name" dropdown menu to retrieve a list of source object storage buckets. Alternatively, you may opt to manually input the name of the source bucket.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the destination for migration.



Service Provider: The default value is Tencent Cloud COS. SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths. For example: For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration. If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration mode.




Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.



Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and Start to start the migration task.




Viewing Migration Status and Progress

You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.

Estimating File Migration Time

The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.

S3 Protocol Migration

Last updated：2024-10-22 18:09:27

Overview

The following part describes how to configure a fully-managed public network migration task to achieve data migration when the data source is a storage system that supports the S3 protocol (such as Ceph, MinIO, and GCP).

Preparations

Origin Server

Prepare the required AccessKey and SecretKey for S3 protocol interface authentication as per the origin server instructions.

Tencent Cloud COS

1. Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2. Create a sub-user for migration and grant the required permissions:
2.1 Log in to the CAM console.
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user, and check the Programming access and Tencent Cloud console access options.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete the sub-user creation and save the username, login password, SecretId, and SecretKey.
Note:
You can also log in to Cloud Migration with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.

Directions

Logging in to Cloud Migration

1. Log in to Cloud Migration console.
2. Click Object Storage Migration in the left sidebar to enter the object storage migration page.

Creating Migration Task

1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.

Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.



4. Set the file source for migration.



Service Provider: Select Origin servers supporting S3 protocol for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Enter the name of the bucket to store the migrated data.
Endpoint: Enter the complete S3 protocol endpoint of the source bucket. (HTTP or HTTPS can be included in the domain name. If not included, HTTPS is used by default.) For S3 systems requiring region verification, enter the source region. Otherwise, leave the field blank.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the destination for migration.



Service Provider: The default value is Tencent Cloud COS. SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths. For example: For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration. If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration mode.



Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.



Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and Start to start the migration task.


Viewing Migration Status and Progress

You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.

Estimating File Migration Time

The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.