tencent cloud


Migrating from AWS S3 (International)

Last updated: 2022-12-23 14:59:01


    This document describes how to configure an Agent semi-managed migration task to migrate data from AWS S3 (International) to Tencent Cloud COS.


    You need to log in to the AWS Management Console (International) for migration from AWS S3 (International).

    Only semi-managed migration is supported for AWS S3 (International).


    AWS S3

    Create an AWS IAM account and grant required permissions:

    1. Log in to the AWS Management Console.
    2. In the navigation pane, select Users > Add users. Then, enter a username.
    3. Select the type of access this user will have. You can select Programmatic access and AWS Management Console access.
    4. Click Next: Permissions. On the Set permissions page, specify how you want to assign permissions to this new user. Grant the user bucket read and write permissions under the AWS Identity and Access Management (IAM) account.
    5. Click Create user.
    6. To view the user's access key (access key ID and secret access key), select Show next to the password and access key that you want to see. To save the access key, select Download .csv, which contains the AccessKeyID and AccessKeySecret.

    Tencent Cloud COS

    1. Create a bucket:
      Create a bucket to store the migrated data as instructed in Creating Bucket.
    2. Create a sub-user for migration and grant the required permissions:
      1. Log in to the CAM console.
      2. Click User > User List on the left sidebar to enter the user list page.
      3. Create a sub-user and select Programming access and Tencent Cloud console access.
      4. Search for and select the QcloudCOSAccessForMSPRole and QcloudCOSFullAccess policies.
      5. Complete creating the sub-user and save the sub-username, login password, SecretId, and SecretKey.
      6. Click here to download the Agent.

    You can use Migration Service Platform (MSP) with your root account. However, for security reasons, we recommend you create a sub-account, use the sub-account's API key for migration, and delete the sub-user after migration.


    Logging in to MSP

    1. Log in to the MSP console.
    2. Click Object Storage Migration on the left sidebar.

    Creating a migration task

    1. On the Object Storage Migration page, click Create a Task to enter the object storage migration task configuration page and set migration parameters.

    2. Set the migration task name.

      Task Name: It can contain 1–60 letters, digits, underscores, and hyphens. You will need to use this name when viewing the migration status and progress in the task list.

    3. Estimate the task size.

      We recommend you enter an optional accurate task size for us to better prepare resources.

    4. Set the source of files to be migrated.
      Here, select AWS S3 (International) for Service Provider and enter the AccessKeyID and AccessKeySecret of the AWS sub-account created above in AccessKey and SecretKey boxes respectively. Then, click the Refresh icon on the right of the Migration bucket name drop-down list to get the list of source buckets.

      You can also manually enter a source bucket name.

    5. Select the header migration mode.
      If the file in the source bucket has headers and tags set and they need to be retained after migration, select to keep them or set the replacement rule.

    6. Set the migration rule.
      Choose to migrate all the files of the specified bucket, or only migrate files with a specified prefix.

    7. Set a time range.
      Enable Time Range to migrate only files added or changed within the specified time range.

    8. Select a file storage type.
      Set the storage type of the migrated files based on your migration needs. You can select Standard Storage, Infrequent Access Storage, Keep the original storage properties, or Archive Storage.

    9. Set the execution speed.
      All public cloud object storage services have speed limits. To ensure business stability, check with the source vendor and set the maximum available migration bandwidth in Mbps before migrating.

    10. Select the migration target.
      In Migration Target Information, enter the SecretId and SecretKey of the Tencent Cloud sub-user for migration. Then, click the Refresh icon on the right of the Migration bucket name drop-down list to get the list of target buckets.

    11. Specify the directory in the target bucket for migration and the method to process files with the same name.

      • Save to the root directory: Directly save the files in the source bucket to the target bucket's root directory while maintaining the original relative path.
      • Save to the specified directory: Save the files in the source bucket to the specified directory while maintaining the original relative path.

        For example:
        Two source bucket files /a.txt and /dir/b.txt are to be migrated, and "dest" is entered in the text box. In this case, after migration, the paths of these two files in the target bucket will be /dest/a.txt and /dest/dir/b.txt.
        If dest/20180901 is entered in the text box, then after migration, the paths of these two files in the target bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt.

      • If you select Overwrite for files with the same name, the source files will directly overwrite the files with the same name in the target bucket.
      • If you select Skip for files with the same name, the system will process them based on the last modification time of the files (LastModified):
      • If the LastModified of the file in the source address is at or after that of the file in the target address, overwriting will be performed.
      • If the LastModified of the file in the source address is before that of the file in the target address, no overwriting will be performed.
      • Perform secondary migration if the object (file) content is changed during migration.
    12. Select a migration mode.

      • Create a migration task and download the Agent manually to start migration: Select the Agent migration mode. After you click Create and Start, only the task configuration will be created. You must manually download the Agent and deploy it on the migration source server as instructed in Using Semi-Managed Migration Agent to start the migration.
    13. Click Create and Start to start the migration task.

    Viewing the Migration Status and Progress

    You can view the status and progress of all file migration tasks on the file migration tool homepage:
    • For the Task completed status, green indicates that the migration task was completed and all files were successfully migrated, while yellow indicates that the migration task was completed but some files failed to be migrated.
    • After you click Retry failed task, files that failed in the task will be retried for migration, while files that have been successfully migrated will not be transferred again.
    • You can click Export to export the list of files that failed to be migrated.

    Estimating the File Migration Time

    The migration speed is subject to the lowest speed during each stage of the migration process and is affected by factors such as the network transfer speed and maximum concurrency as detailed below:

    Affecting Factor Description
    Read speed of the migration source The read speed of the data source varies by vendor.
  • The transfer speed is generally 50–20,000 Mbps.
  • The file read QPS is generally 500–5,000 (the transfer of a large number of small files is subject to this concurrency).
  • Transfer speed of MSP MSP offers up to 2,000 Mbps migration bandwidth in fully managed mode. In semi-managed mode, the bandwidth is subject to your migration resource size.
    Write speed of the migration target COS: The default write speed is 15,000 Mbps, and the write QPS is 3,000–30,000.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support