Hadoop DistCp (Distributed copy) is a tool used for large inter- and intra-cluster copying. It uses MapReduce to perform distribution, error handling, recovery, and reporting. With the parallel processing capabilities of MapReduce, Hadoop DistCp can quickly perform large-scale data migration through map tasks, each of which copies a portion of the files specified under the source path.
Since Hadoop-COS implements the semantics of the Hadoop Distributed File System (HDFS), Hadoop DistCp can help you easily migrate data between COS and HDFS. This document outlines this process below.
Prerequisites
1. The Hadoop-COS plugin has been installed on the Hadoop cluster, and the access key for COS has been configured correctly. You can use the following Hadoop commands to check if COS access is normal:
hadoop fs -ls cosn://examplebucket-1250000000/
If the files in the COS bucket can be listed correctly, it means that Hadoop-COS has been installed and configured correctly, and the following steps can be performed.
2. The COS access account must have read and write permission for the destination path of the COS bucket.
The custom monitoring feature requires Cloud Monitoring to have permission to report metrics and read API operations. Please grant the QcloudMonitorFullAccess permission with caution.
Directions
Copying files from HDFS to a COS bucket
Use Hadoop DistCp to migrate files in the / test directory of the local HDFS cluster to the COS bucket hdfs-test-1250000000.
1. Run the following command to start the migration:
Hadoop DistCp starts MapReduce tasks to copy the files, and outputs a brief report like so:
2. Run the hadoop fs -ls -R cosn://hdfs-test-1250000000/ command to list the directories and files that have just been migrated to the bucket hdfs-test-1250000000.
Copying files from a COS bucket to a local HDFS cluster
Hadoop DistCp is a tool that supports copying data between different clusters and file systems. To copy COS files to a local HDFS cluster, simply use the object path in the COS bucket as the source path and the HDFS file path as the destination path.
Hadoop DistCp supports a variety of parameters. For example, you can use -m to specify the maximum number of concurrent Map tasks, and -bandwidth to limit the maximum bandwidth used by each map. For more information, please see Apache Hadoop DistCp Guide.
