产品动态
产品公告
hadoop-cos-2.x.x-${version}.jar 、 cos_api-bundle-${version}.jar 和 chdfs_hadoop_plugin_network-${version}.jar 这三个 jar 包分别拷贝到 Datax 解压路径 plugin/reader/hdfsreader/libs/ 和 plugin/writer/hdfswriter/libs/ 下。CLASS_PATH = ("%s/lib/*:%s/plugin/reader/hdfsreader/libs/*:%s/plugin/writer/hdfswriter/libs/*:.") % (DATAX_HOME, DATAX_HOME, DATAX_HOME)
{"job": {"setting": {"speed": {"byte": 10485760},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "hdfsreader","parameter": {"path": "/test/","defaultFS": "cosn://examplebucket1-1250000000/","column": ["*"],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.trsf.fs.ofs.bucket.region": "ap-guangzhou","fs.cosn.bucket.region": "ap-guangzhou","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.trsf.fs.ofs.tmp.cache.dir": "/tmp/","fs.cosn.userinfo.secretId": "COS_SECRETID","fs.cosn.userinfo.secretKey": "COS_SECRETKEY","fs.cosn.trsf.fs.ofs.user.appid": "1250000000"},"fieldDelimiter": ","}},"writer": {"name": "hdfswriter","parameter": {"path": "/","fileName": "hive.test","defaultFS": "cosn://examplebucket2-1250000000/","column": [{"name":"col1","type":"int"},{"name":"col2","type":"string"}],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.trsf.fs.ofs.bucket.region": "ap-guangzhou","fs.cosn.bucket.region": "ap-guangzhou","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.trsf.fs.ofs.tmp.cache.dir": "/tmp/","fs.cosn.userinfo.secretId": "COS_SECRETID","fs.cosn.userinfo.secretKey": "COS_SECRETKEY","fs.cosn.trsf.fs.ofs.user.appid": "1250000000"},"fieldDelimiter": ",","writeMode": "append"}}}]}}
cosn://examplebucket-1250000000/。ap-guangzhou，详情请参见 地域和访问域名。[root@172 /usr/local/service/datax]# python bin/datax.py job/hdfs_job.json
2022-10-23 00:25:24.954 [job-0] INFO JobContainer -[total cpu info] =>averageCpu | maxDeltaCpu | minDeltaCpu-1.00% | -1.00% | -1.00%[total gc info] =>NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTimePS MarkSweep | 1 | 1 | 1 | 0.034s | 0.034s | 0.034sPS Scavenge | 14 | 14 | 14 | 0.059s | 0.059s | 0.059s2022-10-23 00:25:24.954 [job-0] INFO JobContainer - PerfTrace not enable!2022-10-23 00:25:24.954 [job-0] INFO StandAloneJobContainerCommunicator - Total 1000003 records, 9322478 bytes | Speed 910.40KB/s, 100000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1.000s | All Task WaitReaderTime 6.259s | Percentage 100.00%2022-10-23 00:25:24.955 [job-0] INFO JobContainer -任务启动时刻 : 2022-10-23 00:25:12任务结束时刻 : 2022-10-23 00:25:24任务总计耗时 : 12s任务平均流量 : 910.40KB/s记录写入速度 : 100000rec/s读出记录总数 : 1000003读写失败总数 : 0
cosn-ranger-interface-1.x.x-${version}.jar 和 hadoop-ranger-client-for-hadoop-${version}.jar 这两个 jar 包拷贝到 Datax 解压路径 plugin/reader/hdfsreader/libs/ 和 plugin/writer/hdfswriter/libs/ 下。单击前往 Github 下载{"job": {"setting": {"speed": {"byte": 10485760},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "hdfsreader","parameter": {"path": "/test/","defaultFS": "cosn://examplebucket1-1250000000/","column": ["*"],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.trsf.fs.ofs.bucket.region": "ap-guangzhou","fs.cosn.bucket.region": "ap-guangzhou","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.trsf.fs.ofs.tmp.cache.dir": "/tmp/","fs.cosn.trsf.fs.ofs.user.appid": "1250000000","fs.cosn.credentials.provider": "org.apache.hadoop.fs.auth.RangerCredentialsProvider","qcloud.object.storage.zk.address": "172.16.0.30:2181","qcloud.object.storage.ranger.service.address": "172.16.0.30:9999","qcloud.object.storage.kerberos.principal": "hadoop/172.16.0.30@EMR-5IUR9VWW"},"haveKerberos": "true","kerberosKeytabFilePath": "/var/krb5kdc/emr.keytab","kerberosPrincipal": "hadoop/172.16.0.30@EMR-5IUR9VWW","fieldDelimiter": ","}},"writer": {"name": "hdfswriter","parameter": {"path": "/","fileName": "hive.test","defaultFS": "cosn://examplebucket2-1250000000/","column": [{"name":"col1","type":"int"},{"name":"col2","type":"string"}],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.trsf.fs.ofs.bucket.region": "ap-guangzhou","fs.cosn.bucket.region": "ap-guangzhou","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.trsf.fs.ofs.tmp.cache.dir": "/tmp/","fs.cosn.trsf.fs.ofs.user.appid": "1250000000","fs.cosn.credentials.provider": "org.apache.hadoop.fs.auth.RangerCredentialsProvider","qcloud.object.storage.zk.address": "172.16.0.30:2181","qcloud.object.storage.ranger.service.address": "172.16.0.30:9999","qcloud.object.storage.kerberos.principal": "hadoop/172.16.0.30@EMR-5IUR9VWW"},"haveKerberos": "true","kerberosKeytabFilePath": "/var/krb5kdc/emr.keytab","kerberosPrincipal": "hadoop/172.16.0.30@EMR-5IUR9VWW","fieldDelimiter": ",","writeMode": "append"}}}]}}
cosn-ranger-interface-1.x.x-${version}.jar 和 hadoop-ranger-client-for-hadoop-${version}.jar 拷贝到 Datax 解压路径 plugin/reader/hdfsreader/libs/ 以及 plugin/writer/hdfswriter/libs/ 下。单击前往 Github 下载hadoop/_HOST@EMR-5IUR9VWW。因为 datax 并不能解析 _HOST 域名，所以需要将 _HOST 换成 ip。可以使用 klist -ket /var/krb5kdc/emr.keytab 命令来查找合适的 Principal。文档反馈