HDFS 命令_hadoop error recovery for block in pipeline-优快云博客

本文链接：https://blog.youkuaiyun.com/pys52055/article/details/112132043

本文介绍了HDFS的常用操作命令，包括查看目录信息、创建目录、文件的移动与复制、查看文件内容、修改副本数量等。同时提到了在安全模式下进行操作的注意事项以及如何合并下载多个文件。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

显示目录信息
hadoop fs -ls /
创建目录
hadoop fs -mkdir -p /pys/mkdirtest
剪切文件本地目录 HDFS目录
hadoop fs -moveFromLocal ./test.txt /pys/
拷贝文件本地目录 HDFS目录
hadoop fs -copyFromLocal ./test.txt /pys/
hadoop fs -put ./test.txt /pys/
拷贝文件到本地 HDFS目录本地目录
hadoop fs -copyToLocal /pys/test.txt ./
hadoop fs -get /pys/test.txt ./
追加文件到已存在的文件末尾当前文件目标文件
不支持修改文件
hadoop fs -appendToFile ./test01.txt /pys/test.txt
查看HDFS文件内容
hadoop fs -cat /pys/test.txt

无法使用appenToFile

hadoop fs -appendToFile 3 /2

20/07/15 09:29:18 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Got error, status message , ack with firstBadLink as 172.21.0.7:50010
	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1363)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1188)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:455)
20/07/15 09:29:18 WARN hdfs.DFSClient: Error Recovery for block BP-31621113-127.0.0.1-1594741021462:blk_1073741841_1021 in pipeline DatanodeInfoWithStorage[106.55.63.155:50010,DS-37f9b2de-2d14-4193-9767-e247b6058ae9,DISK], DatanodeInfoWithStorage[120.53.240.85:50010,DS-53d8f2de-e259-4361-b878-1a1a96225ef7,DISK], DatanodeInfoWithStorage[172.21.0.7:50010,DS-4f026c85-cdc7-446e-9664-478aac9a5d0f,DISK]: bad datanode DatanodeInfoWithStorage[172.21.0.7:50010,DS-4f026c85-cdc7-446e-9664-478aac9a5d0f,DISK]
20/07/15 09:29:19 WARN hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[106.55.63.155:50010,DS-37f9b2de-2d14-4193-9767-e247b6058ae9,DISK], DatanodeInfoWithStorage[120.53.240.85:50010,DS-53d8f2de-e259-4361-b878-1a1a96225ef7,DISK]], original=[DatanodeInfoWithStorage[106.55.63.155:50010,DS-37f9b2de-2d14-4193-9767-e247b6058ae9,DISK], DatanodeInfoWithStorage[120.53.240.85:50010,DS-53d8f2de-e259-4361-b878-1a1a96225ef7,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:918)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:992)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1160)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:455)
20/07/15 09:29:19 ERROR hdfs.DFSClient: Failed to close inode 16424
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[106.55.63.155:50010,DS-37f9b2de-2d14-4193-9767-e247b6058ae9,DISK], DatanodeInfoWithStorage[120.53.240.85:50010,DS-53d8f2de-e259-4361-b878-1a1a96225ef7,DISK]], original=[DatanodeInfoWithStorage[106.55.63.155:50010,DS-37f9b2de-2d14-4193-9767-e247b6058ae9,DISK], DatanodeInfoWithStorage[120.53.240.85:50010,DS-53d8f2de-e259-4361-b878-1a1a96225ef7,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:918)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:992)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1160)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:455)

修改hdfs-site.xml文件，添加或者修改如下两项：

<!-- appendToFile追加 -->
<property>
       <name>dfs.support.append</name>
       <value>true</value>
</property>

<property>
       <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
       <value>NEVER</value>
</property>

<property>
       <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
       <value>true</value>
</property>

xsync 分开一下修改hdfs 文件
然后重启集群
Cannot append to file/pys/test.txt. Name node is in safe mode.
hadoop dfsadmin -safemode leave命令离开安全模式

从HDFS一个路径拷贝到HDFS另一个路径
hadoop fs -cp /pys/test.txt /pys/mkdirtest/
从HDFS一个路径移动到HDFS另一个路径
hadoop fs -mv /pys/test.txt /pys/mkdirtest/
从HDFS上合并下载多个文件
hadoop fs -getmerge /pys/* ./hebing.txt
删除HDFS上的文件或者文件夹
hadoop fs -rm /pys/test.txt 删除文件
hadoop fs -rm -r /pys/mkdirtest/ 删除文件夹
删除HDFS上的空目录
hadoop fs -rmdir /test
统计文件夹大小
hadoop fs -du /pys
设置HDFS文件的副本数量
hadoop fs -setrep 5 /pys/test.txt