spark 中写hdfs的异常解决

最新推荐文章于 2024-04-10 22:30:00 发布

原创

最新推荐文章于 2024-04-10 22:30:00 发布 · 3.6k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#spark #hdfs #carbondata

在尝试关闭HDFS的inode 16604时遇到错误，问题源于无法在现有的数据节点管道中替换故障节点。通过修改`dfs.client.block.write.replace-datanode-on-failure.policy`为`NEVER`并启用`dfs.client.block.write.replace-datanode-on-failure.enable`，最终在Spark Shell中使用Scala环境设置这两个属性解决了问题。

报错信息：

ERROR DFSClient: Failed to close inode 16604
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]], original=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setup