spark 中写hdfs的异常解决

最新推荐文章于 2024-04-10 22:30:00 发布

leeajax

最新推荐文章于 2024-04-10 22:30:00 发布

阅读量3.5k

点赞数

CC 4.0 BY-SA版权

分类专栏： carbondata 文章标签： spark hdfs carbondata

本文链接：https://blog.youkuaiyun.com/leeajax/article/details/75309346

在尝试关闭HDFS的inode 16604时遇到错误，问题源于无法在现有的数据节点管道中替换故障节点。通过修改`dfs.client.block.write.replace-datanode-on-failure.policy`为`NEVER`并启用`dfs.client.block.write.replace-datanode-on-failure.enable`，最终在Spark Shell中使用Scala环境设置这两个属性解决了问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

报错信息：

ERROR DFSClient: Failed to close inode 16604
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]], original=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988)
at org.apache.hadoop.hdfs.DFSOut