hdfs datanode启动失败 org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable

Hadoop DataNode启动失败：磁盘错误与容错配置

原创已于 2022-06-24 14:51:33 修改 · 7.3w 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#hdfs

于 2022-06-24 14:51:20 首次发布

异常处理同时被 2 个专栏收录

24 篇文章

订阅专栏

hdfs

5 篇文章

订阅专栏

博客讲述了在运行Hadoop集群时遇到DataNode启动失败的问题，日志显示磁盘不可读。通过检查发现挂载的磁盘不存在，且配置的容错磁盘数量为0。解决方案包括增加容错磁盘数量或调整配置文件中不正确的磁盘路径。

问题：

datanode启动失败,查看datanode日志

org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /var/lib/docker/overlay2/2f1ec1e560a2d5d6d51fa8bb23bc3d0fe7dbcd07e6460273da8f70b7bc472af0/merged/hadoop/hdfs/data
	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:128)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:44)
	at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:127)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-06-24 14:23:47,782 WARN  checker.StorageLocationChecker (StorageLocationChecker.java:check(208)) - Exception checking StorageLocation [DISK]file:/var/lib/docker/overlay2/de2f6b02326e3ebe8f209474868b6c0d55bb0c5fe3c798f9aa5bd5048ed38dbc/merged/hadoop/hdfs/data/
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /var/lib/docker/overlay2/de2f6b02326e3ebe8f209474868b6c0d55bb0c5fe3c798f9aa5bd5048ed38dbc/merged/hadoop/hdfs/data
	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:128)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:44)
	at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:127)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-06-24 14:23:47,782 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 4, volumes configured: 6, volumes failed: 2, volume failures tolerated: 0
	at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:216)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2583)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708)

原因：
查看服务器文件系统和挂载发现挂载的磁盘不存在

df -h

查看hdfs数据目录 hdfs-site.xml的配置dfs.datanode.data.dir数据目录和dfs.datanode.failed.volumes.tolerated磁盘容错

<property>
    <name>dfs.datanode.data.dir</name>
    <value> file://目录</value>
</property>
<property>
        <name>dfs.datanode.failed.volumes.tolerated</name>
        <value>0</value>
</property>

解决：

因为没磁盘换，可将容错数量提高

<property>
        <name>dfs.datanode.failed.volumes.tolerated</name>
        <value>2</value>
</property>

或者将磁盘目录从配置项中删除