hadoop集群运维(updating)

最新推荐文章于 2023-11-27 18:31:15 发布

原创

最新推荐文章于 2023-11-27 18:31:15 发布 · 827 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #big data #spark

本文讨论了Hadoop集群在运行过程中遇到的问题，包括文件关闭异常和hostname未知错误。异常可能由高负载导致，建议降低任务并发量和控制CPU使用率以减轻网络传输压力。同时，出现UnknownHostException可能是节点配置问题，解决方法是检查并修复节点hostname配置。此外，还提到了NTP服务对于集群同步的重要性，当NTP服务异常时，需要重启相关服务以恢复正常。

Job运行中出错

Unable to close file because the last block BP-1820686335-10.201.48.27-144816918

ava.io.IOException: Unable to close file because the last block BP-1820686335-10.201.48.27-1448169181587:blk_1850383542_781036567 does not have enough number of replicas.
        at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2705)
        at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2667)
        at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2621)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
        at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:248)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:380)
        at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:1060)
        at org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:67)
        at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:83)
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStor