Hadoop cannot find namenode pid file when shutdown

最新推荐文章于 2023-04-24 23:33:27 发布

原创最新推荐文章于 2023-04-24 23:33:27 发布 · 1.5k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #namenode #kill #关闭

Java 同时被 3 个专栏收录

4 篇文章

订阅专栏

Hadoop

3 篇文章

订阅专栏

云计算

2 篇文章

订阅专栏

本文详细介绍了如何手动关闭Hadoop集群并更改默认设置以避免pid文件清理导致的集群自动关闭问题。包括手动关闭集群的步骤、修改配置文件以指定pid文件目录、以及重启集群后检查pid文件的正确存放位置。

By default settings Hadoop use /tmp/ folder to track NameNode and JobTracker process id alive. But /tmp folder is cleanup once a week. So after some time, you will get error when you try to shutdown Hadoop cluster. You have to shutdown hadoop cluster manually. And don't use default setting to track pid files.

1. Shutdown hadoop cluster manually.

jps | grep 'TaskTracker\|DataNode' | sed 's/ .*//g' | sed 's/^/kill -9 /g' | sh

This code is ugly but works. You can try some graceful way to do this, like :

jps | grep 'TaskTracker\|DataNode' | awk 'echo $1' | xargs kill -9 # I've not try this.

2. Change Hadoop default setting.

Revise hadoop-env.sh file in ${HADOOP_HOME}/conf, add somethings like below.

export HADOOP_PID_DIR=${HADOOP_HOME}/pid

3. Restart your cluster, you should find some pid files in that folder.

ls -l
-rw-r--r-- 1 ana grid 6 Nov 13 01:54 hadoop-ana-jobtracker.pid
-rw-r--r-- 1 ana grid 6 Nov 13 01:54 hadoop-ana-namenode.pid
-rw-r--r-- 1 ana grid 6 Nov 13 01:54 hadoop-ana-secondarynamenode.pid