Hadoop:HA 踩坑 - 所有 namenode 都是standby

本文记录了一次Hadoop高可用(HA)环境中所有namenode处于standby状态的问题。经过检查,发现Zookeeper服务虽然运行正常,但未能实现HA切换。尝试一通过手动将namenode转换为active状态,但重启后问题依旧。接着,通过初始化Zookeeper服务(hdfszkfc-formatZK)解决了问题,使得在启动后HA功能恢复正常。这表明问题在于Zookeeper配置或状态。

状况:

所有namenode都是standby,但是zookeeper启动正常,即ZK服务未生效,不能高可用切换HA

尝试一:手动强制转化某个namenode为active

操作:在某台namenode上,执行 hdfs haadmin -transitionToActive --forcemanual nn1 (nn1是你的某台nameservice-id)

[root@node01 ~]# hdfs haadmin -getServiceState nn1
active
[root@node02 ~]# hdfs haadmin -getServiceState nn2
standby

结果:nn1被成功转为active。但是在stop-dfs.sh后再一次start-dfs.sh后,所有namenode仍然都是standby

结论:果然因该是ZK的问题

尝试二:初始化ZK

操作:在某台namenode上,执行 hdfs zkfc -formatZK

结果:重新 start-dfs.sh后,一切正常

我的自建hadoop集群,3台master和4台core,我在部署完后发现master1的hadoop/logs下的log文件不知道什么原因不写入了,只有out文件在写。正常是out文件很小log文件比较大对吧[ec2-user@hadoop-master01 logs]$ ll total 51876 -rw-rw-r-- 1 ec2-user ec2-user 1019984 Sep 28 18:27 hadoop-ec2-user-historyserver-hadoop-master01.log -rw-rw-r-- 1 ec2-user ec2-user 130399 Oct 16 09:56 hadoop-ec2-user-historyserver-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 45247 Oct 15 16:22 hadoop-ec2-user-historyserver-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 51151 Oct 15 16:17 hadoop-ec2-user-historyserver-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 116382 Oct 15 14:05 hadoop-ec2-user-historyserver-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 46976 Oct 14 14:33 hadoop-ec2-user-historyserver-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 46688 Oct 14 13:53 hadoop-ec2-user-historyserver-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 12621739 Sep 28 18:28 hadoop-ec2-user-journalnode-hadoop-master01.log -rw-rw-r-- 1 ec2-user ec2-user 1339323 Oct 16 09:58 hadoop-ec2-user-journalnode-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 257238 Oct 15 18:11 hadoop-ec2-user-journalnode-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 65845 Oct 15 16:22 hadoop-ec2-user-journalnode-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 66349 Oct 15 16:17 hadoop-ec2-user-journalnode-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 55910 Oct 15 15:57 hadoop-ec2-user-journalnode-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 3240752 Oct 15 15:49 hadoop-ec2-user-journalnode-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 1900111 Oct 16 09:57 hadoop-ec2-user-namenode-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 18:09 hadoop-ec2-user-namenode-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 240450 Oct 15 18:09 hadoop-ec2-user-namenode-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 241594 Oct 15 18:05 hadoop-ec2-user-namenode-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 236717 Oct 15 17:57 hadoop-ec2-user-namenode-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 198509 Oct 15 17:40 hadoop-ec2-user-namenode-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 10266208 Sep 28 18:27 hadoop-ec2-user-resourcemanager-hadoop-master01.log -rw-rw-r-- 1 ec2-user ec2-user 765717 Oct 16 09:23 hadoop-ec2-user-resourcemanager-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 577288 Oct 15 16:22 hadoop-ec2-user-resourcemanager-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 572997 Oct 15 16:17 hadoop-ec2-user-resourcemanager-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 573291 Oct 15 15:33 hadoop-ec2-user-resourcemanager-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 727212 Oct 15 15:20 hadoop-ec2-user-resourcemanager-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 848229 Oct 15 15:05 hadoop-ec2-user-resourcemanager-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 482956 Sep 28 18:27 hadoop-ec2-user-timelineserver-hadoop-master01.log -rw-rw-r-- 1 ec2-user ec2-user 2669030 Oct 16 09:26 hadoop-ec2-user-timelineserver-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 1069795 Oct 15 16:22 hadoop-ec2-user-timelineserver-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 1076432 Oct 15 16:17 hadoop-ec2-user-timelineserver-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 1070341 Oct 15 15:33 hadoop-ec2-user-timelineserver-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 1064610 Oct 15 15:20 hadoop-ec2-user-timelineserver-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 3670854 Oct 15 15:06 hadoop-ec2-user-timelineserver-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 5036578 Sep 28 18:28 hadoop-ec2-user-zkfc-hadoop-master01.log -rw-rw-r-- 1 ec2-user ec2-user 99301 Oct 15 18:12 hadoop-ec2-user-zkfc-hadoop-master01.out -rw-rw-r-- 1 ec2-user ec2-user 195445 Oct 15 18:11 hadoop-ec2-user-zkfc-hadoop-master01.out.1 -rw-rw-r-- 1 ec2-user ec2-user 80419 Oct 15 16:22 hadoop-ec2-user-zkfc-hadoop-master01.out.2 -rw-rw-r-- 1 ec2-user ec2-user 79427 Oct 15 16:17 hadoop-ec2-user-zkfc-hadoop-master01.out.3 -rw-rw-r-- 1 ec2-user ec2-user 80419 Oct 15 15:57 hadoop-ec2-user-zkfc-hadoop-master01.out.4 -rw-rw-r-- 1 ec2-user ec2-user 79427 Oct 15 15:49 hadoop-ec2-user-zkfc-hadoop-master01.out.5 -rw-rw-r-- 1 ec2-user ec2-user 0 Sep 17 11:54 SecurityAuth-ec2-user.audit [ec2-user@hadoop-master01 logs]$ pwd /data/module/hadoop-3.3.4/logs [ec2-user@hadoop-master02 hadoop-3.3.4]$ cd logs/ [ec2-user@hadoop-master02 logs]$ ll total 142064 -rw-rw-r-- 1 ec2-user ec2-user 41175348 Oct 16 09:58 hadoop-ec2-user-journalnode-hadoop-master02.log -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 18:12 hadoop-ec2-user-journalnode-hadoop-master02.out -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 16:23 hadoop-ec2-user-journalnode-hadoop-master02.out.1 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 16:18 hadoop-ec2-user-journalnode-hadoop-master02.out.2 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:57 hadoop-ec2-user-journalnode-hadoop-master02.out.3 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:49 hadoop-ec2-user-journalnode-hadoop-master02.out.4 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 13 17:36 hadoop-ec2-user-journalnode-hadoop-master02.out.5 -rw-rw-r-- 1 ec2-user ec2-user 87261368 Oct 16 09:57 hadoop-ec2-user-namenode-hadoop-master02.log -rw-rw-r-- 1 ec2-user ec2-user 6371 Oct 15 18:13 hadoop-ec2-user-namenode-hadoop-master02.out -rw-rw-r-- 1 ec2-user ec2-user 6371 Oct 15 17:46 hadoop-ec2-user-namenode-hadoop-master02.out.1 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 16:18 hadoop-ec2-user-namenode-hadoop-master02.out.2 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:57 hadoop-ec2-user-namenode-hadoop-master02.out.3 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:49 hadoop-ec2-user-namenode-hadoop-master02.out.4 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 13 17:36 hadoop-ec2-user-namenode-hadoop-master02.out.5 -rw-rw-r-- 1 ec2-user ec2-user 11161047 Oct 16 05:24 hadoop-ec2-user-resourcemanager-hadoop-master02.log -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 16:23 hadoop-ec2-user-resourcemanager-hadoop-master02.out -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 16:18 hadoop-ec2-user-resourcemanager-hadoop-master02.out.1 -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 15:34 hadoop-ec2-user-resourcemanager-hadoop-master02.out.2 -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 15:21 hadoop-ec2-user-resourcemanager-hadoop-master02.out.3 -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 15:06 hadoop-ec2-user-resourcemanager-hadoop-master02.out.4 -rw-rw-r-- 1 ec2-user ec2-user 2218 Oct 15 14:23 hadoop-ec2-user-resourcemanager-hadoop-master02.out.5 -rw-rw-r-- 1 ec2-user ec2-user 5751751 Oct 15 18:12 hadoop-ec2-user-zkfc-hadoop-master02.log -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 18:12 hadoop-ec2-user-zkfc-hadoop-master02.out -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 16:23 hadoop-ec2-user-zkfc-hadoop-master02.out.1 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 16:18 hadoop-ec2-user-zkfc-hadoop-master02.out.2 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:58 hadoop-ec2-user-zkfc-hadoop-master02.out.3 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 15 15:49 hadoop-ec2-user-zkfc-hadoop-master02.out.4 -rw-rw-r-- 1 ec2-user ec2-user 693 Oct 13 17:36 hadoop-ec2-user-zkfc-hadoop-master02.out.5 -rw-rw-r-- 1 ec2-user ec2-user 0 Sep 17 11:55 SecurityAuth-ec2-user.audit [ec2-user@hadoop-master02 logs]$ pwd /data/module/hadoop-3.3.4/logs 我不记得对master01做过什么了 但他的log就是不写日志了
最新发布
10-17
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值