Error log:
2015-06-23 17:35:05,995 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave.chff.dc:2183,master.chff.dc:2183,slave01.chff.dc:2183, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase2015-06-23 17:35:05,995 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2015-06-23 17:35:05,997 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2044)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2058)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:541)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:347)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2039)
... 5 more
Resolve:
This kind of error is alwayse caused by zookeeper cluster issue. Need to check zookeeper log. There are 3 zookeeper nodes in cluster, but all their myid is 1. Found following similiar log in all nodes' zookeeper.out:
2015-06-23 17:24:28,941 [myid:1]
This is wrong, as zookeeper server should use different id. Based on zoo.cfg:
server.1=host1:2888:3888
server.2=host2:2888:3888
server.3=host3:2888:3888
Update content in myid file to different one: 1,2,3. Restart zookeeper, then HBase worked.
server.1=host1:2888:3888
server.2=host2:2888:3888
server.3=host3:2888:3888
Update content in myid file to different one: 1,2,3. Restart zookeeper, then HBase worked.