regionserver.HRegionServer: error telling master we are up

针对HBase集群中slave节点启动失败的问题,本文分析了错误日志,并提供了详细的排查步骤及解决方案,包括检查配置文件、重启服务等。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

问题:
修改hbase的配置文件hbase-env.sh,hbase-site.xml后,启动hbase集群,slave2启动异常,查看日志时报如下错误:

regionserver.HRegionServer: reportForDuty to master=master,16000,1535782912381 with port=16020, startcode=1535782914403
2018-09-01 14:27:06,028 WARN [regionserver/localhost/127.0.0.1:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.net.SocketException: Invalid argument
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:240)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2316)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:907)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)

原因:
报错提示

com.google.protobuf.ServiceException: java.net.SocketException: Invalid argument
at

Invalid argument,应该是配置文件写错了,
解决办法:
检查刚更改的配置文件,比如我的hbase-env.sh,hbase-site.xml。删除了添加的注释,重启hdfs,zookeeper,hbase。还是报错,推荐重启主机

base/MasterData/oldWALs, maxLogs=10 2025-06-24 21:56:39,697 INFO [master/hadoop102:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL hadoop102%2C16000%2C1750773392926:(num 1750773399658) 2025-06-24 21:56:39,701 ERROR [master/hadoop102:16000:becomeActiveMaster] master.HMaster: Failed to become active master java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:535) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:615) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:610) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:623) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:53) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:190) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:719) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:884) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:577) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:518) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295) at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200) at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:263) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:344) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:856) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2199) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:529) at java.lang.Thread.run(Thread.java:750) 2025-06-24 21:56:39,702 ERROR [master/hadoop102:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hadoop102,16000,1750773392926: Unhandled exception. Starting shutdown. ***** java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:535) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:615) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:610) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:623) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:53) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:190) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:719) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:884) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:577) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:518) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295) at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200) at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:263) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:344) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:856) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2199) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:529) at java.lang.Thread.run(Thread.java:750) 2025-06-24 21:56:39,703 INFO [master/hadoop102:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'hadoop102,16000,1750773392926' ***** 2025-06-24 21:56:39,703 INFO [master/hadoop102:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/hadoop102:16000:becomeActiveMaster 2025-06-24 21:56:40,607 INFO [master/hadoop102:16000] ipc.NettyRpcServer: Stopping server on /192.168.10.102:16000 2025-06-24 21:56:40,628 INFO [master/hadoop102:16000] regionserver.HRegionServer: Stopping infoServer 2025-06-24 21:56:40,652 INFO [master/hadoop102:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.w.WebAppContext@45acdd11{master,/,null,STOPPED}{file:/opt/module/hbase-2.4.18/hbase-webapps/master} 2025-06-24 21:56:40,658 INFO [master/hadoop102:16000] server.AbstractConnector: Stopped ServerConnector@7efd28bd{HTTP/1.1, (http/1.1)}{0.0.0.0:16010} 2025-06-24 21:56:40,659 INFO [master/hadoop102:16000] server.session: node0 Stopped scavenging 2025-06-24 21:56:40,659 INFO [master/hadoop102:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@5f7da3d3{static,/static,file:///opt/module/hbase-2.4.18/hbase-webapps/static/,STOPPED} 2025-06-24 21:56:40,660 INFO [master/hadoop102:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@2b10ace9{logs,/logs,file:///opt/module/hbase-2.4.18/logs/,STOPPED} 2025-06-24 21:56:40,664 INFO [master/hadoop102:16000] regionserver.HRegionServer: aborting server hadoop102,16000,1750773392926 2025-06-24 21:56:40,665 INFO [master/hadoop102:16000] regionserver.HRegionServer: stopping server hadoop102,16000,1750773392926; all regions closed. 2025-06-24 21:56:40,665 INFO [master/hadoop102:16000] hbase.ChoreService: Chore service for: master/hadoop102:16000 had [] on shutdown 2025-06-24 21:56:40,672 WARN [master/hadoop102:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2025-06-24 21:56:40,782 INFO [ReadOnlyZKClient-hadoop102:2181,hadoop103:2181,hadoop104:2181@0x1a9293ba] zookeeper.ZooKeeper: Session: 0x20000754c8c0002 closed 2025-06-24 21:56:40,782 INFO [ReadOnlyZKClient-hadoop102:2181,hadoop103:2181,hadoop104:2181@0x1a9293ba-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x20000754c8c0002 2025-06-24 21:56:40,797 INFO [master/hadoop102:16000] zookeeper.ZooKeeper: Session: 0x100007708c00000 closed 2025-06-24 21:56:40,797 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x100007708c00000 2025-06-24 21:56:40,797 INFO [master/hadoop102:16000] regionserver.HRegionServer: Exiting; stopping=hadoop102,16000,1750773392926; zookeeper connection closed. 2025-06-24 21:56:40,798 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:254) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2969) [manager1@hadoop102 logs]$
最新发布
06-25
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值