hiveserver2服务异常停止java.lang.OutOfMemoryError: Java heap space

最新推荐文章于 2025-10-08 07:39:30 发布

原创最新推荐文章于 2025-10-08 07:39:30 发布 · 8.4k 阅读

1 ·

CC 4.0 BY-SA版权

Hive 专栏收录该内容

31 篇文章

订阅专栏

本文记录了HiveServer2在使用过程中遇到的Java堆内存溢出问题及其解决方案。通过对日志的分析，发现内存溢出是由于heap size设置过小导致，并通过调整Hadoop opts参数成功解决了该问题。

ACE-Step

音乐合成

ACE-Step

ACE-Step是由中国团队阶跃星辰（StepFun）与ACE Studio联手打造的开源音乐生成模型。它拥有3.5B参数量，支持快速高质量生成、强可控性和易于拓展的特点。最厉害的是，它可以生成多种语言的歌曲，包括但不限于中文、英文、日文等19种语言

2017-01-02 15:34:04,461 WARN [IPC Client (188363230) connection to nn01/192.168.1.4:8020 from hive]: ipc.Client (Client.java:run(955)) - Unexpected error reading responses on connection Thread[IPC Client (188363230) connection to nn01.100.cloud/172.16.8.4:8020 from hive,5,main]
java.lang.OutOfMemoryError: Java heap space
   at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)
   at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
   at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)
   at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
   at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
   at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
   at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
   at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
   at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: Java heap space
   at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
   at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)
   at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)
   at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
   at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
   at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)
   at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)
   at java.lang.Thread.run(Thread.java:745)
2017-01-02 15:34:29,186 INFO [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2
2017-01-02 15:34:29,186 INFO [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2017-01-02 15:34:29,188 INFO [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5

分析及解决：

java.lang.OutOfMemoryError: Java heap space内存溢出报错

This Error happened during connect hiveserver2 via beeline, both happened from hiveserver node and remote node.

heapsize太小了，可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:

if [ "$SERVICE" = "cli" ]; then
   if [ -z "$DEBUG" ]; then
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
   else
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
   fi
fi

# The heap size of the jvm stared by hive shell script can be controlled via:
#
export HADOOP_HEAPSIZE=2048

继续分析查看下进程：

监控语句如下：

jstat -gcutil [进程号] 5000 --每5秒打印一次jvm各个内存区的状态

另外找了一个监控系统内存的语句：

vmstat | sed -n '/[0-9]/p'

观察2天后发现新生代和老年代都已经爆满，永久代几乎爆满，jvm一直在做full GC，但是没有效果，各个内存区都被占用了。
就日志来看，老年代和持久代的内存只要上去了，就再没有下降过，而当这两块区域饱和后，新生代也出现了这种现象。

是有什么资源没释放吧？一看代码，果然，有的连接没有关闭！

如果还是有问题，hive0.14版本，自身也存在着bug，可以考虑进行定期重启hiveserver2服务，别笑实际工作中很多大的公司都是这样做的。

参考：

http://blog.youkuaiyun.com/gklifg/article/details/50418109

http://blog.youkuaiyun.com/lulynn/article/details/46455097

您可能感兴趣的与本文相关的镜像