1.前言
用户使用Beeline访问HiveServer2 (3.1.2版本) 执行离线SQL任务,持续运行一周后HiveServer2就出现OOM现象,严重影响数据查询与报表产出,经过几轮修复问题终于解决。作者把修复过的问题进行了汇总,避免其他小伙伴再遇到此问题时束手无策。
2.案例
2.1 HIVE-16455
HiveServer2 在使用ADD JAR语句时导致文件句柄泄漏
[root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
java 29588 hive 391u REG 252,3 125987 2099944 /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
java 29588 hive 392u REG 252,3 125987 2099946 /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
java 29588 hive 393r REG 252,3 125987 2099825 /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
java 29588 hive 394r REG 252,3 125987 2099833 /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
java 29588 hive 395r REG 252,3 125987 2099827 /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
java 29588 hive 396r REG 252,3 125987 2099822 /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
java 29588 hive 397r REG 252,3 125987 2099831 /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
java 29588 hive 399r REG 252,3 125987 2099835 /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
2.2 HIVE-24236
不容易复现,只能某些特定条件下可能存在连接泄漏风险
2020-09-29T18:44:26,563 INFO [Heartbeater-0]: txn.TxnHandler (TxnHandler.java:checkRetryable(3733)) - Non-retryable error in heartbeat(HeartbeatRequest(lockid:0, txnid:11908)) : Cannot get a connection, general error (SQLState=null, ErrorCode=0)
2020-09-29T18:44:26,564 ERROR [Heartbeater-0]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Unable to select from transaction database org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3605)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3598)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeat(TxnHandler.java:2739)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.heartbeat(HiveMetaStore.java:8452)
at sun.reflect.GeneratedMethodAccessor415
HiveServer2 内存溢出问题

最低0.47元/天 解锁文章
4352





