hive-client heap内存大小的配置优先级

hive-client heap内存大小的配置优先级

hive-client Heap大小的配置优先级

其实主要解决，hive作为数据仓库（hive -e “select ····”） 如果是分区表且分区较多可能导致hive 堆内存溢出情况
==注意：HEAP大小配置优先级有配置语句顺序无关==
以下为hive 堆配置的优先级 由大到小

vi conf/hive-env.sh

export  HADOOP_CLIENT_OPTS=" -Xmx3000m" > if···Xms512m -Xmx2048m··fi > export HADOOP_HEAPSIZE=1024

如图为hive-env.sh 配置（证明heap配置优先级有配置语句顺序无）

# Hive Client memory usage can be an issue if a large number of clients
# are running at the same time. The flags below have been useful in 
# reducing memory usage:
#
if [ "$SERVICE" = "cli" ]; then
   if [ -z "$DEBUG" ]; then
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms500m -Xmx2048m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
   else
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms500m -Xmx2048m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
   fi
fi


# The heap size of the jvm stared by hive shell script can be controlled via:
#
#export HADOOP_HEAPSIZE=1024
export  HADOOP_CLIENT_OPTS=" -Xmx3000m"
export HADOOP_HEAPSIZE=1024
#
# Larger heap size may be required when running queries over large number of files or partitions. 
# By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be 
# appropriate for hive server (hwi etc).


# Set HADOOP_HOME to point to a specific hadoop install directory
# HADOOP_HOME=${bin}/../../hadoop

---

jps -lvm

meicai@dev04:~$ sudo su - hive

hive@dev04:~$ jps -m
30689 RunJar /opt/hive/hive-1.1.0/lib/hive-cli-1.1.0.jar org.apache.hadoop.hive.cli.CliDriver
27382 RunJar /opt/hive/hive-1.1.0/lib/hive-service-1.1.0.jar org.apache.hadoop.hive.metastore.HiveMetaStore
30904 Jps -m
27295 RunJar /opt/hive/hive-1.1.0/lib/hive-service-1.1.0.jar org.apache.hive.service.server.HiveServer2

hive@dev04:~$ jps -lvm

30689 org.apache.hadoop.util.RunJar /opt/hive/hive-1.1.0/lib/hive-cli-1.1.0.jar org.apache.hadoop.hive.cli.CliDriver -Xmx1024m -XX:NewRatio=12 -Xms500m -Xmx2048m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/hadoop/hadoop-2.6.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/hadoop-2.6.0 -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/hadoop/hadoop-2.6.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Xmx3000m -Dhadoop.security.logger=INFO,NullAppender

27382 org.apache.hadoop.util.RunJar /opt/hive/hive-1.1.0/lib/hive-service-1.1.0.jar org.apache.hadoop.hive.metastore.HiveMetaStore -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/hadoop/hadoop-2.6.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/hadoop-2.6.0 -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/hadoop/hadoop-2.6.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Xmx3000m -Dhadoop.security.logger=INFO,NullAppender

30925 sun.tools.jps.Jps -lvm -Dapplication.home=/home/jdk1.8.0_20 -Xms8m

27295 org.apache.hadoop.util.RunJar /opt/hive/hive-1.1.0/lib/hive-service-1.1.0.jar org.apache.hive.service.server.HiveServer2 -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/hadoop/hadoop-2.6.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/hadoop-2.6.0 -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/hadoop/hadoop-2.6.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Xmx3000m -Dhadoop.security.logger=INFO,NullAppender

hive@dev04:~$

posted on 2017-02-08 11:40 姜小嫌 阅读( ...) 评论( ...) 编辑 收藏

转载于:https://www.cnblogs.com/jiangxiaoxian/p/6377471.html