Hadoop/Yarn/MapReduce内存分配（配置）方案

分享一下我老师大神的人工智能教程！零基础，通俗易懂！http://blog.youkuaiyun.com/jiangjunshow

也欢迎大家转载本篇文章。分享知识，造福人民，实现我们中华民族伟大复兴！

                以horntonworks给出推荐配置为蓝本，给出一种常见的Hadoop集群上各组件的内存分配方案。方案最右侧一栏是一个8G VM的分配方案，方案预留1-2G的内存给操作系统，分配4G给Yarn/MapReduce，当然也包括了HIVE，剩余的2-3G是在需要使用HBase时预留给HBase的。

Configuration File	Configuration Setting	Value Calculation	8G VM (4G For MR)
yarn-site.xml	yarn.nodemanager.resource.memory-mb	= containers * RAM-per-container	4096
yarn-site.xml	yarn.scheduler.minimum-allocation-mb	= RAM-per-container	1024
yarn-site.xml	yarn.scheduler.maximum-allocation-mb	= containers * RAM-per-container	4096
mapred-site.xml	mapreduce.map.memory.mb	= RAM-per-container	1024
mapred-site.xml	mapreduce.reduce.memory.mb	= 2 * RAM-per-container	2048
mapred-site.xml	mapreduce.map.java.opts	= 0.8 * RAM-per-container	819
mapred-site.xml	mapreduce.reduce.java.opts	= 0.8 * 2 * RAM-per-container	1638
yarn-site.xml (check)	yarn.app.mapreduce.am.resource.mb	= 2 * RAM-per-container	2048
yarn-site.xml (check)	yarn.app.mapreduce.am.command-opts	= 0.8 * 2 * RAM-per-container	1638
tez-site.xml	tez.am.resource.memory.mb	= RAM-per-container	1024
tez-site.xml	tez.am.java.opts	= 0.8 * RAM-per-container	819
tez-site.xml	hive.tez.container.size	= RAM-per-container	1024
tez-site.xml	hive.tez.java.opts	= 0.8 * RAM-per-container	819

            

给我老师的人工智能教程打call！http://blog.youkuaiyun.com/jiangjunshow