How-to: set yarn mapreduce memory properties

chenfangfang_2015

于 2015-08-12 14:40:44 发布

阅读量558

点赞数

CC 4.0 BY-SA版权

文章标签： hadoop memory

本文链接：https://blog.youkuaiyun.com/lulynn/article/details/47444353

本文详细介绍了YARN及MapReduce的相关配置参数，包括内存分配、CPU核心数等关键设置，并提供了基于实际系统资源的具体配置示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Property Name	Value	Description
1. yarn.nodemanager.resource.memory-mb	55296(54g)	Total: 62.8g 54g for nm 8.8g forsystem
2. yarn.scheduler.minimum-allocation-mb	2304(2.25g)	1# / 3#: 54g/24core
3. yarn.nodemanager.resource.cpu-vcores	24	1.5*16cpus
4. mapreduce.map.memory.mb	6144(6g)
5. mapreduce.reduce.memory.mb	8192(8g)
6. mapreduce.map.java.opts	-Xmx5120M	< #4
7. mapreduce.reduce.java.opts	-Xmx6144M	< #5

The memory of containers, nodemanager, map task, reduce task should based on system memory, cpu numbers.
There are 62.8g memory, 16cpus in nodemanager nodes. The memory could be like following.
Notice: Mostly, 4# = 2*2#, 5 = 2*4#. The following configurations are based on our mapreduce jobs.