How-to: set yarn mapreduce memory properties

原创于 2015-08-12 14:40:44 发布 · 558 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #memory

本文详细介绍了YARN及MapReduce的相关配置参数，包括内存分配、CPU核心数等关键设置，并提供了基于实际系统资源的具体配置示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Property Name	Value	Description
1. yarn.nodemanager.resource.memory-mb	55296(54g)	Total: 62.8g 54g for nm 8.8g forsystem
2. yarn.scheduler.minimum-allocation-mb	2304(2.25g)	1# / 3#: 54g/24core
3. yarn.nodemanager.resource.cpu-vcores	24	1.5*16cpus
4. mapreduce.map.memory.mb	6144(6g)
5. mapreduce.reduce.memory.mb	8192(8g)
6. mapreduce.map.java.opts	-Xmx5120M	< #4
7. mapreduce.reduce.java.opts	-Xmx6144M	< #5

The memory of containers, nodemanager, map task, reduce task should based on system memory, cpu numbers.
There are 62.8g memory, 16cpus in nodemanager nodes. The memory could be like following.
Notice: Mostly, 4# = 2*2#, 5 = 2*4#. The following configurations are based on our mapreduce jobs.