YARN(含MR2)常用配置

YARN架构图

YARN工作机制

调度器

资源调度器的类

yarn.resourcemanager.scheduler.class

原文：

The class to use as the resource scheduler.

译文：

资源调度器的类
容量调度器是 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
公平调度器是 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

最大优先级

yarn.cluster.max-application-priority

原文

Defines maximum application priority in a cluster.
If an application is submitted with a priority higher than this value, it will be reset to this maximum value.

译文

定义集群中应用程序的最大优先级

处理调度器请求的线程数量

yarn.resourcemanager.scheduler.client.thread-count

原文

Number of threads to handle scheduler interface.

译文

处理调度器接口的线程数

NodeManager

单节点 NM 可分配的物理内存

yarn.nodemanager.resource.memory-mb

原文：

Amount of physical memory, in MB, that can be allocated for containers.
If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux).
In other cases, the default is 8192MB.

译文：

当前节点NodeManager可分配给容器们的物理内存量（以MB为单位）
如果设置为-1且 yarn.nodemanager.resource.detect-hardware-capabilities为true，就会自动计算
其它情况默认8192MB

例如有10个NM，每个 NM 配置内存50G，则总内存是500G

单节点 NM 可分配的虚拟核心数

yarn.nodemanager.resource.cpu-vcores

原文

Number of vcores that can be allocated for containers.
This is used by the RM scheduler when allocating resources for containers.
This is not used to limit the number of CPUs used by YARN containers.
If it is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically determined from the hardware in case of Windows and Linux.
In other cases, number of vcores is 8 by default.

译文

当前节点NodeManager可分配给容器们的虚拟核心数
如果设置为-1且 yarn.nodemanager.resource.detect-hardware-capabilities是true，就依据硬件来自动确定虚拟核数
其它情况默认8

单节点预留给非YARN进程的物理内存总量

yarn.nodemanager.resource.system-reserved-memory-mb

原文：

Amount of physical memory, in MB, that is reserved for non-YARN processes.
This configuration is only used if yarn.nodemanager.resource.detect-hardware-capabilities is set to true and yarn.nodemanager.resource.memory-mb is -1.
If set to -1, this amount is calculated as 20% of (system memory - 2* HADOOP_HEAPSIZE)

译文：

预留给非YARN进程的物理内存总量，单位为MB
当 yarn.nodemanager.resource.detect-hardware-capabilities为true且 yarn.nodemanager.resource.memory-mb为-1时生效
如果设置为-1， $计算值=(系统内存-2\times HADOOP\_HEAPSIZE)\times 20\%$

每个容器可分配的最小内存

yarn.scheduler.minimum-allocation-mb

原文：

The minimum allocation for every container request at the RM in MBs.
Memory requests lower than this will be set to the value of this property.
Additionally, a node manager that is configured to have less memory than this value will be shut down by the resource manager.

译文：

每个 容器（向RM请求）可分配的最小内存，单位MB
低于此值的内存请求将被设置为此属性的值
此外，内存小于此值的NM节点将被RM关闭

每个容器可分配的最大内存

yarn.scheduler.maximum-allocation-mb

原文：

The maximum allocation for every container request at the RM in MBs.
Memory requests higher than this will throw an InvalidResourceRequestException.

译文：

每个 容器（向RM请求）可分配的最大内存，单位MB
高于此值的内存请求将抛出 InvalidResourceRequestException

每个容器可分配的最少虚拟核数

yarn.scheduler.minimum-allocation-vcores

原文：

The minimum allocation for every container request at the RM in terms of virtual CPU cores.
Requests lower than this will be set to the value of this property.
Additionally, a node manager that is configured to have fewer virtual cores than this value will be shut down by the resource manager.

译文：

每个容器（向RM请求）可分配的最少虚拟核数
低于此值的请求将被设置为此属性的值
此外，虚拟核心数少于此值的NM节点将被RM关闭

每个容器可分配的最多虚拟核数

yarn.scheduler.maximum-allocation-vcores

原文：

The maximum allocation for every container request at the RM in terms of virtual CPU cores.
Requests higher than this will throw an InvalidResourceRequestException.

译文：

每个容器（向RM请求）可分配的最多虚拟核数
高于此值的请求将抛出 InvalidResourceRequestException

自动检测节点资源

yarn.nodemanager.resource.detect-hardware-capabilities

原文：

Enable auto-detection of node capabilities such as memory and CPU.

译文：

是否启用自动检测节点资源（内存、CPU…）
false为禁用，true为启用

虚拟核数和物理核数的转换乘数

yarn.nodemanager.resource.pcores-vcores-multiplier

原文：

Multiplier to determine how to convert phyiscal cores to vcores.
This value is used if yarn.nodemanager.resource.cpu-vcores is set to -1(which implies auto-calculate vcores) and yarn.nodemanager.resource.detect-hardware-capabilities is set to true.
The number of vcores will be calculated as number of CPUs * multiplier.

译文：

虚拟核数和物理核数的转换乘数
当 yarn.nodemanager.resource.cpu-vcores为-1且 yarn.nodemanager.resource.detect-hardware-capabilities为true时，此值生效
$虚拟核心总数=物理核心总数\times 转换乘数$

例如：4核8线程，该参数就设为2

容器们的CPU百分比限制

yarn.nodemanager.resource.percentage-physical-cpu-limit
Containers CPU Limit Percentage

原文：

Percentage of CPU that can be allocated for containers.
This setting allows users to limit the amount of CPU that YARN containers use.
Currently functional only on Linux using cgroups.
The default is to use 100% of CPU.

译文：

可分配给容器们的CPU百分比
用于限制YARN容器们使用的CPU数量
默认值100%

MapReduce

每个Map任务的虚拟核心数

mapreduce.map.cpu.vcores

原文：

The number of virtual cores to request from the scheduler for each map task.

每个Map任务的内存

mapreduce.map.memory.mb

原文：

The amount of memory to request from the scheduler for each map task.
If this is not specified or is non-positive, it is inferred from mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
If java-opts are also not specified, we set it to 1024.

译文

每个向调度器请求的Map任务的内存，单位MB
如果冇指定，就根据 mapreduce.map.java.opts和 mapreduce.job.heap.memory-mb.ratio来推断
如果 mapreduce.map.java.opts也没指定，就1024MB

单个Map任务内存应小于单个容器可分配的最大内存
单个Map任务内存应小于单节点 NM 可分配的物理内存的最大值

每个Reduce任务的虚拟核心数

mapreduce.reduce.cpu.vcores

原文：

The number of virtual cores to request from the scheduler for each reduce task.

每个Reduce任务的内存

mapreduce.reduce.memory.mb

原文：

The amount of memory to request from the scheduler for each reduce task.
If this is not specified or is non-positive, it is inferred from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
If java-opts are also not specified, we set it to 1024.

译文

每个向调度器请求的Reduce任务的内存，单位MB
如果冇指定，就根据 mapreduce.reduce.java.opts和 mapreduce.job.heap.memory-mb.ratio来推断
如果 mapreduce.reduce.java.opts也没指定，就1024MB

单个Reduce任务内存应小于单个容器可分配的最大内存
单个Reduce任务内存应小于单节点 NM 可分配的物理内存的最大值

堆大小与容器大小的比率

mapreduce.job.heap.memory-mb.ratio

原文：

The ratio of heap-size to container-size.
If no -Xmx is specified, it is calculated as ( mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio).
If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is calculated as (heapSize / mapreduce.heap.memory-mb.ratio).

译文

堆大小与容器大小的比率
当冇指定 -Xmx时：堆大小= mapreduce.{map|reduce}.memory.mb $\times$ mapreduce.heap.memory-mb.ratio
当指定了 -Xmx，但冇指定 mapreduce.{map|reduce}.memory. mb时： mapreduce.{map|reduce}.memory.mb=堆大小/ mapreduce.heap.memory-mb.ratio

说明

默认值是0.8；当 mapreduce.map.memory.mb=4G时，Map任务堆大小=3.2G

MR的ApplicationMaster所需的内存大小

yarn.app.mapreduce.am.resource.mb

原文：

The amount of memory the MR AppMaster needs.

MR的ApplicationMaster所需的虚拟核心数

yarn.app.mapreduce.am.resource.cpu-vcores

原文：

The number of virtual CPU cores the MR AppMaster needs.

---

Appendix

• 1个ApplicationMaster申请1个Container
• 1个Spark的Executor申请1个Container
• 1个MapTask申请1个Container
  1个ReduceTask申请1个Container
• 假设某节点服务器内存128G
  则
  该节点 NM 可分配的物理内存应当<128G，可以是64G、96G
• 假设有3个 NM 节点的可分配内存(G)为：8、8、16
  则
  $总内存(G)=8+8+16=32$
  则
  单个 容器 的最大内存 最多只能给到16G
  则
  单个Map任务的内存 不能大于16G
  • 假设1个作业的Map需要20G内存
    如果mapreduce.map.memory.mb设为8G，YARN就会分配3个8G的 容器，24>20浪费了4G
    如果mapreduce.map.memory.mb设为4G，YARN就会分配5个4G的 容器
  • 假设1个作业的Map需要1G内存
    如果mapreduce.map.memory.mb设为5G，YARN就会分配1个5G的 容器，
    多个小作业会把YARN可分配给容器的内存用满，
    用free -h可看到：实际并没有用到5G内存

en	🔉	cn
invalid	ɪnˈvælɪd	作废的；不能识别的；
infer	ɪnˈfɜːr	v. 推断
specify	ˈspesɪfaɪ	v. 明确指出；具体说明
convert	kənˈvɜːrt	v. （使）转换
multiplier	ˈmʌltɪplaɪər	n. [数] 乘数；[电子] 倍增器；增加者；繁殖者
imply	ɪmˈplaɪ	v. 暗示；意味着；必然包含
-Xms		初始Java堆内存大小
-Xmx		最大Java堆内存大小