YARN(含MR2)常用配置

YARN架构图

YARN工作机制

调度器

资源调度器的类

yarn.resourcemanager.scheduler.class

原文：

The class to use as the resource scheduler.

译文：

资源调度器的类
容量调度器是 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
公平调度器是 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

最大优先级

yarn.cluster.max-application-priority

原文

Defines maximum application priority in a cluster.
If an application is submitted with a priority higher than this value, it will be reset to this maximum value.

译文

定义集群中应用程序的最大优先级

处理调度器请求的线程数量

yarn.resourcemanager.scheduler.client.thread-count

原文

Number of threads to handle scheduler interface.

译文

处理调度器接口的线程数

NodeManager

单节点 NM 可分配的物理内存

yarn.nodemanager.resource.memory-mb

原文：

Amount of physical memory, in MB, that can be allocated for containers.
If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux).
In other cases, the default is 8192MB.

译文：

当前节点NodeManager可分配给容器们的物理内存量（以MB为单位）
如果设置为-1且 yarn.nodemanager.resource.detect-hardware-capabilities为true，就会自动计算
其它情况默认8192MB

例如有10个NM，每个 NM 配置内存50G，则总内存是500G

单节点 NM 可分配的虚拟核心数

yarn.nodemanager.resource.cpu-vcores

原文

Number of vcores that can be allocated for containers.
This is used by the RM scheduler when allocating resources for containers.
This is not used to limit the number of CPUs used by YARN containers.
If it is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically determined from the hardware in case of Windows and Linux.
In other cases, number of vcores is 8 by default.

译文

当前节点NodeManager可分配给容器们的虚拟核心数
如果设置为-1且 yarn.nodemanager.resource.detect-hardware-capabilities是true，就依据硬件来自动确定虚拟核数
其它情况默认8

单节点预留给非YARN进程的物理内存总量

yarn.nodemanager.resource.system-reserved-memory-mb

原文：

Amount of physical memory, in MB, that is reserved for non-YARN processes.
This configuration is only used if yarn.nodemanager.resource.detect-hardware-capabilities is set to true and yarn.nodemanager.resource.memory-mb is -1.
If set to -1, this amount is calculated as 20% of (system memory - 2* HADOOP_HEAPSIZE)

译文：

预留给非YARN进程的物理内存总量，单位为MB
当 yarn.nodemanager.resource.detect-hardware-capabilities为true且 yarn.nodemanager.resource.memory-mb为-1时生效
如果设置为-1， 计算值 = ( 系统内存 − 2 × H A D O O P _ H E A P S I Z E ) × 20 % 计算值=(系统内存 - 2 \times HADOOP\_HEAPSIZE)\times 20 \%