yarn label scheduler

最新推荐文章于 2024-11-30 12:57:32 发布

原创最新推荐文章于 2024-11-30 12:57:32 发布 · 244 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

11 篇文章

订阅专栏

本文详细介绍了如何在YARN中配置节点标签和资源调度策略，包括修改yarn-site.xml和capacity-scheduler.xml文件，添加和分配节点标签，以及调整队列参数来实现更精细的资源管理和任务调度。

1、在yarn-site.xml中添加以下属性

   <property>
      <name>yarn.node-labels.enabled</name>
      <value>true</value>
   </property>
   <property>
      <name>yarn.node-labels.fs-store.root-dir</name>
      <value>/yarn/node-label</value>
   </property>
   <property>
      <name>yarn.node-labels.manager-class</name>
      <value>org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager</value>
   </property>

2、在capacity-scheduler.xml添加以下属性

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,other</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>20000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-applications</name>
    <value>10000</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.other.maximum-applications</name>
    <value>10000</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.other.capacity</name>
    <value>0</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.other.accessible-node-labels</name>
    <value>high_disk</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.high_disk.capacity</name>
    <value>100</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.other.accessible-node-labels.high_disk.capacity</name>
    <value>100</value>
  </property>


  <property>
    <name>yarn.scheduler.capacity.root.other.default-node-label-expression</name>
    <value>high_disk</value>
  </property>

3、添加标签

yarn rmadmin -addToClusterNodeLabels high_disk #多个标签用逗号分隔

4、给节点添加标签

yarn rmadmin -replaceLabelsOnNode 192.168.1.101,high_disk #ip后可以跟端口，若没有端口，则该节点上所有nodemanager都是一个标签，添加完并刷新后在web ui 中可以看到每个节点的node labels

5、刷新队列配置

yarn rmadmin -refreshQueues