How-to: enable fair scheduler in hadoop

本文详细介绍了如何在Apache Hadoop环境中配置Fair Scheduler,包括修改配置文件和YARN-site.xml来实现资源公平分配策略。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >


Steps:
  • Add following in yarn-site.xml:
            <property>
                    <name>yarn.resourcemanager.scheduler.class</name>
                    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
            </property>
            <property>
                    <name>yarn.resourcemanager.webapp.https.address.rm2</name>
                    <value>master02.dc.tj:8089</value>
            </property>
            <property>
                    <name>yarn.scheduler.fair.allocation.file</name>
                    <value>${HADOOP_HOME}/etc/hadoop/fair-scheduler.xml</value>
            </property>
            <property>
                    <name>yarn.scheduler.fair.user-as-default-queue</name>
                    <value>true</value>
            </property>
            <property>
                    <name>yarn.scheduler.fair.preemption</name>
                    <value>true</value>
            </property>
  • Generate file ${HADOOP_HOME}/etc/hadoop/fair-scheduler.xml:
    <?xml version="1.0"?>
    <allocations>
    <!--Currently for jobs submitted with user hadoop(hadoop admin) -->
      <queue name="hadoop">
        <aclSubmitApps>hadoop</aclSubmitApps>
        <aclAdministerApps>hadoop</aclAdministerApps>
        <minResources>5000 mb,0vcores</minResources>
        <maxResources>90000 mb,0vcores</maxResources>
        <maxRunningApps>100</maxRunningApps>
        <maxAMShare>0.5</maxAMShare>
        <weight>10.0</weight>
      </queue>


    <!--Currently for jobs submitted with user chenfangfang -->
      <queue name="chenfangfang">
         <aclSubmitApps>chenfangfang</aclSubmitApps>
         <aclAdministerApps>hadoop,chenfangfang</aclAdministerApps>
         <minResources>5000 mb,0vcores</minResources>
         <maxResources>45000 mb,0vcores</maxResources>
         <maxRunningApps>30</maxRunningApps>
         <maxAMShare>0.5</maxAMShare>
         <weight>1.0</weight>
      </queue>
      <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>

      <!-- All other users besides hadoop and chenfangfang, will be included in clients queue -->
      <queue name="clients" type="parent">
        <aclSubmitApps>*</aclSubmitApps>
        <aclAdministerApps>hadoop</aclAdministerApps>
        <minResources>5000 mb,0vcores</minResources>
        <maxResources>45000 mb,0vcores</maxResources>
        <maxRunningApps>20</maxRunningApps>
        <maxAMShare>0.5</maxAMShare>
        <weight>1.0</weight>
      </queue>
                
      <queuePlacementPolicy>
        <!-- Firstly, use spacified queue name as submitted queue-->
        <rule name="specified" create="false" />
        <!-- Then, use submitter user name as submitted queue--> 
        <rule name="user" create="false" />
        <!-- Other jobs use clients queue as submitted queue--> 
        <rule name="default" queue="clients" />
      </queuePlacementPolicy>
    </allocations>
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值