hadoop2.*能力调度器capacity-scheduler

最新推荐文章于 2022-07-15 16:05:21 发布

Scojen

最新推荐文章于 2022-07-15 16:05:21 发布

阅读量606

点赞数

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签：能力调度器 capacity-scheduler

本文链接：https://blog.youkuaiyun.com/www_jun/article/details/49468993

Hadoop 专栏收录该内容

5 篇文章

订阅专栏

本文介绍如何在Hadoop集群中配置Capacity Scheduler，通过创建用户组dwgroup并调整yarn-site.xml及capacity-scheduler.xml来实现资源分配。具体包括设置默认队列与dwgroup队列的容量、最大容量、状态等参数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

capacity-scheduler能力调度器

1. 添加用户组dwgroup（集群中的namenode主备机器都要做以下操作）

groupadd dwgroup //添加用户组

useradd -G dwgroup 用户//将用户加到dwgroup用户组里

2. yarn-site.xml

新增配置

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

</property>

去掉的配置

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<name>yarn.scheduler.minimum-allocation-vcores</name>

</property>

3. capacity-scheduler.xml

更改的配置文件

<name>yarn.scheduler.capacity.root.queues</name>

<value>default,dwgroup</value>

The queues at the this level (root is theroot queue).

</description>

</property>

更改的配置

#--------------------default-----------------------------

<name>yarn.scheduler.capacity.root.default.capacity</name>

<description>Defaultqueue target capacity.</description>

</property>

<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>

Default queue user limita percentage from 0.0 to 1.0.

</description>

</property>

<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>

The maximum capacity ofthe default queue.

</description>

</property>

<name>yarn.scheduler.capacity.root.default.state</name>

<value>RUNNING</value>

The state of the defaultqueue. State can be one of RUNNING or STOPPED.

</description>

</property>

<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>

The ACL of who can submitjobs to the default queue.

</description>

</property>

<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>

The ACL of who canadminister jobs on the default queue.

</description>

</property>

#------------------------dwgroup -----------------------------------

<name>yarn.scheduler.capacity.root.dwgroup.capacity</name>

<description>Defaultqueue target capacity.</description>

</property>

<name>yarn.scheduler.capacity.root.dwgroup.user-limit-factor</name>

Default queue user limit apercentage from 0.0 to 1.0.

</description>

</property>

<name>yarn.scheduler.capacity.root.dwgroup.maximum-capacity</name>

The maximum capacity of thedefault queue.

</description>

</property>

<name>yarn.scheduler.capacity.root.dwgroup.state</name>

<value>RUNNING</value>

The state of the defaultqueue. State can be one of RUNNING or STOPPED.

</description>

</property>

<name>yarn.scheduler.capacity.root.dwgroup.acl_submit_applications</name>

The ACL of who can submitjobs to the default queue.

</description>

</property>

<name>yarn.scheduler.capacity.root.dwgroup.acl_administer_queue</name>

The ACL of who canadminister jobs on the default queue.

</description>

</property>

更改的配置

<name>yarn.scheduler.capacity.queue-mappings</name>

<value>g:dwgroup:dwgroup,g:%:default</value>

A list of mappings that will be used toassign jobs to queues

The syntax for this list is[u|g]:[name]:[queue_name][,next mapping]*

Typically this list will be used to mapusers to queues,

for example, u:%user:%user maps all usersto queues with the same name

as the user.

</description>

</property>

更改完以上3个配置文件让配置文件生效

/data/hadoop/bin下执行以下命令

yarnrmadmin –refreshQueues

或者重启yarn执行以下命令

/data/hadoop/sbin/stop-yarn.sh

/data/hadoop/sbin/start-yarn.sh

注意：后续有新增用户，集群中的每台机器都要创建该用户，并且指定对应的dwgroup用户组