hadoop-10 资源管理器yarn

本文深入探讨了YARN集群资源管理器中三种核心调度器的工作原理:FIFOScheduler、CapacityScheduler与FairScheduler。FIFOScheduler采用简单直接的先进先出原则;CapacityScheduler则通过多队列实现资源的公平分配,特别适用于需要隔离的离线和实时任务;而FairScheduler确保了多任务间的资源公平共享。

yarn管理集群中的资源,主要包含内存和处理器资源,分配资源。

yarn目前支持三种调度器:

FIFO Scheduler:先进先出,hadoop1中默认的一种调度器,如果一个离线任务占用资源太多,对实时任务不友好。只有一个队列

CapacityScheduler:可以看做是FIFOScheduler的多队列版本,在队列内部为先进先出,eg:离线任务队列和实时任务队列,两个队列之间是隔离的

FairScheduler:多队列,多任务共享资源。公平调度器,每个任务都有一定的资源,如果任务过多,每个任务可以分得的资源很少,也不太好。

stu@stu-pc:~$ find / -name "hadoop" -type d 2>/dev/null | grep -i hadoop/home/stu/hadoop/usr/local/hadoop/usr/local/hadoop/etc/hadoop/usr/local/hadoop/share/hadoop/usr/local/hadoop/share/doc/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-app/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/org/apache/hadoop/lib/service/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-auth/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-api/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-registry/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-client/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/api/org/apache/hadoop/lib/service/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs-client/build/source/hadoop-hdfs-project/hadoop-hdfs-client/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-common/build/source/hadoop-common-project/hadoop-common/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs-rbf/build/source/hadoop-hdfs-project/hadoop-hdfs-rbf/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/build/source/hadoop-hdfs-project/hadoop-hdfs/target/api/org/apache/hadoop/usr/local/hive/lib/php/packages/serde/org/apache/hadoop/usr/local/hive/apache-hive-3.1.2-bin/lib/php/packages/serde/org/apache/hadoop/tmp/hadoop/tmp/hadoop-unjar8342885896351558271/org/apache/hadoop/tmp/hadoop-unjar6484726597599168525/org/apache/hadoopstu@stu-pc:~$ ls /usr/local/hadoopbin data etc include lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share tmpstu@stu-pc:~$ ls /opt/hadoopls: 无法访问'/opt/hadoop': 没有那个文件或目录stu@stu-pc:~$
最新发布
12-09
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值