Hadoop MapReduce2 -单节点集群配置

本文详细介绍了如何从源代码构建MapReduce环境,并指导如何配置MapReduce与YARN的相关参数,包括设置环境变量、搭建配置文件以及启动相关守护进程。通过实践步骤,帮助开发者快速部署并运行MapReduce实例。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Mapreduce 压缩包
你可以从release下载一个MapReduce压缩包。如果没有你可以自己创建一个压缩包。
	$ mvn clean install -DskipTests
	$ cd hadoop-mapreduce-project
	$ mvn clean install assembly:assembly -Pnative
	注意:你需要安装protoc2.5
	忽略本地的MapReduce你可以忽略maven的 —Pnative参数。这个压缩包应该在target/directory下。
设置环境变量
	假设你已经安装了hadoop-common/hadoop-hdfs,并设置了 $HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME
解压hadoop-mapreduce压缩包设置环境变量 $HADOOP_MAPRED_HOME 设置$HADOOP_YARN_HOME的环境变量和 $HADOOP_MAPRED_HOME 一样。

搭建配置文件
	要启动ResourceManger和NodeManager,需要设置配置信息。假设$HADOOP_CONF_DIR 是配置HDFS和yarn-site.xml的配置目录。有两个文件你必须进行设置:mapred-site.xml and yarn-site.xml
配置mapred-site.xml:
	添加以下配置到你的mapred-site.xml
	<property>
    <name>mapreduce.cluster.temp.dir</name>
    <value></value>
    <description>No description</description>
    <final>true</final>
  </property>

  <property>
    <name>mapreduce.cluster.local.dir</name>
    <value></value>
    <description>No description</description>
    <final>true</final>
  </property>
设置yarn-site.xml
  添加以下的配置到你的yarn-site.xml中
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>host:port</value>
    <description>host is the hostname of the resource manager and 
    port is the port on which the NodeManagers contact the Resource Manager.
    </description>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>host:port</value>
    <description>host is the hostname of the resourcemanager and port is the port
    on which the Applications in the cluster talk to the Resource Manager.
    </description>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    <description>In case you do not want to use the default scheduler</description>
  </property>

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>host:port</value>
    <description>the host is the hostname of the ResourceManager and the port is the port on
    which the clients can talk to the Resource Manager. </description>
  </property>

  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value></value>
    <description>the local directories used by the nodemanager</description>
  </property>

  <property>
    <name>yarn.nodemanager.address</name>
    <value>0.0.0.0:port</value>
    <description>the nodemanagers bind to this port</description>
  </property>  

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>10240</value>
    <description>the amount of memory on the NodeManager in GB</description>
  </property>
 
  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/app-logs</value>
    <description>directory on hdfs where the application logs are moved to </description>
  </property>

   <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value></value>
    <description>the directories used by Nodemanagers as log directories</description>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <description>shuffle service that needs to be set for Map Reduce to run </description>
  </property>
接下来配置capacity-scheduler.xml
   添加以下配置到capacity-scheduler.xml
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>unfunded,default</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.unfunded.capacity</name>
    <value>50</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>50</value>
  </property>

运行deamons
   假设环境变量$HADOOP_COMMON_HOME$HADOOP_HDFS_HOME$HADOO_MAPRED_HOME$HADOOP_YARN_HOME,$JAVA_HOME and $HADOOP_CONF_DIR已经正确配置。设置$$YARN_CONF_DIR和$HADOOP_CONF_DIR配置也一样。

运行ResourceManager和NodeManager需要以下命令:
    $ cd $HADOOP_MAPRED_HOME
  $ sbin/yarn-daemon.sh start resourcemanager
  $ sbin/yarn-daemon.sh start nodemanager
如果ResourceManager和NodeManager能启动运行,你可以运行randomwriter例子测试:
  $HADOOP_COMMON_HOME/bin/hadoop jar hadoop-examples.jar randomwriter out
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值