linux hadoop完全分布式集群安装

最新推荐文章于 2024-12-20 20:31:24 发布

原创最新推荐文章于 2024-12-20 20:31:24 发布 · 183 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

10 篇文章

订阅专栏

下载地址：
http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz

快速清理残留进程
rm -rf /tmp/hsperfdata_*

修改权限
sudo chown -R hadoop:hadoop app

hadoop-env.sh、mapred-env.sh、yarn-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64

core-site.xml

<property>
<name>fs.defaultFS</name>
 <value>hdfs://node1.paa.name:8020</value>
 </property>
 
 <property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/app/hadoop-2.8.5/data/tmp</value>
 </property>
 
<property>
  <name>hadoop.http.staticuser.user</name>
  <value>yourname</value>
 </property>

hdfs-site.xml

 <property>
  <name>dfs.permissions.enabled</name>
  <value>false</value>
 </property>

 <property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>node3.paa,name:50090</value>
 </property>

mapred-site.xml

 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.address</name>
  <value>node1.paa.name:10020</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>node1.paa.name:19888</value>
 </property>

yarn-site.xml

 <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
 </property>
 <property>
  <name>yarn.resourcemanager.hostname</name>
  <value>node2.paa.name</value>
 </property>
  <property>
  <name>yarn.log-aggregation-enable</name>
  <value>true</value>
 </property>
  <property>
  <name>yarn.log-aggregation.retain-seconds</name>
  <value>106800</value>
 </property>

slaves

node2.paa.name
node1.paa.name
node3.paa.name

删除share目录下的doc目录
配置完以后使用scp分发：scp -r hadoop root@node2.paa.name:/opt/app
格式化：bin/hdfs namenode -format
启动集群

验证：
1、在hdfs上创建文件夹，存放输入数据
2、bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /test/input /test/output3