通过二进制安装Hadoop集群

最新推荐文章于 2025-01-24 11:12:56 发布

MirkoLin596

最新推荐文章于 2025-01-24 11:12:56 发布

阅读量263

点赞数

CC 4.0 BY-SA版权

文章标签： hadoop 大数据 hdfs

本文链接：https://blog.youkuaiyun.com/weixin_45759901/article/details/115353016

本文详细指导如何在三台机器上通过二进制安装Hadoop，并配置核心、HDFS、MapReduce及YARN组件，包括配置文件设置、环境变量和权限管理，以及首次启动和验证步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

通过二进制安装Hadoop集群

此前需完成三台机器的免密登录

1.下载二进制Hadoop安装包(在三台节点上都执行)

[root@hadoop-master hadoop]# tar -xvzf hadoop-3.1.4.tar.gz -C /usr/local/hadoop/ 
[root@hadoop-master hadoop]# mkdir -p /usr/local/hadoop
[root@hadoop-master hadoop]# mv hadoop-3.1.4-src /usr/local/hadoop/
[root@hadoop-master hadoop]# cd /usr/local/hadoop/

在Hadoop-master上

2.配置Hadoop

[root@hadoop-master hadoop]# vim /usr/local/hadoop/hadoop-3.1.4/etc/hadoop/core-site.xml
[root@hadoop-master hadoop]# cat core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-master:9000</value>
</property>
</configuration>

3.修改hdfs-site.xml配置文件

[root@hadoop-master hadoop]#vim /usr/local/hadoop/hadoop-3.1.4/etc/hadoop/hdfs-site.xml
[root@hadoop-master hadoop]# cat hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/usr/local/hadoopdata/namenode</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/usr/local/hadoopdata/datanode</value>
	</property>
	<property>
		<name>dfs.secondary.http.address</name>
		<value>hadoop-datanode1:50090</value>
	</property>
	<property>
		<name>dfs.permissions.enabled</name>
		<value>false</value>
	</property>
</configuration>

4.修改slaves文件

[root@hadoop-master hadoop]#vi slaves
[root@hadoop-master hadoop]# cat slaves 
hadoop-master
hadoop-datanode1
hadoop-datanode2

5.修改mapred-site.xml文件

[root@hadoop-master hadoop]# vim mapred-site.xml
[root@hadoop-master hadoop]# cat mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>hadoop-master:10020</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>hadoop-master:19888</value>
	</property>

    <property>
      <name>mapreduce.application.classpath</name>
      <value>/usr/local/hadoop/hadoop-3.1.4/share/hadoop/mapreduce/*.jar</value>
    </property>



</configuration>

6.修改yarn-site.xml文件

[root@hadoop-master hadoop]# vim yarn-site.xml
[root@hadoop-master hadoop]# cat yarn-site.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
	<property>
		<name>yarn.resourcemanager.hostname</name>
		<value>hadoop-master</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	
	<property>
      <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
	
	<property>
		<name>yarn.application.classpath</name>
		<value>
			/usr/local/hadoop/hadoop-3.1.4/etc/hadoop,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/common/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/common/lib/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/hdfs/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/hdfs/lib/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/mapreduce/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/mapreduce/lib/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/yarn/*,
			/usr/local/hadoop/hadoop-3.1.4/share/hadoop/yarn/lib/*
		</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.scheduler.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>
	
	<property>
      <name>yarn.resourcemanager.webapp.address</name>
      <value>hadoop-master:8089</value>
    </property>
    
    <property>
      <name>yarn.resourcemanager.webapp.address.rm1</name>
      <value>hadoop-master:8089</value>
    </property>
	
	<property>
      <name>yarn.resourcemanager.webapp.address.rm2</name>
      <value>hadoop-datanode1:8088</value>
    </property>
	
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>yarn-cluster</value>
	</property>
	
	<property>
      <name>yarn.resourcemanager.address</name>
      <value>hadoop-master:8050</value>
    </property>
	
	<property>
      <name>yarn.resourcemanager.admin.address</name>
      <value>hadoop-master:8141</value>
    </property>
	
	<property>
      <name>yarn.resourcemanager.resource-tracker.address</name>
      <value>hadoop-master:8025</value>
    </property>
	
	<property>
      <name>yarn.resourcemanager.ha.rm-ids</name>
      <value>rm1,rm2</value>
    </property>
	
	<property>
		<name>yarn.resourcemanager.hostname.rm1</name>
		<value>hadoop-master</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.hostname.rm2</name>
		<value>hadoop-datanode1</value>
	</property>
	
	<property>
      <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
      <value>hadoop-master:8025</value>
    </property>
	
	<property>
      <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
      <value>hadoop-datanode1:8025</value>
    </property>
	
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>hadoop-master:2181,hadoop-datanode1:2181,hadoop-datanode2:2181</value>
	</property>
	
	<property>
      <name>yarn.nodemanager.log-aggregation.compression-type</name>
      <value>gz</value>
    </property>
    
    <property>
      <name>yarn.nodemanager.log-aggregation.debug-enabled</name>
      <value>false</value>
    </property>
    
    <property>
      <name>yarn.nodemanager.log-aggregation.num-log-files-per-app</name>
      <value>30</value>
    </property>
    
    <property>
      <name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
      <value>3600</value>
    </property>
    
    <property>
      <name>yarn.nodemanager.log-dirs</name>
      <value>/database/yarn/log</value>
    </property>
    
    <property>
      <name>yarn.nodemanager.log.retain-seconds</name>
      <value>604800</value>
    </property>
	
</configuration>

7.将配置拷贝到另外两台上

[root@hadoop-master hadoop]# scp -r /usr/local/hadoop/hadoop-3.1.4/etc/hadoop/* hadoop-datanode1:/usr/local/hadoop/hadoop-3.1.4/etc/hadoop/
[root@hadoop-master hadoop]# scp -r /usr/local/hadoop/hadoop-3.1.4/etc/hadoop/* hadoop-datanode2:/usr/local/hadoop/hadoop-3.1.4/etc/hadoop/

8.配置环境变量

[root@hadoop-master hadoop]# vim /etc/profile
[root@hadoop-master hadoop]# source /etc/profile
[root@hadoop-master hadoop]# tail /etc/profile

export JAVA_HOME=/usr/java/jre1.8.0_281-amd64
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

unset MAILCHECK

export HADOOP_HOME=/usr/local/hadoop/hadoop-3.1.4
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

9.启动验证

第一次启动HDFS之前，需要先对集群做格式化处理

[root@hadoop-master hadoop]# hdfs namenode -format

参考：https://blog.youkuaiyun.com/u010993514/article/details/82935834

[root@hadoop-master ~]# jps
13713 Bootstrap
13745 Bootstrap
15938 NodeManager
13378 jar
28435 Jps
15782 ResourceManager #yarn进程
13398 jar
18521 AzkabanWebServer
15116 DataNode
14959 NameNode

10.测试上传下载

[root@hadoop-master ~]# hadoop fs -ls /
Found 6 items
drwxr-xr-x   - root supergroup          0 2021-03-26 09:37 /imput
drwxrwxrwx   - root supergroup          0 2021-03-29 16:37 /input
drwxr-xr-x   - root supergroup          0 2021-03-29 17:12 /output
drwxr-xr-x   - root supergroup          0 2021-03-25 11:24 /output1
drwx-wx-wx   - root supergroup          0 2021-03-25 11:30 /tmp
drwxr-xr-x   - root supergroup          0 2021-03-26 09:18 /user