centos7+hadoop2.7.3集群环境搭建（3个datanode）

最新推荐文章于 2021-07-24 11:05:06 发布

ARM小妖

最新推荐文章于 2021-07-24 11:05:06 发布

阅读量755

点赞数

CC 4.0 BY-SA版权

文章标签：大数据

本文链接：https://blog.youkuaiyun.com/yang849315801/article/details/89338001

本文详细介绍了如何在Vmvare上安装CentOS7，并配置Hadoop2.7.3的完全分布式集群，包括安装JDK、配置静态IP和主机名、设置环境变量、克隆虚拟机、实现SSH无密登录、配置Hadoop相关XML文件、创建符号链接以及启动和检查Hadoop进程。同时提到了通过WEBUI进行监控。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Vmvare安装Centos7

不再赘述

ip地址主机名修改

修改静态ip

1.切换root用户
	$>su root
2.编辑/etc/sysconfig/network-scripts/ifcfg-eno16777736
	a.备份文件
		$>cd /etc/sysconfig/network-scripts
		$>cp ifcfg-eno16777736 ifcfg-eno16777736.bak
	b.进入/etc/sysconfig/network-scripts
		$>cd /etc/sysconfig/network-scripts
	c.编辑ifcfg-eno16777736文件
		    TYPE=Ethernet					
			BOOTPROTO=none					
			DEFROUTE=yes					
			IPV4_FAILURE_FATAL=no			
			IPV6INIT=no
			IPV6_AUTOCONF=no
			IPV6_DEFROUTE=no
			IPV6_PEERDNS=no
			IPV6_PEERROUTES=no
			IPV6_FAILURE_FATAL=no
			NAME=eno16777736
			UUID=33f3ce5f-8b5c-41af-90ed-863736e09c29
			DEVICE=eno16777736
			ONBOOT=yes
			IPADDR=10.3.40.21
			PREFIX=24
			GATEWAY=10.
			DNS=192.168.231.2
	d.重启网络服务
		$>su root
		$>service network restart
	e.编辑/etc/resolv.conf,添加名称服务器，内容是网关地址。
        nameserver 10.1.0.1
        nameserver 10.3.1.4
	f.重启网络服务
		$>su root
		$>service network restart

修改主机名

1./etc/hostname
	s201
2./etc/hosts
	127.0.0.1 localhost
	10.3.40.21 s201
	10.3.40.22 s202
	10.3.40.23 s203
	10.3.40.24 s204

hadoop安装

安装jdk

	a)下载jdk-8u65-linux-x64.tar.gz
	b)tar开
		$>su centos ; cd ~
		$>mkdir downloads
		$>cp /mnt/hdfs/downloads/bigdata/jdk-8u65-linux-x64.tar.gz ~/downlooads
		$>tar -xzvf jdk-8u65-linux-x64.tar.gz
	c)创建/soft文件夹
		$>sudo mkdir /soft
		$>sudo chown centos:centos /soft
	d)移动tar开的文件到/soft下
		$>mv ~/downloads/jdk-1.8.0_65 /soft/
	e)创建符号连接
		$>ln -s /soft/jdk-1.8.0_65 /soft/jdk
	f)验证jdk安装是否成功
		$>cd /soft/jdk/bin
		$>./java -version

安装hadoop

	a)下载hadoop-2.7.3.tar.gz
	b)tar开
		$>su centos ; cd ~
		$>cp /mnt/hdfs/downloads/bigdata/hadoop-2.7.3.tar.gz ~/downloads
		$>tar -xzvf hadoop-2.7.3.tar.gz
	c)无
	d)移动tar开的文件到/soft下
		$>mv ~/downloads/hadoop-2.7.3 /soft/
	e)创建符号连接
		$>ln -s /soft/hadoop-2.7.3 /soft/hadoop
	f)验证jdk安装是否成功
		$>cd /soft/hadoop/bin
		$>./hadoop version

设置环境变量

编辑/etc/profile

    export JAVA_HOME=/soft/jdk
    export PATH=$PATH:$JAVA_HOME/bin
    
    export HADOOP_HOME=/soft/hadoop
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    		
    export PS1='[\u@\h `pwd`]\$'

$>source /etc/profile

配置完全分布式

虚拟机克隆

1.克隆3台client(centos7)
	右键centos-7-->管理->克隆-> ... -> 完整克隆
2.启动client
3.启用客户机共享文件夹。
4.修改hostname和ip地址文件
	[/etc/hostname]
	s202

	[/etc/sysconfig/network-scripts/ifcfg-ethxxxx]
	...
	IPADDR=..

5.重启网络服务
	$>sudo service network restart

6.修改/etc/resolv.conf文件

7.重复以上3 ~ 6过程.

ssh无密登录

1.在每台主机上创建/home/centos/.ssh，并清空ssh目录内部所有文件，修改ssh目录权限为700
    $>mkdir ~/.ssh
    $>chmod 700 ~/.ssh

2.在s201主机上生成密钥对
	$>ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

3.将s201的公钥文件id_rsa.pub远程复制到202 ~ 204主机上。
  并放置/home/centos/.ssh/authorized_keys
  	$>scp id_rsa.pub centos@s201:/home/centos/.ssh/authorized_keys
	$>scp id_rsa.pub centos@s202:/home/centos/.ssh/authorized_keys
	$>scp id_rsa.pub centos@s203:/home/centos/.ssh/authorized_keys
	$>scp id_rsa.pub centos@s204:/home/centos/.ssh/authorized_keys

配置完全分布式(${hadoop_home}/etc/hadoop/)

配置文件

[core-site.xml]

		<?xml version="1.0" encoding="UTF-8"?>
		<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
		<configuration>
				<property>
						<name>fs.defaultFS</name>
						<value>hdfs://s201/</value>
				</property>
		</configuration>

[hdfs-site.xml]

		<?xml version="1.0" encoding="UTF-8"?>
		<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
		<configuration>
				<property>
						<name>dfs.replication</name>
						<value>3</value>
				</property>
		</configuration>

[mapred-site.xml]

注意:cp mapred-site.xml.template mapred-site.xml

		<?xml version="1.0"?>
		<configuration>
			<property>
				<name>mapreduce.framework.name</name>
				<value>yarn</value>
			</property>
		</configuration>

[yarn-site.xml]

		<?xml version="1.0"?>
		<configuration>
				<property>
						<name>yarn.resourcemanager.hostname</name>
						<value>s201</value>
				</property>
				<property>
						<name>yarn.nodemanager.aux-services</name>
						<value>mapreduce_shuffle</value>
				</property>
		</configuration>

[slaves]

		s202
		s203
		s204

[hadoop-env.sh]

		export JAVA_HOME=/soft/jdk
		...

脚本文件

xcall.sh

#!/bin/bash

params=$@
i=201
for (( i=201 ; i <= 204 ; i = $i + 1 )) ; do
    echo ============= s$i $params =============
    ssh s$i "$params"
done

xsync.sh

#!/bin/bash

if [[ $# -lt 1 ]] ; then echo no params ; exit ; fi

p=$1
dir=`dirname $p`

filename=`basename $p`

cd $dir
fullpath=`pwd -P .`

user=`whoami`
for (( i = 202 ; i <= 204 ; i = $i + 1 )) ; do
   echo ======= s$i =======
   rsync -lr $p ${user}@s$i:$fullpath
done ;

分发配置

$> xsync.sh /soft/hadoop/etc/full

创建符号链接

$> xcall.sh "ln -sfT /soft/hadoop/etc/full /soft/hadoop/etc/hadoop"

启动hadoop

格式化文件系统

$>hadoop namenode -format

启动hadoop进程

$>start-all.sh

查看启动进程

使用xcall.sh在所有节点上创建jps符号连接，指向/soft/jdk/bin/jps

	1.切换到root用户
		$>su root
	2.创建符号连接
		$>xcall.sh "ln -sfT /soft/jdk/bin/jps /usr/local/bin/jps"
	3.修改jps符号连接的owner
		$>xcall.sh "chown -h centos:centos /usr/local/bin/jps"
	4.查看所有主机上的java进程
		$>xcall.sh jps


========== s201 jps ===========
17937 Jps
15602 NameNode
16106 ResourceManager
15883 SecondaryNameNode
========== s202 jps ===========
8627 DataNode
8835 NodeManager
10591 Jps
========== s203 jps ===========
7376 NodeManager
7169 DataNode
8746 Jps
========== s204 jps ===========
7713 NodeManager
7507 DataNode
8603 Jps

	[hdfs]start-dfs.sh
	NameNode			NN
	DataNode			DN
	SecondaryNamenode	2NN

	[yarn]start-yarn.sh
	ResourceMananger	RM
	NodeManager			NM

查错

1.删除临时目录文件

$>cd /tmp
$>rm -rf hadoop-centos
$>ssh s202 rm -rf /tmp/hadoop-centos
$>ssh s203 rm -rf /tmp/hadoop-centos
$>ssh s204 rm -rf /tmp/hadoop-centos

2.删除hadoop日志

$>cd /soft/hadoop/logs
$>rm -rf *
$>ssh s202 rm -rf /soft/hadoop/logs/*
$>ssh s203 rm -rf /soft/hadoop/logs/*
$>ssh s204 rm -rf /soft/hadoop/logs/*

WEBUI

http://localhost:50070/

hadoop的端口

50070		//namenode http port
50075		//datanode http port
50090		//2namenode	http port

8020		//namenode rpc port
50010		//datanode rpc port

hadoop四大模块

common
hdfs		//namenode + datanode + secondarynamenode

mapred
yarn		//resourcemanager + nodemanager

在这里插入图片描述