文章目录
Vmvare安装Centos7
不再赘述
ip地址主机名修改
修改静态ip
1.切换root用户
$>su root
2.编辑/etc/sysconfig/network-scripts/ifcfg-eno16777736
a.备份文件
$>cd /etc/sysconfig/network-scripts
$>cp ifcfg-eno16777736 ifcfg-eno16777736.bak
b.进入/etc/sysconfig/network-scripts
$>cd /etc/sysconfig/network-scripts
c.编辑ifcfg-eno16777736文件
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
NAME=eno16777736
UUID=33f3ce5f-8b5c-41af-90ed-863736e09c29
DEVICE=eno16777736
ONBOOT=yes
IPADDR=10.3.40.21
PREFIX=24
GATEWAY=10.
DNS=192.168.231.2
d.重启网络服务
$>su root
$>service network restart
e.编辑/etc/resolv.conf,添加名称服务器,内容是网关地址。
nameserver 10.1.0.1
nameserver 10.3.1.4
f.重启网络服务
$>su root
$>service network restart
修改主机名
1./etc/hostname
s201
2./etc/hosts
127.0.0.1 localhost
10.3.40.21 s201
10.3.40.22 s202
10.3.40.23 s203
10.3.40.24 s204
hadoop安装
安装jdk
a)下载jdk-8u65-linux-x64.tar.gz
b)tar开
$>su centos ; cd ~
$>mkdir downloads
$>cp /mnt/hdfs/downloads/bigdata/jdk-8u65-linux-x64.tar.gz ~/downlooads
$>tar -xzvf jdk-8u65-linux-x64.tar.gz
c)创建/soft文件夹
$>sudo mkdir /soft
$>sudo chown centos:centos /soft
d)移动tar开的文件到/soft下
$>mv ~/downloads/jdk-1.8.0_65 /soft/
e)创建符号连接
$>ln -s /soft/jdk-1.8.0_65 /soft/jdk
f)验证jdk安装是否成功
$>cd /soft/jdk/bin
$>./java -version
安装hadoop
a)下载hadoop-2.7.3.tar.gz
b)tar开
$>su centos ; cd ~
$>cp /mnt/hdfs/downloads/bigdata/hadoop-2.7.3.tar.gz ~/downloads
$>tar -xzvf hadoop-2.7.3.tar.gz
c)无
d)移动tar开的文件到/soft下
$>mv ~/downloads/hadoop-2.7.3 /soft/
e)创建符号连接
$>ln -s /soft/hadoop-2.7.3 /soft/hadoop
f)验证jdk安装是否成功
$>cd /soft/hadoop/bin
$>./hadoop version
设置环境变量
编辑/etc/profile
export JAVA_HOME=/soft/jdk
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/soft/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PS1='[\u@\h `pwd`]\$'
$>source /etc/profile
配置完全分布式
虚拟机克隆
1.克隆3台client(centos7)
右键centos-7-->管理->克隆-> ... -> 完整克隆
2.启动client
3.启用客户机共享文件夹。
4.修改hostname和ip地址文件
[/etc/hostname]
s202
[/etc/sysconfig/network-scripts/ifcfg-ethxxxx]
...
IPADDR=..
5.重启网络服务
$>sudo service network restart
6.修改/etc/resolv.conf文件
7.重复以上3 ~ 6过程.
ssh无密登录
1.在每台主机上创建/home/centos/.ssh,并清空ssh目录内部所有文件,修改ssh目录权限为700
$>mkdir ~/.ssh
$>chmod 700 ~/.ssh
2.在s201主机上生成密钥对
$>ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
3.将s201的公钥文件id_rsa.pub远程复制到202 ~ 204主机上。
并放置/home/centos/.ssh/authorized_keys
$>scp id_rsa.pub centos@s201:/home/centos/.ssh/authorized_keys
$>scp id_rsa.pub centos@s202:/home/centos/.ssh/authorized_keys
$>scp id_rsa.pub centos@s203:/home/centos/.ssh/authorized_keys
$>scp id_rsa.pub centos@s204:/home/centos/.ssh/authorized_keys
配置完全分布式(${hadoop_home}/etc/hadoop/)
配置文件
- [core-site.xml]
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://s201/</value>
</property>
</configuration>
- [hdfs-site.xml]
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
- [mapred-site.xml]
注意:cp mapred-site.xml.template mapred-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
- [yarn-site.xml]
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>s201</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
- [slaves]
s202
s203
s204
export JAVA_HOME=/soft/jdk
...
脚本文件
#!/bin/bash
params=$@
i=201
for (( i=201 ; i <= 204 ; i = $i + 1 )) ; do
echo ============= s$i $params =============
ssh s$i "$params"
done
#!/bin/bash
if [[ $# -lt 1 ]] ; then echo no params ; exit ; fi
p=$1
dir=`dirname $p`
filename=`basename $p`
cd $dir
fullpath=`pwd -P .`
user=`whoami`
for (( i = 202 ; i <= 204 ; i = $i + 1 )) ; do
echo ======= s$i =======
rsync -lr $p ${user}@s$i:$fullpath
done ;
分发配置
$> xsync.sh /soft/hadoop/etc/full
创建符号链接
$> xcall.sh "ln -sfT /soft/hadoop/etc/full /soft/hadoop/etc/hadoop"
启动hadoop
格式化文件系统
$>hadoop namenode -format
启动hadoop进程
$>start-all.sh
查看启动进程
- 使用xcall.sh在所有节点上创建jps符号连接,指向/soft/jdk/bin/jps
1.切换到root用户
$>su root
2.创建符号连接
$>xcall.sh "ln -sfT /soft/jdk/bin/jps /usr/local/bin/jps"
3.修改jps符号连接的owner
$>xcall.sh "chown -h centos:centos /usr/local/bin/jps"
4.查看所有主机上的java进程
$>xcall.sh jps
========== s201 jps ===========
17937 Jps
15602 NameNode
16106 ResourceManager
15883 SecondaryNameNode
========== s202 jps ===========
8627 DataNode
8835 NodeManager
10591 Jps
========== s203 jps ===========
7376 NodeManager
7169 DataNode
8746 Jps
========== s204 jps ===========
7713 NodeManager
7507 DataNode
8603 Jps
[hdfs]start-dfs.sh
NameNode NN
DataNode DN
SecondaryNamenode 2NN
[yarn]start-yarn.sh
ResourceMananger RM
NodeManager NM
查错
1.删除临时目录文件
$>cd /tmp
$>rm -rf hadoop-centos
$>ssh s202 rm -rf /tmp/hadoop-centos
$>ssh s203 rm -rf /tmp/hadoop-centos
$>ssh s204 rm -rf /tmp/hadoop-centos
2.删除hadoop日志
$>cd /soft/hadoop/logs
$>rm -rf *
$>ssh s202 rm -rf /soft/hadoop/logs/*
$>ssh s203 rm -rf /soft/hadoop/logs/*
$>ssh s204 rm -rf /soft/hadoop/logs/*
WEBUI
http://localhost:50070/
hadoop的端口
50070 //namenode http port
50075 //datanode http port
50090 //2namenode http port
8020 //namenode rpc port
50010 //datanode rpc port
hadoop四大模块
common
hdfs //namenode + datanode + secondarynamenode
mapred
yarn //resourcemanager + nodemanager