搭建hadoop集群

1.安装虚拟机

•  修改ip: vi   /etc/sysconfig/network-scripts/ifcfg-eth0
• 修改主机名:   vi  /etc/sysconfig/network
• 修改域名(映射): vi  /etc/hosts
• 本地的hosts文件也改   
• 关闭防火墙   chkconfig  iptables off
• ping 一下,看看通不通

2.克隆虚拟机

• 改网卡:  vi   /etc/udev/rules.d/70-persistent-net.rules
  • 删除eth0,修改eth1为eth0
• 改ip:  vi  /etc/sysconfig/network-scripts/ifcfg-eth0
  • 改IPADDR的值
• reboot

3.用shell编程写免密登录

vi   auto.sh

#!/bin/bash
#yum安装expect
yun install openssh-clients -y
yum -y install expect
#PWD_1是登陆密码，可以自己设定
PWD_1=123456
ips=$(cat /etc/hosts |grep -v "::" | grep -v "127.0.0.1")
key_generate() {
    expect -c "set timeout -1;
        spawn ssh-keygen -t rsa;
        expect {
            {Enter file in which to save the key*} {send -- \r;exp_continue}
            {Enter passphrase*} {send -- \r;exp_continue}
            {Enter same passphrase again:} {send -- \r;exp_continue}
            {Overwrite (y/n)*} {send -- n\r;exp_continue}
            eof             {exit 0;}
    };"
}
auto_ssh_copy_id () {
    expect -c "set timeout -1;
        spawn ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@$1;
            expect {
                {Are you sure you want to continue connecting *} {send -- yes\r;exp_continue;}
                {*password:} {send -- $2\r;exp_continue;}
                eof {exit 0;}
            };"
}
rm -rf ~/.ssh

key_generate

for ip in $ips
do
    auto_ssh_copy_id $ip  $PWD_1
done

 4.安装JDK1.8

• 拖入jdk-8u141-linux-x64.tar.gz
  • 解压 tar -zxvf jdk-8u141-linux-x64.tar.gz  -C /usr/local
  • vi   /etc/profile  修改配置
    • export JAVA_HOME=/usr/local/jdk1.8.0_141
    • export PATH=$PATH:$JAVA_HOME/bin
    • source /etc/profile    
    • 检测  java -version

5.安装HADOOP

• 上传   hadoop-2.7.3.tar.gz
• 解压tar -zxvf hadoop-2.7.3.tar.gz   -C  /usr/local
• cd   /usr/local/hadoop-2.7.3/etc/hadoop 开始配置文件
• 配置文件
• vi  hadoop-env.sh

• export JAVA_HOME=/usr/local/jdk1.8.0_102

• vi  core-site.xml

• <configuration>
  <property>
  <name>fs.defaultFS</name>
  <value>hdfs://hadoop01:9000</value>  //看你的虚拟机主机名叫什么
  </property>
  <property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop-2.7.3/tmp</value>
  </property>
  </configuration>

   

• vi  hdfs-site.xml

• <configuration>
  <property>
  <name>dfs.namenode.name.dir</name>
  <value>/usr/local/hadoop-2.7.3/data/name</value>
  </property>
  <property>
  <name>dfs.datanode.data.dir</name>
  <value>/usr/local/hadoop-2.7.3/data/data</value>
  </property>
  
  <property>
  <name>dfs.replication</name>
  <value>3</value>
  </property>
  
  <property>
  <name>dfs.secondary.http.address</name>
  <value>hadoop01:50090</value>    //看主机名来改
  </property>
  </configuration>

• cp mapred-site.xml.tmp*   mapred-site.xml

• vi  mapred-site.xml

• <configuration>
  <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  </property>
  </configuration>

• vi  yarn-site.xml

• <configuration>
  <property>
  <name>yarn.resourcemanager.hostname</name>
  <value>hadoop01</value>      //看主机名
  </property>
  
  <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
  </property>
  </configuration>

• vi  slaves(批量自启集群)

• hadoop01
  hadoop02
  hadoop03

• 修改  vi  /etc/profile
• export HADOOP_HOME=/usr/local/hadoop-2.7.3
• export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

6.把第一台安装好的配置发送给其他台

• hosts文件

  • scp  -r  /etc/hosts   hadoop02:/etc

• jdk安装后的文件夹

  • scp -r  /usr/local/jdk1.8.0_141  hadoop02:/usr/local/

  • Hadoop安装后的文件夹

    •      scp   -r  /usr/local/hadoop-2.7.3    hadoop02:/usr/local

    • /etc/profile 文件

      •      scp   -r /etc/profile   hadoop02:/etc/

7. 启动集群

• 初始化HDFS(在hadoop01进行操作)（操作一次就ok）

  • hadoop  namenode  -format

  • 启动HDFS  :      start-dfs.sh

    • 启动YARN:       start-yarn.sh

    • netstat -nltp   监听端口

网页查看:hadoop01:50070

查看错误日志:/usr/local/hadoop-2.7.3/logs/hadoop-root*.log