hadoop2.6.0三个节点集群环境搭建(一)

本文详细介绍了一套完整的Hadoop集群搭建过程,包括操作系统配置、无密码登录设置、JDK安装、防火墙与SELinux关闭等前期准备。此外,还介绍了Hadoop软件的安装、配置文件调整以及格式化namenode并启动集群的具体步骤。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

一、操作系统配置


(1) 主机名更改
(2) Ip地址配置
(3) /etc/hosts文件配置
(4) 无密码登入
(5) Jdk安装
(6) /etc/hosts文件配置防火墙关闭
(7) Selinux关闭
(8) vm.swappiness参数配置
(9) 创建hadoop用户及目录


(1)主机名更改:


[root@elephant catchup]# vi /etc/sysconfig/network

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=elephant


(2)ip地址配置


[root@elephant catchup]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="static"
ONBOOT="yes"
IPADDR=192.168.123.1
[root@elephant catchup]# service network restart


(3) / etc/hosts文件配置

[root@elephant catchup]# vi /etc/hosts
127.0.0.1               localhost.localdomain localhost

::1             localhost6.localdomain6 localhost6
192.168.123.10 master
192.168.123.11 slave1
192.168.123.12 slave2
192.168.123.13 hadoop04  <===将来做hadoop节点添加用,现在不用


(4)无密码登入


su - hadoop

ssh-keygen -t rsa -P ''
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
scp .ssh/id_rsa.pub hadoop@master:/home/hadoop/id_rsa_03.pub
on master
cat id_rsa_03.pub >> .ssh/authorized_keys
scp .ssh/authorized_keys hadoop@hadoop04:/home/hadoop/.ssh/authorized_keys


(5)jdk安装


安装JDK1.7或者以上版本。这里安装jdk1.7.0_79。 

下载地址:http://www.oracle.com/technetwork/java/javase/downloads/index.html 


1,下载jdk-7u79-Linux-x64.gz,解压到/usr/Java/jdk1.7.0_79。 


2,在/root/.bash_profile中添加如下配置:

export JAVA_HOME=/usr/java/jdk1.7.0_79
export PATH=$JAVA_HOME/bin:$PATH


3,使环境变量生效,#source ~/.bash_profile 


4,安装验证# java -version 

Java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)


(6) 关闭防火墙


 [root@elephant catchup]# chkconfig iptables off

[root@elephant catchup]# /etc/init.d/iptables stop
[root@elephant catchup]# /etc/init.d/iptables status

iptables: Firewall is not running.


(7) Selinux关闭


[root@elephant catchup]# vi /etc/selinux/config

SELINUX=disabled


[root@elephant catchup]# setenforce 0  <===不用重启即可生效


(8) vm.swappiness参数配置


[root@elephant catchup]# vi /etc/sysctl.conf

vm.swappiness = 0


(9)创建hadoop用户及目录


su - root

useradd hadoop
passwd hadoop
usermod -g root hadoop

su - hadoop
mkdir -p /home/hadoop/dfs/name       《======根据自己的实际情况创建路径
mkdir -p /home/hadoop/dfs/data
mkdir -p /home/hadoop/tmp


二、安装hadoop


(1)解压tar -xzvf hadoop-2.6.0.tar.gz


(2)
move到指定目录下


mv hadoop-2.6.0 /opt/

ln –s hadoop-2.6.0 hadoop


(3)
修改配置文件


~/hadoop/etc/hadoop/hadoop-env.sh

~/hadoop/etc/hadoop/yarn-env.sh
~/hadoop/etc/hadoop/slaves
~/hadoop/etc/hadoop/core-site.xml
~/hadoop/etc/hadoop/hdfs-site.xml
~/hadoop/etc/hadoop/mapred-site.xml
~/hadoop/etc/hadoop/yarn-site.xm


1)~/hadoop/etc/hadoop/hadoop-env.sh

# The java implementation to use.

export JAVA_HOME=/usr/java/default


2)~/hadoop/etc/hadoop/yarn-env.sh

# some Java parameters
export JAVA_HOME=/usr/java/default


3)~/hadoop/etc/hadoop/slaves

slave1
slave2


4)~/hadoop/etc/hadoop/core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://master:9000</value>
 </property>
 <property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
 </property>
 <property>
  <name>hadoop.tmp.dir</name>
  <value>file:/home/spark/opt/hadoop-2.6.0/tmp</value>
  <description>Abasefor other temporary directories.</description>
 </property>


5)~/hadoop/etc/hadoop/hdfs-site.xml

<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>master:9001</value>
 </property>


  <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/opt/hadoop/dfs/name</value>
 </property>


 <property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/opt/hadoop/dfs/data</value>
  </property>


 <property>
  <name>dfs.replication</name>
  <value>2</value>
 </property>


 <property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
 </property>


6)~/hadoop/etc/hadoop/mapred-site.xml

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.address</name>
  <value>master:10020</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>master:19888</value>
 </property>


7)~/hadoop/etc/hadoop/yarn-site.xm

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
  </property>
  <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
   <name>yarn.resourcemanager.address</name>
   <value>master:8032</value>
  </property>
  <property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>master:8030</value>
  </property>
  <property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>master:8035</value>
  </property>
  <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>master:8033</value>
  </property>
  <property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>master:8088</value>
  </property>


(4)copy hadoop家目录到slave1、slave2节点


三、格式化namenode,并启动集群


[hadoop@master ~]$ hdfs namenode –format


y expiry time is 600000 millis
17/03/16 03:00:29 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/03/16 03:00:29 INFO util.GSet: VM type       = 64-bit
17/03/16 03:00:29 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/03/16 03:00:29 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/03/16 03:00:29 INFO namenode.NNConf: ACLs enabled? false
17/03/16 03:00:29 INFO namenode.NNConf: XAttrs enabled? true
17/03/16 03:00:29 INFO namenode.NNConf: Maximum size of an xattr: 16384
17/03/16 03:00:29 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1977394629-192.168.123.10-1489647629240
17/03/16 03:00:29 INFO common.Storage: Storage directory /opt/hadoop/dfs/name has been successfully formatted.  《==表名成功
17/03/16 03:00:30 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/03/16 03:00:30 INFO util.ExitUtil: Exiting with status 0
17/03/16 03:00:30 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.123.10
************************************************************/


[hadoop@master ~]$ start-dfs.sh 
17/03/16 04:07:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-namenode-master.out
slave2: starting datanode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-slave2.out
slave1: starting datanode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-slave1.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-master.out
17/03/16 04:08:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@master ~]$ start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave1: starting nodemanager, logging to /opt/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-slave2.out


[hadoop@master ~]$ hdfs dfsadmin -report
17/03/16 04:12:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 50395455488 (46.93 GB)
Present Capacity: 36794449920 (34.27 GB)
DFS Remaining: 36794400768 (34.27 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0


-------------------------------------------------
Live datanodes (2):


Name: 192.168.123.11:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 25197727744 (23.47 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6842331136 (6.37 GB)
DFS Remaining: 18355372032 (17.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 72.85%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 16 04:12:46 EDT 2017




Name: 192.168.123.12:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 25197727744 (23.47 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6758674432 (6.29 GB)
DFS Remaining: 18439028736 (17.17 GB)
DFS Used%: 0.00%
DFS Remaining%: 73.18%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 16 04:12:46 EDT 2017


[hadoop@master ~]$

http://master:50070



http://master:8088


评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值