VMware虚拟机搭建Hadoop集群

最近搭建了hadoop2.5的集群，在这里分享一下希望能帮助到初学者，也为自己做下笔记。（一年前安装过hadoop0.23的集群，现在都不记得了）

[b]一、环境：[/b]
1、JDK1.7
2、Hadoop2.5.2
3、VMware8.5
4、CentOs6.4 32为
下载地址：[url]http://pan.baidu.com/s/1qW4saAS[/url]
[b]二、环境要求：[/b]
1、使用VMware创建三个CentOs6.4系统(创建一个复制两份)；
2、在每个操作系统上安装好jdk1.7，测试命令：java -verison看到如下输出就说明安装好了;
[code="shell"]
# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode, sharing)
[/code]
[b]三、安装规划：[/b]
一个nameNode,两个dataNode；
我本机的hosts如下：
192.168.80.130 hdp01nn #hdp=hadoop nn=nameNode dn=dataNode
192.168.80.131 hdp02dn
192.168.80.132 hdp03dn

[b]四、开始安装：[/b]
[b]1、修改虚拟系统的IP为静态分配、配置hosts文件[/b]
[b]1.1将IP地址改为静态指定[/b]
这一步需要修改两个文件：
[b]a)/etc/sysconfig/network[/b]
NETWORKING=yes
HOSTNAME=localhost.localdomain
[color=red]GATEWAY=192.168.80.2[/color]

[b]b)/etc/sysconfig/network-scripts/ifcfg-eth0[/b]
DEVICE="eth0"
[color=red]BOOTPROTO="static"
IPADDR=192.168.80.130
NETMASK=255.255.255.0[/color]
HWADDR="00:0C:29:0B:48:B5"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="ef892391-3951-46f6-8826-239cd2e5689b"
[color=red]DNS1=202.106.196.115[/color]

[b]查看虚拟机的网关和dns[/b]
edit>virtal network editor
选择VMNet8>NATSettings 看到如下界面
[img]http://dl2.iteye.com/upload/attachment/0104/1244/2dacba47-283e-38bf-8541-dae3aeaa4722.jpg[/img]

[color=brown]还有个文件/etc/resolv.conf会自动修改，如果没有自动修改可以参考如下文件：
# Generated by NetworkManager
nameserver 202.106.196.115[/color]

[b]重启网卡 service network restart[/b]
# service network restart
Shutting down interface eth0: Device state: 3 (disconnected)
[ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: Active connection state: activating
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/5
state: activated
Connection activated
[ OK ]
[color=orange]如果虚拟机是复制的这里你可能会遇到问题不能重启网络，你执行下ifconfig 找到正确的HWADDR地址，替换ifcfg-eth0里面的就可以重启了；[/color]
[b]1.2修改hosts文件[/b]
vi /etc/hosts #文件尾添加如下内容
192.168.80.130 hdp01nn
192.168.80.131 hdp02dn
192.168.80.132 hdp03dn


[b]2、生成SSH公共秘钥[/b]
[b]2.1安装和配置[/b]
安装 yum -y install openssh*
[b]2.2生成公共秘钥[/b]
cd $home\.ssh

[b]生成秘钥[/b]
ssh-keygen -t rsa
#保存到authorized_keys
cat id_rsa.pub >> authorized_keys
#传递到所有的节点(同一目录和用户名，第一次需要输入root密码)
scp authorized_keys root@hdp02dn:/root/.ssh/
#测试
ssh root@hdp02dn
不输入密码如果可以登陆表示配置成功
# ssh root@hdp02dn

Last login: Fri Dec 5 14:20:14 2014 from 192.168.80.1

[b]2.2关闭防火墙[/b]
#/etc/init.d/iptable stop 本次关闭
#chkconfig --level 35 iptables off 开机不启动防火墙


[b]3、修改Hadoop配置文件[/b]
配置文件位于：hadoop-2.5.1/etc/hadoop打开后将对应内容复制到指定文件中
[b]a)core-site.xml[/b]

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/hadoop01/hdptmp</value>
		<description>Abase for other temporary directories.</description>
	</property>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://hdp01nn:9000</value>
	</property>
	<property>
		<name>io.file.buffer.size</name>
		<value>4096</value>
	</property>
</configuration>

[b]b)hdfs-site.xml[/b]

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
	<property>
		<name>dfs.nameservices</name>
		<value>hadoop-cluster1</value>
	</property>
	<property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>hdp01nn:50090</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:///home/hadoop01/dfs/name</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:///home/hadoop/dfs/data</value>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
	<property>
		<name>dfs.webhdfs.enabled</name>
		<value>true</value>
	</property>
</configuration>

[b]c)mapred-site.xml[/b]

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<property>
		<name>mapreduce.jobtracker.http.address</name>
		<value>hdp01nn:50030</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>hdp01nn:10020</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>hdp01nn:19888</value>
	</property>
</configuration> 

[b]d)yarn-site.xml[/b]

<?xml version="1.0"?>
<configuration>
	<!-- Site specific YARN configuration properties -->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address</name>
		<value>hdp01nn:8032</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address</name>
		<value>hdp01nn:8030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address</name>
		<value>hdp01nn:8031</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address</name>
		<value>hdp01nn:8033</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address</name>
		<value>hdp01nn:8088</value>
	</property>
</configuration>

[b]e)slaves[/b]

hdp02dn
hdp03dn

[b]f)hadoop-env.sh和yarn-env.sh[/b]
找到第一个# export JAVA_HOME... 一行替换成如下内容(jdk地址是你自己的实际路径)

export JAVA_HOME=/usr/java/jdk1.7.0_65

[b]4、复制hadoop到所有slave节点[/b]
scp -r /root/devtools/hadoop-2.5.2 root@hdp02dn:/root/devtools/

[b]5、格式化namenode[/b]
bin/hdfs namenode -format

[b]6、启动和停止[/b]
hadoop有时间心跳机制，所以集群要配置时间同步；如果你没配置记得一定要统一时区(cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime)，统一系统时间(每个机器上执行：ntpdate time.windows.com)；

sbin/start-all.sh
sbin/stop-all.sh


[b]7、测试[/b]
[b]7.1浏览器测试[/b]
[color=orange]需要在你实体机上 hosts文件中配置映射路径,如果没配置就直接通过ip地址访问[/color]
访问：http://hdp01nn:50070
[img]http://dl2.iteye.com/upload/attachment/0104/1287/05303952-eb31-3d56-adec-0e5076894bcb.jpg[/img]
访问：http://hdp01nn:8088
[img]http://dl2.iteye.com/upload/attachment/0104/1289/a93a3ef3-d740-3b7e-b6ca-57d2a5ed4599.jpg[/img]
[b]7.2使用jps命令查看hadoop进程[/b]
namenode运行：
# jps
3850 NameNode
4193 ResourceManager
4581 Jps
slave运行:
# jps
3004 NodeManager
2901 DataNode
3773 Jps