搭建HDFS集群

克隆三台linux虚拟机

1、保证三台虚拟机在同一个网段

master:192.168.1.90

slave1: 192.168.1.91

slave2: 192.168.1.92

2、修改主机名

vim /etc/sysconfig/network

3、关闭防火墙和selinux

vim /etc/sysconfig/selinux

4、域名解析

vim /etc/hosts

192.168.1.90 master

192.168.1.91 slave1

192.168.1.92 slave2

5、配置ssh等效性

ssh-keygen -t rsa

6、将id_rsa.pub重写到authorized_keys

cat id_rsa.pub >authorized_keys

7、向slave1发送一份，将slave1的追加上之后，再发送至slave2，slave2追加完成后分别回传给master机器和slave1机器

scp authorized_keys slave1:~/.ssh/

8、修改authorized_keys拥有者权限

chmod 600 authorized_keys

配置JDK

1、配置java环境变量

Vim .bashrc

export JAVA_HOME=/usr/java/jdk1.6.0_27

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib

2、使环境变量生效

. .bashrc

配置Hadoop

1、 vim hadoop-env.sh

2、 vim core-site.xml

<configuration>

       <property>

               <name>fs.default.name</name>

               <value>hdfs://master:9000</value>

       </property>

       <property>

               <name>hadoop.tmp.dir</name>

               <value>/home/hadoop/tmp</value>

       </property>

</configuration>

3、vim hdfs-site.xml

<configuration>

       <property>

               <name>dfs.data.dir</name>

               <value>/data/hadoop</value>

       </property>

       <property>

               <name>dfs.replication</name>

                <value>2</value>

       </property>

</configuration>

4、vimmapred-site.xml

<configuration>

       <property>

               <name>mapred.job.tracker</name>

                <value>master:9001</value>

       </property>

</configuration>

5、vim masters

6、vim slaves

7、在slave节点上创建data/hadoop/

chown hadoop:hadoop data/hadoop/

8、修改环境变量

Vim .bashrc

export JAVA_HOME=/usr/java/jdk1.6.0_27

export HADOOP1_HOME=/home/hadoop/hadoop

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP1_HOME/bin

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib

9、是环境变量生效

. .bashrc

启动集群

1、 格式化namenode

hadoop namenode –format

2、 启动集群

start-all.sh