如何搭建hadoop集群

最新推荐文章于 2025-09-13 12:54:57 发布

iteye_4515

最新推荐文章于 2025-09-13 12:54:57 发布

阅读量77

点赞数

文章标签：大数据运维 java

今天决定重新搭建hadoop集群，如果是新手看这篇文章一定不要觉得很难，你只需要耐心和细心就ok了

准备工作

1.判断机器上是否有ssh服务，

[jinyang@OnlyKing Desktop]$ ssh -verison
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
Bad escape character 'rison'.

我的系统自带的，所以不用装了。

2.判断机器上是否有JDK

[jinyang@OnlyKing Desktop]$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (rhel-1.45.1.11.1.el6-i386)
OpenJDK Server VM (build 20.0-b12, mixed mode)
[jinyang@OnlyKing Desktop]$ javac -version
javac 1.6.0_24

如果是系统自带的JDK，最好重装一下。http://blog.youkuaiyun.com/a15039096218/article/details/7378293

进入主题

1.下载和安装hadoop，我下载的是hadoop-0.20.2.tar.gz

解压文件：[root@OnlyKing Downloads]#tar-zxvfhadoop-0.20.2.tar.gz

移动文件：[root@OnlyKing Downloads]#mvhadoop-0.20.2/usr/local/

安装文件：[root@OnlyKing Downloads]#ln-shadoop-0.20.2hadoop

2.修改环境变量

[root@OnlyKing local]#vi/etc/profile

在文件的下面添加，不能直接在文件的上面添加

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

[root@OnlyKing local]#./etc/profile

[root@OnlyKing local]#vi/usr/local/hadoop/conf/hadoop.env.sh(配置JAVA_HOME)

[root@OnlyKing Desktop]# hadoop version
Hadoop 0.20.2
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707
Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010

压轴大戏

1.NameNode配置

[hadoop@hadoop1 ~]# vi /etc/hosts
192.168.127.145 hadoop1
192.168.127.146 hadoop2
192.168.127.147 hadoop3
192.168.127.148 hadoop4

[root@OnlyKing conf]# vi core-site.xml

<configuration>
   <property>
     <name>fs.default.name</name>
     <value>hdfs://hadoop1:9000</value>
   </property>
</configuration>

[root@OnlyKing conf]# vi hdfs-site.xml

<configuration>
    <property>
       <name>dfs.replication</name>
       <value>3</value>
    </property>

    <property>
       <name>dfs.name.dir</name>
       <value>/usr/local/hadoop/namenode/</value>
    </property>

    <property>
       <name>hadoop.tmp.dir</name>
       <value>/usr/local/hadoop/tmp/</value>
    </property>
</configuration>

[root@OnlyKing conf]# vi mapred-site.xml

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>hadoop1:9001</value>
    </property>

    <property>
       <name>mapred.tasktracker.map.tasks.maximum</name>
       <value>4</value>
    </property>

    <property>
       <name>mapred.tasktracker.reduce.tasks.maximum</name>
       <value>4</value>
    </property>
</configuration>

datanode配置（只需修改hdfs-site.xml，mapred-site.xml和core-site.xml跟NameNode一样)
[hadoop@hadoop2 ~]$ vi hdfs-site.xml

    <property>
       <name>dfs.replication</name>
       <value>3</value>
    </property>

    <property>
       <name>dfs.data.dir</name>
       <value>/home/hadoop/data</value>
    </property>

    <property>
       <name>hadoop.tmp.dir</name>
       <value>/usr/local/hadoop/tmp/</value>
    </property>
<configuration>

[hadoop@hadoop1 conf]$ vi masters
hadoop1
[hadoop@hadoop1 conf]$ vi slaves
hadoop2
hadoop3
hadoop4
[hadoop@hadoop1 ~]$ start-all.sh
[hadoop@hadoop1 ~]$ stop-all.sh