hadoop2.6安装篇(hadoop平台搭建)

关注

原创最新推荐文章于 2026-03-14 11:23:54 发布 788 阅读 AI 写同款· GEO 优化 ›

Hadoop-2.6.2-cdh5.16.2安装部署+Hadoop包以及jdk包下载 1.hadoop下载我下载的是hadoop-2.6.0.-cdh5.16.2.tar.gz 如果有想换其他版本的可以去官网下载：https://hadoop.apache.org 下面是官网打开初始界面往下拉到底部，选择Release archive,点击进去进去以后各种版本任你挑选，选中你想要下载的版本，点击Release 最后再点击Download,下载tar包下载完Hadoop再进行jdk下载（jdk版本参考Hadoop推荐的版本，不然有可能你的j.... 阅读详情

上篇已经介绍完CentOS7的安装。现在来说说hadoop的安装.

上 hadoop.apache.org 下载2.6的版本。

安装jdk, ssh, rsync, wget

>yum -y install java-1.8.0-openjdk-devel

>yum -y install openssh

>yum -y install rsync

>yum -y install wget

>wget http://apache.fayea.com/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

>tar xzvf hadoop-2.6.0.tar.gz

>mv hadoop-2.6.0 /usr/local/

设置ssh免密码登录

Now check that you can ssh to the localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

CentOS 7.0默认使用的是firewall作为防火墙，这里改为iptables防火墙。
firewall：
systemctl start firewalld.service#启动firewall
systemctl stop firewalld.service#停止firewall
systemctl disable firewalld.service#禁止firewall开机启动

设置DNS

>vi /etc/hosts

    192.168.1.147 Master.Hadoop

    192.168.1.148 Slave1.Hadoop

    192.168.1.149 Slave2.Hadoop

    192.168.1.150 Slave3.Hadoop

>mv /usr/local/hadoop-2.6.0 /usr/local/hadoop

>chown -R hadoop:hadoop hadoop

>cd /usr/local/hadoop

>vi etc/hadoop/hadoop-env.sh

# set to the root of your Java installation export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.45-30.b13.el7_1.x86_64/jre # Assuming your installation directory is /usr/local/hadoop export HADOOP_PREFIX=/usr/local/hadoop

>bin/hadoop

test OK

配置cluster

etc/hadoop/core-site.xml:

[root@systdt hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/hadoop/tmp</value>
                <description>Abase for other temporary directories.</description>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://Master.Hadoop:9000</value>
        </property>
        <property>
                <name>io.file.buffer.size</name>
                <value>4096</value>
        </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:///usr/hadoop/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:///usr/hadoop/dfs/data</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>

    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-cluster1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>Master.Hadoop:50090</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

If set dfs.namenode.datanode.registration.ip-hostname-check=false, the name node will not do the check, which could be useful if your cluster is inside AWS VPC and you do not have proper reverse DNS.

Another usage of this feature is to decommission a data node by listing it in the hosts.deny file

etc/hadoop/mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
                <final>true</final>
        </property>

    <property>
        <name>mapreduce.jobtracker.http.address</name>
        <value>Master.Hadoop:50030</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>Master.Hadoop:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>Master.Hadoop:19888</value>
    </property>
        <property>
                <name>mapred.job.tracker</name>
                <value>http://Master.Hadoop:9001</value>
        </property>
</configuration>

etc/hadoop/yarn-site.xml:

<?xml version="1.0"?>
<configuration>

        
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>Master.Hadoop</value>
        </property>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>Master.Hadoop:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>Master.Hadoop:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>Master.Hadoop:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>Master.Hadoop:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>Master.Hadoop:8088</value>
    </property>
</configuration>