搭建Hadoop

最新推荐文章于 2025-09-06 22:11:59 发布

原创最新推荐文章于 2025-09-06 22:11:59 发布 · 119 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop

Hadoop 专栏收录该内容

9 篇文章

订阅专栏

本文详细介绍如何在虚拟机上配置Hadoop集群，包括虚拟机内存调整、SSH免密登录设置、JDK安装、Hadoop各配置文件详解及环境变量配置等步骤。

1.修改虚拟机内存为1GB

[img]http://dl2.iteye.com/upload/attachment/0124/7915/3ac9aeaf-5643-3e02-8e65-4c11325d39ba.png[/img]

2.Xshell / SercureCRT 客户端连接虚拟机

ctrl+alt+t --》打开 terminal 窗口
ifconfig --》查看IP地址

3.关闭防火墙
临时关闭
service iptables stop
永久关闭
chkconfig iptables off

4.修改主机名
查看主机名
hostname

修改主机名
vim /etc/sysconfig/network

修改后立即生效
source /etc/sysconfig/network

需重启
reboot

临时修改主机名
hostname xxxx

5.配置 hosts
vim /etc/hosts

[img]http://dl2.iteye.com/upload/attachment/0124/7917/1330e973-3233-38c4-bf5a-330cef65df93.png[/img]

6.配置免密登录

ssh root@xxx.xxx.xxx.xxx
需输入用户密码

配置
ssh-keygen 回车一路默认

找到生成的id_rsa.pub 文件（su root , cd /root 在 root 文件夹下）
ssh-copy-id -i .ssh/id_rsa.pub root@192.168.76.134

再次执行ssh root@xxx.xxx.xxx.xxx 无需密码

7.安装配置JDK

8.上传安装Hadoop

tar -zxvf hadoop.tar.gz

cd hadoop/etc/haddoop
--------------------------------------------------------------------------
vim hadoop-env.sh
i 插入

[img]http://dl2.iteye.com/upload/attachment/0124/8128/807d9205-f302-337e-a895-12c6d4445829.png[/img]

[img]http://dl2.iteye.com/upload/attachment/0124/8130/8cc7b16d-6fc4-3c2d-9448-86224b63b55c.png[/img]

ESC ：wq 保存

source hadoop-env.sh 立即生效

--------------------------------------------------------------------------

vim core-site.xml

[img]http://dl2.iteye.com/upload/attachment/0124/8132/e3d5500a-1b39-34be-8d32-3a4088b58a5f.png[/img]

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://linux01:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/software/hadoop/usr/local/software/hadoop-2.7.1/tmp</value>
</property>
</configuration>

cd ../hadoop-2.7.1
mkdir tmp

----------------------------------------------------------------------------
vim hdfs-site.xml

<configuration>
<property>
<name>dfs-replication</name>
<value>1</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/software/hadoop/usr/local/software/hadoop-2.7.1/tmp</value>
</property>

</configuration>

此处：datanode.dir 配置需要与 core-site.xml 中 datanode.dir 的配置一致
否则，上传时报错

[url]http://blog.sina.com.cn/s/blog_61d8d9640102whof.html[/url]

--------------------------------------------------------------------------

cp mapred-site.xml.template mapred-site.xml

vim mapred-site.xml

[img]http://dl2.iteye.com/upload/attachment/0124/8136/61576a88-c81d-3926-bc8d-50a2d6d8f75f.png[/img]

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

-------------------------------------------------------------------------

vim yarn-site.xml

[img]http://dl2.iteye.com/upload/attachment/0124/8138/30fc1000-92bf-313f-a076-ee179e857562.png[/img]

<configuration>


<property>
<name>yarn.resourcemanager.hostname</name>
<value>linux01</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

-------------------------------------------------------------------------

vim slaves

linux01

从属机器名称
--------------------------------------------------------------------------

配置环境变量
vim /etc/profile

[img]http://dl2.iteye.com/upload/attachment/0124/8140/8f8eeffb-0389-3c58-a584-1bafe8ab0207.png[/img]

source /etc/profile

9.测试
hadoop namenode -formate

cd /sbin
sh start-all.sh

jps 查看进程是否启动成功