hadoop部署

最新推荐文章于 2024-09-09 10:10:14 发布

yexueming1983

最新推荐文章于 2024-09-09 10:10:14 发布

阅读量116

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/yexueming1983/article/details/86562175

本文详细介绍Hadoop集群的搭建过程，包括环境准备、伪分布配置、核心文件修改及服务启动，确保集群稳定运行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、伪分布配置

su - root
visudo
username ALL=(ALL) ALL

mkdir /opt/xxx
sudo chown -R master:master *

tar -zxf jdk-7u67-linux-x64.tar.gz -C /opt/modules/
tar -xf jdk-7u67-linux-x64.tar.gz -C /opt/modules/

rpm -qa | grep java
sudo rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64

vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/hadoop-env.sh
/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/mapred-env.sh
/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/yarn-env.sh
export JAVA_HOME=/opt/modules/jdk1.7.0_67

/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/core-site.xml
<property>
<name>fs.defaultFS</name>

<value>hdfs://hadoop.master:8020</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/opt/modules/hadoop-2.6.0-cdh5.15.0/data/tmp</value>

</property>

/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/hdfs-site.xml
<property>

<name>dfs.replication</name>

</property>

/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/slaves
hostname

bin/hdfs namenode -format

sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode

close the firewall of VT and PC
hostname:50070 review namenode web UI

bin/hdfs dfs ...

/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.resourcemanager.hostname</name>

<value>hadoop.slave01</value>

</property>
/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop/mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

</property>

sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
hostname:8088

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.15.0.jar wordcount /user/slave01/data /user/slave01/output
bin/hdfs dfs -text /user/slave01/output/part*