Hadoop安装新手教程手把手

最新推荐文章于 2025-08-18 10:00:00 发布

原创最新推荐文章于 2025-08-18 10:00:00 发布 · 241 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #linux #大数据 #centos

大数据专栏收录该内容

23 篇文章

订阅专栏

本文详细介绍如何在单机环境下配置Hadoop，包括虚拟机设置、JDK安装、SSH免密登录、网络配置、主机名设定及Hadoop各组件配置。通过步骤化的指导，帮助读者完成从环境搭建到Hadoop集群的初始化。

一、配置虚拟机（配置好jdk）
配置
生成秘钥： ssh-keygen -t rsa -P “”
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
ssh-copy-id -i .ssh/id_rsa.pub -p22 root@192.168.xxx.xxx

免密登录
配置网卡
cd /etc/sysconfig/network-scripts/
修改：
ifcfg-enp0s3
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.xxx.xxx
ifcfg-enp0s8
BOOTPROTO=dhcp
ONBOOT=yes

配置完成后重启网卡
重新启动网络：systemctl restart network或者service network restart
配置主机名

vi /etc/hostname
vi /etc/hosts                                           # 编辑下hosts文件， 给127.0.0.1添加hostname
cat /etc/hosts                                         
  # 检查127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.xxx.xxx  hadoop01
 vi /etc/hostname
 hadoop01

hadoop加zookeeper的配置文件等

2.配置Hadoop
配置Hadoop ./etc/hadoop目录下的文件

hadoop-env.sh：

'#' The java implementation to use.
export JAVA_HOME=/opt/jdk1.8.0_221

core-site.xml

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.xxx.xxx:9000</value>
</property>
<property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/hadoop/tmp</value>
</property>
<property>
   <name>hadoop.proxyuser.root.hosts</name>
   <value>*</value>
 </property>
<property>
   <name>hadoop.proxyuser.root.groups</name>
   <value>*</value>
 </property>
</configuration>

hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>192.168.xxx.xxx:50090</value>
</property>
------单机用下面
<property>
  <name>dfs.permissions.enabled</name>
  <value>false</value>
</property>
-------
</configuration>

mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>192.168.xxx.xxx:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>192.168.xxx.xxx:19888</value>
</property>
</configuration>

yarn-site.xml

<configuration>
<!-- reducer获取数据方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>*HostName*</value>
</property>
<!-- 日志聚集功能使用 -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 日志保留时间设置7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>
</configuration>

vi ./slaves
localhost
单机不配集群配


3.2Hadoop环境变量配置
vi /etc/profile
export HADOOP_HOME=*/opt/hadoop*
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

再source /etc/profile
3.3格式化HDFS
hadoop namenode -format
3.4启动hadoop
start-all.sh
启动历史服务
[root@qx1 hadoop]# mr-jobhistory-daemon.sh start historyserver

在这里插入图片描述
3.5访问Hadoop
http://192.168.56.110:50070/