Hadoop配置 Ⅱ

最新推荐文章于 2024-11-16 14:51:05 发布

原创最新推荐文章于 2024-11-16 14:51:05 发布 · 203 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hadoop

Ubuntu操作同时被 2 个专栏收录

10 篇文章

订阅专栏

Hadoop

3 篇文章

订阅专栏

本文详细介绍了Hadoop在完全分布式环境下的配置过程，包括修改虚拟机名称及目录、配置各节点的-site文件以指定具体IP地址，以及如何复制配置文件至其他虚拟机并验证配置成功。最后，讲解了分布式集群的启动步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Hadoop配置 Ⅱ

2.1 完全分布式配置

[外链图片转存失败(img-49vAdQDm-1562246814055)(en-resource://database/2268:0)]

2.1.1 修改虚拟的名称以及存放目录

1、修改名称

2、改变目录

3、修改虚拟机配置文件

`xuqiao@s0:/usr/soft/hadoop-3.1.2/etc/hadoop_cluster$ gedit *-site.xml`

修改各个-site文件的内容

在配置文件中指定具体的ip地址

core-site.xml

  <?xml version="1.0" encoding="UTF-8"?>

  <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

   <configuration>

  <property>
      <name>fs.defaultFS</name>
      <value>hdfs://s0/</value>
  </property>

   </configuration>`

hdfs-site.xml

  `<?xml version="1.0" encoding="UTF-8"?>
  <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  <configuration>
      <property>
          <name>dfs.replication</name>
          <value>2</value>
      </property>
  </configuration>`

mapred-site.xml

 `<?xml version="1.0"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 <configuration>
     <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
     </property>
 </configuration>`

yarn-site.xml

 `<?xml version="1.0"?>
 <configuration>
     <property>
         <name>yarn.resourcemanager.hostname</name>
         <value>s0</value>
     </property>
     <property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
     </property>
 </configuration>`

4、将这四个文件复制到其他虚拟机中

`xuqiao@s0:/usr/soft/hadoop-3.1.2/etc$ scp -r hadoop_cluster  xuqiao@s1:/usr/soft/hadoop-3.1.2/etc/
`

查看是否成功

xuqiao@s0:~$ ssh s1
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.18.0-25-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

14 packages can be updated.
0 updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2023.
Last login: Thu Jul  4 19:43:56 2019
# 1920x1080 59.96 Hz (CVT 2.07M9) hsync: 67.16 kHz; pclk: 173.00 MHz
Modeline "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync
Can't open display 
Can't open display 
Can't open display 
xuqiao@s1:~$ cd /usr/soft/hadoop-3.1.2/etc/hadoop_cluster/
xuqiao@s1:/usr/soft/hadoop-3.1.2/etc/hadoop_cluster$ cat core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://s0/</value>
    </property>
</configuration>

xuqiao@s1:/usr/soft/hadoop-3.1.2/etc/hadoop_cluster$

登陆上，显示已经成功，nice

复制到其他虚拟机中

输入代码查看下

  `xuqiao@s0:~$ ssh s3 cat /usr/soft/hadoop-3.1.2/etc/hadoop_cluster/core-site.xml
  <?xml version="1.0" encoding="UTF-8"?>
  <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://s0/</value>
      </property>
  </configuration>
  `

完成~