Hadoop 伪分布式运行模式

最新推荐文章于 2024-04-26 16:14:14 发布

VanasWang

最新推荐文章于 2024-04-26 16:14:14 发布

阅读量269

点赞数

分类专栏： Hadoop 文章标签： hadoop 大数据

本文链接：https://blog.youkuaiyun.com/VanasWang/article/details/105441661

版权

Hadoop 专栏收录该内容

8 篇文章

订阅专栏

伪分布式

启动HDFS

（1）配置集群

[vanas@hadoop132 ~]$ cd /opt/
[vanas@hadoop132 opt]$ cd module/hadoop-3.1.3/etc/hadoop/
[vanas@hadoop132 hadoop]$ vim hadoop-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_212

[vanas@hadoop132 hadoop]$ vim core-site.xml 
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
    <value>hdfs://hadoop101:9820</value>
</property>

<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
	<name>hadoop.tmp.dir</name>
	<value>/opt/module/hadoop-3.1.3/data/tmp</value>
</property>


[vanas@hadoop132 hadoop]$ vim hdfs-site.xml
<!-- 指定HDFS副本的数量 -->
<property>
	<name>dfs.replication</name>
	<value>1</value>
</property>

格式化后多了data 和logs

[vanas@hadoop132 hadoop]$ cd ..
[vanas@hadoop132 etc]$ cd ..
[vanas@hadoop132 hadoop-3.1.3]$ ll
总用量 204
drwxr-xr-x. 2 vanas vanas   4096 9月  12 2019 bin
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 etc
drwxr-xr-x. 2 vanas vanas   4096 9月  12 2019 include
drwxrwxr-x. 2 vanas vanas   4096 4月   8 16:43 input
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 lib
drwxr-xr-x. 4 vanas vanas   4096 9月  12 2019 libexec
-rw-rw-r--. 1 vanas vanas 147145 9月   4 2019 LICENSE.txt
-rw-rw-r--. 1 vanas vanas  21867 9月   4 2019 NOTICE.txt
-rw-rw-r--. 1 vanas vanas   1366 9月   4 2019 README.txt
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 sbin
drwxr-xr-x. 4 vanas vanas   4096 9月  12 2019 share
[vanas@hadoop132 hadoop-3.1.3]$ hdfs namenode -format

[vanas@hadoop132 hadoop-3.1.3]$ ll
总用量 212
drwxr-xr-x. 2 vanas vanas   4096 9月  12 2019 bin
drwxrwxr-x. 3 vanas vanas   4096 4月  10 09:49 data
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 etc
drwxr-xr-x. 2 vanas vanas   4096 9月  12 2019 include
drwxrwxr-x. 2 vanas vanas   4096 4月   8 16:43 input
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 lib
drwxr-xr-x. 4 vanas vanas   4096 9月  12 2019 libexec
-rw-rw-r--. 1 vanas vanas 147145 9月   4 2019 LICENSE.txt
drwxrwxr-x. 2 vanas vanas   4096 4月  10 09:49 logs
-rw-rw-r--. 1 vanas vanas  21867 9月   4 2019 NOTICE.txt
-rw-rw-r--. 1 vanas vanas   1366 9月   4 2019 README.txt
drwxr-xr-x. 3 vanas vanas   4096 9月  12 2019 sbin
drwxr-xr-x. 4 vanas vanas   4096 9月  12 2019 share

（2）启动、测试集群增、删、查

注意：jps是JDK中的命令，不是Linux命令。不安装JDK不能使用jps

[vanas@hadoop132 hadoop-3.1.3]$ hdfs --daemon start namenode
[vanas@hadoop132 hadoop-3.1.3]$ hdfs --daemon start datanode
[vanas@hadoop132 hadoop-3.1.3]$ jps
14027 Jps
13149 NameNode
13694 DataNode
[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -mkdir /input
[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -put input/my.txt /input
[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - vanas supergroup          0 2020-04-10 11:57 /input      1
[vanas@hadoop132 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

web端查看HDFS文件系统

http://hadoop132:9870

启动YARN并运行MapReduce程序

（1）配置集群在YARN上运行MR

[vanas@hadoop132 logs]$ cd ..
[vanas@hadoop132 hadoop-3.1.3]$ cd etc/hadoop/
[vanas@hadoop132 hadoop]$ vim yarn-site.xml 
<!-- Reducer获取数据的方式 -->
<property>
 		<name>yarn.nodemanager.aux-services</name>
 		<value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop101</value>
</property>
	  <property>
        <name>yarn.nodemanager.env-whitelist</name>        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>
 
[vanas@hadoop132 hadoop]$ vim mapred-site.xml 
<!-- 指定MR运行在YARN上 -->

<property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

</property>

（2）启动、测试集群增、删、查

[vanas@hadoop132 hadoop]$ cd ..
[vanas@hadoop132 etc]$ cd ..
[vanas@hadoop132 hadoop-3.1.3]$ jps
94464 Jps
13149 NameNode
13694 DataNode

[vanas@hadoop132 hadoop-3.1.3]$ yarn --daemon start resourcemanager
[vanas@hadoop132 hadoop-3.1.3]$ yarn --daemon start nodemanager
[vanas@hadoop132 hadoop-3.1.3]$ jps
97376 Jps
97207 NodeManager
13149 NameNode
95791 ResourceManager
13694 DataNode

（3）在YARN上执行WordCount案例
删除output否则运行失败或者改为output1

[vanas@hadoop132 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -ls /
Found 4 items
drwxr-xr-x   - vanas supergroup          0 2020-04-10 11:57 /input
drwxr-xr-x   - vanas supergroup          0 2020-04-10 12:55 /output
drwx------   - vanas supergroup          0 2020-04-10 11:04 /tmp

[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -ls /output
Found 2 items
-rw-r--r--   1 vanas supergroup          0 2020-04-10 12:55 /output/_SUCCESS
-rw-r--r--   1 vanas supergroup         30 2020-04-10 12:55 /output/part-r-00000
[vanas@hadoop132 hadoop-3.1.3]$ hdfs dfs -cat /output/part-r-00000
2020-04-10 12:56:23,251 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
dfs0509 1
dfs111  1
dfsvanas  1

YARN的浏览器页面查看

http://hadoop132:8088