MapReduce——2.运行WordCount

最新推荐文章于 2022-10-10 09:27:40 发布

ant_yi

最新推荐文章于 2022-10-10 09:27:40 发布

阅读量2.1k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： hadoop学习大数据学习文章标签： wordcount

本文链接：https://blog.youkuaiyun.com/weixin_42490528/article/details/89438859

大数据学习同时被 2 个专栏收录

53 篇文章

订阅专栏

hadoop学习

13 篇文章

订阅专栏

本文详细介绍如何使用Hadoop运行WordCount程序，包括启动Hadoop、上传文件至HDFS、执行WordCount任务及查看结果的全过程。同时，针对运行过程中可能出现的错误提供了具体的解决方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

hadoop实现了wordcount.java，并打好了jar包，在目录：

/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar

1.启动hadoop，start-all.sh

2.在hdfs上创建个待分析的文件

[root@master test]# hadoop fs -mkdir -p /test

[root@master test]# hadoop fs -put wordcount.txt /test

[root@master test]# hadoop fs -cat /test/wordcount.txt

3.运行wordcount

命令：hadoop jar jar包类名输入路径输出路径

[root@master mapreduce]# hadoop jar hadoop-mapreduce-examples-3.1.2.jar wordcount /test/wordcount.txt /result

4.查看结果：

[root@master ~]# hadoop fs -ls /result

Found 2 items

-rw-r--r-- 1 root supergroup 0 2019-04-21 20:05 /result/_SUCCESS

-rw-r--r-- 1 root supergroup 1594 2019-04-21 20:05 /result/part-r-00000

[root@master ~]#

结果太长，截选了一些

[root@master ~]# hadoop fs -cat /result/part-r-00000

ANY 1

ASF 1

Apache 2

BASIS, 1

distributed 3

file 3

for 2

governing 1

in 2

one 1

or 3

the 9

this 3

to 3

under 4

use 1

with 2

you 2

[root@master ~]#

遇到错误：

1.hadoop找不到或无法加载主类

org.apache.hadoop.mapreduce.v2.app.MRAppMaster

思路：找不到class path

解决：

查看class path，把得到的结果配置到yarn-site.xml中

[root@hadoop01 ~]# hadoop classpath

yarn-site.xml

<name>yarn.application.classpath</name> <value>/export/servers/hadoop/etc/hadoop:/export/servers/hadoop/share/hadoop/common/lib/*:/export/servers/hadoop/share/hadoop/common/*:/export/servers/hadoop/share/hadoop/hdfs:/export/servers/hadoop/share/hadoop/hdfs/lib/*:/export/servers/hadoop/share/hadoop/hdfs/*:/export/servers/hadoop/share/hadoop/mapreduce/*:/export/servers/hadoop/share/hadoop/yarn:/export/servers/hadoop/share/hadoop/yarn/lib/*:/export/servers/hadoop/share/hadoop/yarn/*</value>

</property>