十五、Hadoop MapReduce之WordCount极速入门

最新推荐文章于 2025-05-14 09:30:00 发布

原创

最新推荐文章于 2025-05-14 09:30:00 发布 · 444 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#mapreduce #wordcount

本文介绍了如何使用Hadoop的MapReduce框架进行WordCount操作，包括通过hadoop-mapreduce-examples-2.8.5.jar计算圆周率，以及WordCount应用的基本原理和输入输出格式。同时，提到了在不同Hadoop模式下运行WordCount以及解决异常的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、利用Hadoop中自带的hadoop-mapreduce-examples-2.8.5.jar中含有的pi示例计算圆周率

$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar pi 5 5

第一个5代表Number of Maps,总共运行10次map任务;Samples per Map则代表每次map任务投掷5次，pi 5 5最后得出的结果是3.68···，当Number of Maps和Samples per Map越大求出的value就越接近π（圆周率）。
在这里插入图片描述

二、通过shell利用wordcount示例统计单词数

[ljj@master hadoopeco]$ cd $HADOOP_HOME/temp
[ljj@master temp]$ mkdir file
[ljj@master temp]$ cd file
[ljj@master file]$ echo "Hello World Bye World" > file01.txt
[ljj@master file]$ echo "Hello Hadoop Goodbye Hadoop" > file02.txt
[ljj@master file]$ ls
file01.txt  file02.txt
[ljj@master file]$ cat file01.txt
Hello World Bye World
[ljj@master file]$ cat file02.txt
Hello Hadoop Goodbye Hadoop
[ljj@master file]$ hdfs dfs -ls /tmp 
Found 3 items
drwx------   - ljj supergroup          0 2019-01-22 15:43 /tmp/hadoop-yarn
drwx-wx-wx   - ljj supergroup          0 2019-01-16 19:23 /tmp/hive
drwxr-xr-x   - ljj supergroup          0 2019-01-22 15:46 /tmp/tianchi
[ljj@master file]$ hdfs dfs -mkdir /tmp/wc 
[ljj@master file]$ hdfs dfs -mkdir /tmp/wc/input
[ljj@master file]$ hdfs dfs -put $HADOOP_HOME/temp/file/file01.txt /tmp/wc/input
[ljj@master file]$ hdfs dfs -put $HADOOP_HOME/temp/file/file02.txt /tmp/wc/input
[ljj@master file]$ hdfs dfs -ls /tmp/wc/input
Found 2 items
-rw-r--r--   1 ljj supergroup         22 2019-01-23 12:31 /tmp/wc/input/file01.txt
-rw-r--r--   1 ljj supergroup         28 2019-01-23 12:31 /tmp/wc/input/file02.txt
[ljj@master file]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /tmp/wc/input /tmp/wc/output19/01/23 12:34:45 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/01/23 12:34:46 INFO input.FileInputFormat: Total input files to process : 2
19/01/23 12:34:46 INFO mapreduce.JobSubmitter: number of splits:2
19/01/23 12:34:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1548259150075_0002
19/01/23 12:34:47 INFO impl.YarnClientImpl: Submitted application application_1548259150075_0002
19/01/23 12:34:47 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1548259150075_0002/
19/01/23 12:34:47 INFO mapreduce.Job: Running job: job_1548259150075_0002
19/01/23 12:34:54 INFO mapreduce.Job: Job job_1548259150075_0002 running in uber mode : false
19/01/23 12:34:54 INFO mapreduce.Job:  map 0% reduce 0%
19/01/23 12:35:00 INFO mapreduce.Job:  map 100% reduce 0%
19/01/23 12:35:05 INFO mapreduce.Job:  map 100% reduce 100%
19/01/23 12:35: