MapReduce实战之WordCount

打开eclipse，新建一个WordCount的java project工程，写WordMapper类继承于Mapper抽象类，覆写map函数，写WordReducer类继承于Reducer，覆写reduce函数,最后写一个场景调用类，调用WordMapper和Reducer类

WordMapper类

WordReduce类

WordMain类

接下来就是导出jar包文件的步骤

Linux桌面上就会出现wordcount.jar文件

准备好要测试的文件file1.txt,file2.txt，两个文件里面的内容是一些单词

启动hadoop集群，命令start-all.sh,创建文件输入路径:hadoop fs -mkdir /user/gznc/input将本地上的file1.txt和file2.txt文件上传到集群的输入文件中，有两种方法可以上传文件，第一种方法是命令：hdfs dfs /home/gznc/file1.txt /user/gznc/input,第一个是本地路径，第二个是集群路径。第二个文件类似。第二种方法是用eclipse写方法上传。注意路径可以变化，不一定要和我的一样

查看文件file1.txt和file2.txt是否已经上传到集群，命令:hadoop fs -ls /user/gznc/input

最后一步，运算，输入命令：hadoop jar /home/gznc/Desktop/wordcount.jar wordcount.WordMain /user/gznc/input /user/gznc/output 格式：hadoop+jar+jar包的路径，因为我是放在本地的桌面上+工程下的包名.main函数所在的那个类+集群上所存放文件的路径+结果输出的路径

成功后的结果：
16/10/19 16:30:11 INFO input.FileInputFormat: Total input paths to process : 2
16/10/19 16:30:11 INFO mapreduce.JobSubmitter: number of splits:2
16/10/19 16:30:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1476858901736_0002
16/10/19 16:30:13 INFO impl.YarnClientImpl: Submitted application application_1476858901736_0002
16/10/19 16:30:13 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1476858901736_0002/
16/10/19 16:30:13 INFO mapreduce.Job: Running job: job_1476858901736_0002
16/10/19 16:30:30 INFO mapreduce.Job: Job job_1476858901736_0002 running in uber mode : false
16/10/19 16:30:30 INFO mapreduce.Job: map 0% reduce 0%
16/10/19 16:30:53 INFO mapreduce.Job: map 100% reduce 0%
16/10/19 16:31:15 INFO mapreduce.Job: map 100% reduce 100%
16/10/19 16:31:16 INFO mapreduce.Job: Job job_1476858901736_0002 completed successfully
16/10/19 16:31:16 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=126
FILE: Number of bytes written=290714
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=304
HDFS: Number of bytes written=40
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=43849
Total time spent by all reduces in occupied slots (ms)=18909
Total time spent by all map tasks (ms)=43849
Total time spent by all reduce tasks (ms)=18909
Total vcore-seconds taken by all map tasks=43849
Total vcore-seconds taken by all reduce tasks=18909
Total megabyte-seconds taken by all map tasks=44901376
Total megabyte-seconds taken by all reduce tasks=19362816
Map-Reduce Framework
Map input records=4
Map output records=16
Map output bytes=150
Map output materialized bytes=132
Input split bytes=218
Combine input records=16
Combine output records=10
Reduce input groups=5
Reduce shuffle bytes=132
Reduce input records=10
Reduce output records=5
Spilled Records=20
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=480
CPU time spent (ms)=3950
Physical memory (bytes) snapshot=510222336
Virtual memory (bytes) snapshot=2516795392
Total committed heap usage (bytes)=256647168
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=86
File Output Format Counters
Bytes Written=40

如果结果路径如：/user/gznc/output在集群上已经存在了，就会报如下错误，所以要保证结果输出的路径不一样

Exception in thread “main” org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://master:9000/user/gznc/output already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at wordcount.WordMain.main(WordMain.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

现在我们去查看集群所统计单词频数的结果，命令：hadoop fs -ls /user/gznc/output