mapreduce编程:wordcount

最新推荐文章于 2021-10-27 16:27:54 发布

原创最新推荐文章于 2021-10-27 16:27:54 发布 · 379 阅读

0 ·

CC 4.0 BY-SA版权

MapReduce编程（maven）专栏收录该内容

4 篇文章

订阅专栏

本文详细介绍了如何使用Maven插件编写并打包Hadoop Wordcount程序，包括下载相关依赖、实现MapReduce操作、以及将程序打包为jar文件。同时，提供了如何运行程序和查看结果的步骤。

用maven插件编写程序，下载相关依赖：

package my.hadoopstudy;  
   
import org.apache.hadoop.conf.Configuration;  
import org.apache.hadoop.fs.Path;  
import org.apache.hadoop.io.IntWritable;  
import org.apache.hadoop.io.Text;  
import org.apache.hadoop.mapreduce.Job;  
import org.apache.hadoop.mapreduce.Mapper;  
import org.apache.hadoop.mapreduce.Reducer;  
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
import org.apache.hadoop.util.GenericOptionsParser;  
   
import java.io.IOException;  
   
public class Wordcount {  
   
    public static class MyMapper extends Mapper<Object, Text, Text, IntWritable>{  
        private final static IntWritable one = new IntWritable(1);  
   
        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {   
                String[] e = value.toString().split(" ");  
                for(int i=0;i<e.length;i++){
                   context.write(new Text(e[i]), one);
                }
        }  
    }  
   
    public static class MyReducer extends Reducer<Text,IntWritable,Text,IntWritable> {  
        private IntWritable result = new IntWritable();  
   
        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {  
            int sum = 0;  
            for (IntWritable val : values) {  
                sum += val.get();  
            }  
            result.set(sum);  
            context.write(key, result);  
        }  
    }  
   
    public static void main(String[] args) throws Exception {  
        Configuration conf = new Configuration();  
        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();  
        if (otherArgs.length < 2) {  
            System.err.println("Usage: EventCount <in> <out>");  
            System.exit(2);  
        }  
        Job job = Job.getInstance(conf, "word count");  
        job.setJarByClass(Wordcount.class);  
        job.setMapperClass(MyMapper.class);  
        job.setCombinerClass(MyReducer.class);  
        job.setReducerClass(MyReducer.class);  
        job.setOutputKeyClass(Text.class);  
        job.setOutputValueClass(IntWritable.class);  
        FileInputFormat.addInputPath(job, new Path(otherArgs[0]));  
        FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));  
        System.exit(job.waitForCompletion(true) ? 0 : 1);  
    }  
}

将程序用打包成jar文件

统计文件：