代码：MapReduce程序模板

最新推荐文章于 2024-05-14 21:30:47 发布

一只行走在闹市的猪

最新推荐文章于 2024-05-14 21:30:47 发布

阅读量280

点赞数 1

CC 4.0 BY-SA版权

分类专栏：大数据文章标签： hadoop mapreduce程序模板

本文链接：https://blog.youkuaiyun.com/qq_38314823/article/details/80777696

大数据专栏收录该内容

17 篇文章

订阅专栏

本文详细介绍了MapReduce编程模型的基本实现方式，包括MRDriver、Mapper和Reducer三个核心组件的具体配置和功能实现。从创建Job开始，到输入输出的设置，再到Mapper与Reducer的功能处理，全面覆盖了MapReduce任务的各个阶段。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

MapReduce：

public class MRDriver extends Configured implements Tool {

    @Override
    public int run(String[] args) throws Exception {
        //1.创建job
        Job job = Job.getInstance(this.getConf(), "xxx");
        job.setJarByClass( MRDriver.class );

        //2.input
        Path inputPath = new Path(args[0]);
        FileInputFormat.setInputPaths(job,inputPath);

        //3.mapper
        job.setMapperClass(null);
        job.setMapOutputKeyClass(null);
        job.setMapOutputValueClass(null);

        //4.shuffle
        job.setPartitionerClass(null);//分区
        job.setGroupingComparatorClass(null);//分组
        job.setSortComparatorClass(null);//排序

        //5.reducer
        job.setReducerClass(null);
        job.setOutputKeyClass(null);
        job.setOutputValueClass(null);

        //6.output
        Path outputPath = new Path(args[1]);
        FileSystem hdfs = FileSystem.get(this.getConf());
        //查看输出路径是否存在，存在则先删除
        if(hdfs.exists(outputPath)){
            //boolean delete(Path f, boolean recursive)
            //如果是目录则为true
            hdfs.delete(outputPath,true);
        }
        FileOutputFormat.setOutputPath(job,outputPath);

        boolean isSuccess = job.waitForCompletion(true);
        return isSuccess?0:1;
    }

    public static void main(String[] args) {
        Configuration configuration = new Configuration();
        try {
            int status = ToolRunner.run(configuration, new MRDriver(), args);
            System.exit(status);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Mapper：

public class MRMapper extends Mapper<LongWritable,Text,Text,Text>{
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //根据业务处理
    }
}

Reducer：

public class MRReducer extends Reducer<Text,Text,Text,Text> {
    @Override
    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        //根据业务处理
    }
}