Hadoop之MR简单例子（手机号分组和统计）

最新推荐文章于 2022-04-19 17:06:05 发布

原创最新推荐文章于 2022-04-19 17:06:05 发布 · 476 阅读

0 ·

CC 4.0 BY-SA版权

本文介绍了一个自定义分组规则的MapReduce程序示例，该程序按手机号的前三位进行分组并统计各分组中号码的出现频率。具体实现包括自定义分区器和Mapper、Reducer组件。

1，自定义分组规则

按手机号的前三位划分，相同则为一组

public class AreaPartitioner<KEY, VALUE> extends Partitioner<KEY, VALUE> {

    private static Map<String, Integer> cacheValues = new HashMap<String, Integer>();

    static {
        cacheValues.put("153", 0);
        cacheValues.put("177", 1);
        cacheValues.put("147", 2);
    }

    @Override
    public int getPartition(KEY key, VALUE value, int numPartitions) {
        Integer phoneNo = cacheValues.get(key.toString().substring(0, 3));
        return phoneNo == null ? 3 : phoneNo;
    }

}

2，Map/Reduce/Main

public class PhoneCount {

    public static class PhoneCountMapper extends Mapper<Object, Text, Text, IntWritable> {

        private static final IntWritable ONE = new IntWritable(1);
        private Text phoneNo = new Text();

        @Override
        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                this.phoneNo.set(itr.nextToken());
                context.write(this.phoneNo, ONE);
            }
        }

    }

    public static class PhoneCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

        private IntWritable result = new IntWritable();

        @Override
        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            IntWritable val;
            for (Iterator i = values.iterator(); i.hasNext(); sum += val.get()) {
                val = (IntWritable) i.next();
            }
            this.result.set(sum);
            context.write(key, this.result);
        }

    }

    public static void main(String[] args) throws Exception {
        Configuration configuration = new Configuration();

        Job job = Job.getInstance(configuration, "PhoneCount");
        job.setJarByClass(PhoneCount.class);

        // 设置Map,Reduce类和分组排序规则
        job.setMapperClass(PhoneCountMapper.class);
        job.setPartitionerClass(AreaPartitioner.class);
        job.setReducerClass(PhoneCountReducer.class);

        // 设置reduce映射数量,应该和分组数量一致
        job.setNumReduceTasks(4);

        // 设置reduce输出类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        //设置map输出类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);


        String[] otherArgs = new String[]{"input/phone.txt", "output2"};
        Path path = new Path(otherArgs[1]);
        FileSystem fileSystem = path.getFileSystem(configuration);
        if (fileSystem.exists(path)) {
            fileSystem.delete(path, true);
        }

        // 设置输入/输出数据存放位置
        FileInputFormat.setInputPaths(job, new Path(otherArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

}

按分组规则，将会分为4组，"153......"，"177......"，"147......"以及其他，output 会生成了4个包含结果的文件，分别存储了4组手机号，并统计每个号码出现的次数