mapreduce实例1-topN

最新推荐文章于 2024-07-21 02:58:52 发布

UncleDu

最新推荐文章于 2024-07-21 02:58:52 发布

阅读量488

点赞数 1

CC 4.0 BY-SA版权

分类专栏： mapreduce

本文链接：https://blog.youkuaiyun.com/qq_42174021/article/details/86507568

该博客介绍了如何使用MapReduce解决TopN问题，具体为计算request.dat文件中每个URL的访问次数，并找出每个网站访问次数最多的前3个URL。Map阶段实现URL计数，Reduce阶段通过TreeMap进行TopN排序并在cleanup方法中输出结果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

读取文件request.dat

2017/07/28 qq.com/a
2017/07/28 qq.com/bx
2017/07/28 qq.com/by
2017/07/28 qq.com/by3
2017/07/28 qq.com/news
2017/07/28 sina.com/news/socail
2017/07/28 163.com/ac
2017/07/28 sina.com/news/socail
2017/07/28 163.com/sport

需求一：求出每个url被访问的总次数，并将结果输出到一个结果文件中
public static class RequestMapper extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);

	public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
		String line = value.toString();
		String[] info = line.split(" ");
		String url = info[1];
		context.write(new Text(url), one);
		}
	}


public static class RequestReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
	private IntWritable result = new IntWritable();

	public void reduce(Text key, Iterable<IntWritable> values, Context context)
			throws IOException, InterruptedException