需求:1、统计每个页面被访问的总次数 2、被访问最多的5个页面
分析:和MapReduce的WordCount一样,以页面地址的字符串作为key,1作为value,相同的统计相加
每统计一组数据后会调用reduce方法,当所有数据统计完以后调完reduce方法后会再调一次cleanup()方法
reduce方法处理完以后会得到每个页面的总次数,得到总次数后先别交给reducetask,自己先缓存起来(在类上写一个成员变量,每处理完一组数据得到页面的总次数后放到hashmap,countMap.put(),放完以后会调cleanup,所以需要在cleanup里面排个序,输出次数最高的几个),但是这样reduce就只有一个,就和分布式关系不大了。
分布式去做:分成多个reduce,再写个程序再聚合,不一步到位。
这两个选取,如果数据不多,就一步搞定;数据量很大,就先局部聚合,然后再读聚合之后的前五个,取出。
非分布式:
public static void main(String[] args) {
TreeMap<FlowBean, String> tm1 = new TreeMap<>(new Comparator<FlowBean>() {
@Override
public int compare(FlowBean o1, FlowBean o2) {
if( o2.getAmountFlow()-o1.getAmountFlow()==0){
return o1.getPhone().compareTo(o2.getPhone());
}
return o2.getAmountFlow()-o1.getAmountFlow();
}
});
FlowBean b1 = new FlowBean("1367788", 500, 300);
FlowBean b2 = new FlowBean("1367766", 400, 200);
FlowBean b3 = new FlowBean("1367755", 600, 400);
FlowBean b4 = new FlowBean("1367744", 300, 500);
tm1.put(b1, null);
tm1.put(b2, null);
tm1.put(b3, null);
tm1.put(b4, null);
Set<Entry<FlowBean,String>> entrySet = tm1.entrySet();
for (Entry<FlowBean,String> entry : entrySet) {
System.out.println(entry.getKey() +"\t"+ entry.getValue());
}
}
分布式:
流程梳理:
实体bean mapper reducer jobsubmitter 直接附代码:
public class PageCount implements Comparable<PageCount>{
private String page;
private int count;
public void set(String page, int count) {
this.page = page;
this.count = count;
}
public String getPage() {
return page;
}
public void setPage(String page) {
this.page = page;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
@Override
public int compareTo(PageCount o) {
return o.getCount()-this.count==0?this.page.compareTo(o.getPage()):o.getCount()-this.count;
}
}
public class PageTopnMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
@Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] split = line.split(" ");
context.write(new Text(split[1]), new IntWritable(1));
}
}
public class PageTopnReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
TreeMap<PageCount, Object> treeMap = new TreeMap<>();
@Override
protected void reduce(Text key, Iterable<IntWritable> values,
Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
int count = 0;
for (IntWritable value : values) {
count += value.get();
}
PageCount pageCount = new PageCount();
pageCount.set(key.toString(), count);
treeMap.put(pageCount,null);
}
@Override
protected void cleanup(Context context)
throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
int topn = conf.getInt("top.n", 5);
Set<Entry<PageCount, Object>> entrySet = treeMap.entrySet();
int i= 0;
for (Entry<PageCount, Object> entry : entrySet) {
context.write(new Text(entry.getKey().getPage()), new IntWritable(entry.getKey().getCount()));
i++;
if(i==topn) return;
}
}
}
public class JobSubmitter {
public static void main(String[] args) throws Exception {
/**
* 通过加载classpath下的*-site.xml文件解析参数
*/
Configuration conf = new Configuration();
conf.addResource("xx-oo.xml");
/**
* 通过代码设置参数
*/
//conf.setInt("top.n", 3);
//conf.setInt("top.n", Integer.parseInt(args[0]));
/**
* 通过属性配置文件获取参数
*/
/*Properties props = new Properties();
props.load(JobSubmitter.class.getClassLoader().getResourceAsStream("topn.properties"));
conf.setInt("top.n", Integer.parseInt(props.getProperty("top.n")));*/
Job job = Job.getInstance(conf);
job.setJarByClass(JobSubmitter.class);
job.setMapperClass(PageTopnMapper.class);
job.setReducerClass(PageTopnReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(job, new Path("F:\\mrdata\\url\\input"));
FileOutputFormat.setOutputPath(job, new Path("F:\\mrdata\\url\\output"));
job.waitForCompletion(true);
}
}
keyMap是按key的字典顺序来排序的