MongoDB JAVA-API执行mapreduce的几种方法

最新推荐文章于 2022-11-05 10:49:02 发布

转载最新推荐文章于 2022-11-05 10:49:02 发布 · 228 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://my.oschina.net/huzorro/blog/75210

文章标签：

#大数据 #java #数据库

本文介绍了如何在MongoDB中使用MapReduce进行数据处理。包括Shell命令和Java API的使用方式，展示了通过MapReduce实现复杂数据聚合的具体示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2019独角兽企业重金招聘Python工程师标准>>>

> db.version();
1.6.3

mongodb-2.0.jar

<!--
#shell下完整的参数列表
#更高版本会多几个参数
//-->
db.runCommand(  
{  
    mapreduce : <collection>,  
    map : <mapfunction>,  
    reduce : <reducefunction>  
    [, query : <query filter object>]  
    [, sort : <sort the query.  useful   optimization>] for  
    [, limit : <number of objects to   from collection>] return  
    [, out : <output-collection name>]  
    [, keeptemp: < | >] true false  
    [, finalize : <finalizefunction>]  
    [, scope : <object where fields go into javascript global scope >]  
    [, verbose :  ] true  
});

//JAVA-API中可以执行mapreduce的方法
//该方法只允许携带4个参数
public MapReduceOutput mapReduce(String map, String reduce, 
                                String outputCollection, 
                                DBObject query) 
                                throws MongoException
//以下两个方法可以使用全部参数列表
public MapReduceOutput mapReduce(DBObject command) throws MongoException
public CommandResult command(DBObject cmd) throws MongoException

<!--
#shell mapreduce
//-->
res = db.runCommand({
	mapreduce:'mo_log_201208',
	query:{Logtime:{$gte:'20120823', $lte:'20120824'}},
	map:function() {
		emit({Spcode:this.Spcode, Spname:this.Spname, 
                Consignid:this.Consignid, Consname:this.Consname, 
                Region:this.Region, Regionname:this.Regionname, 
                Serviceid:this.Serviceid,    
                Servicename:this.Servicename, 
                Srctermid:this.Srctermid}, {count:1});
	},
	reduce:function(key, value) {
		var ret = {count:0};
		ret.count++;
		return ret;
	},
	out:'tmp_mo_spcode_consignid_region_serviceid_201208_1'
})

//JAVA-API mapreduce 
	Mongo db = new Mongo("host", "port");
	DBCollection dbc = db.getCollection("mo_log_201208");
	String mapfun = "function() {" +
		"emit({Spcode:this.Spcode, Spname:this.Spname, " +
                "Consignid:this.Consignid, Consname:this.Consname, " +
                "Region:this.Region, Regionname:this.Regionname, " +
                "Serviceid:this.Serviceid, " +   
                "Servicename:this.Servicename, " +
                "Srctermid:this.Srctermid}, {count:1});" +
				"}";
	String reducefun = "function(key, value) {" +
		"var ret = {count:0};" +
		"ret.count++;" +
		"return ret;" +
		"}";
	DBObject query = new BasicDBObject("Logtime", 
						new BasicDBObject("$gte", "20120823").append("$lte", "20120824"));
	MapReduceOutput mro = dbc.mapReduce(mapfun, reducefun, 
							"tmp_mo_spcode_consignid_region_serviceid_201208_1", query);

<!--
#该mapreduce比上一个多了两个参数
//-->
res = db.runCommand({
	mapreduce:'mo_log_201208',
	query:{Logtime:{$gte:'20120823', $lte:'20120824'}},
	map:function() {
		emit({Spcode:this.Spcode, Spname:this.Spname, 
        Consignid:this.Consignid, Consname:this.Consname, 
        Region:this.Region, Regionname:this.Regionname, 
        Serviceid:this.Serviceid, Servicename:this.Servicename, 
        Srctermid:this.Srctermid}, {count:1});
	},
	reduce:function(key, value) {
		var ret = {count:0};
		ret.count++;
		return ret;
	},
	finalize:function(key, value){
		db.tmp_mo_spcode_consignid_region_serviceid_201208.insert({"_id":key, "value":value});
		return value;
	},
	out:'tmp_mo_spcode_consignid_region_serviceid_201208_1',
	verbose:true
})

//JAVA-API mapreduce
	Mongo db = new Mongo("host", "port");
	DBCollection dbc = db.getCollection("mo_log_201208");
	String mapfun = "function() {" +
		"emit({Spcode:this.Spcode, Spname:this.Spname, " +
                "Consignid:this.Consignid, Consname:this.Consname, " +
                "Region:this.Region, Regionname:this.Regionname, " +
                "Serviceid:this.Serviceid, " +   
                "Servicename:this.Servicename, " +
                "Srctermid:this.Srctermid}, {count:1});" +
				"}";
	String reducefun = "function(key, value) {" +
		"var ret = {count:0};" +
		"ret.count++;" +
		"return ret;" +
		"}";
	String finalizefun = "function(key, value){" +
		"db.tmp_mo_spcode_consignid_region_serviceid_201208.insert({"_id":key, "value":value});" +
		"return value;" +
		"}";
	DBObject query = new BasicDBObject("Logtime", 
						new BasicDBObject("$gte", "20120823").append("$lte", "20120824"));
	DBObject command = new BasicDBObject();
	command.put("mapreduce", "mo_log_201208");
	command.put("query", query);
	command.put("map", mapfun);
	command.put("reduce", reducefun);
	command.put("finalize", finalizefun);
	command.put("out", "tmp_mo_spcode_consignid_region_serviceid_201208_1");
	command.put("verbose", true);
	//在dbcollection上执行mapreduce
	MapReduceOutput mro = dbc.mapReduce(command);
	//在db上使用command执行mapreduce
	CommandResult cr = db.command(command);

>>以上3种方法都可以执行mapreduce, 参数和返回结果各不相同, 根据自己的需要和使用习惯选择合适的方法
>>新版的Mongodb增加了新的聚合框架, 有更多的参数选择, 参见Aggregation Framework

转载于:https://my.oschina.net/huzorro/blog/75210