项目中使用mongo数据库的错误

最新推荐文章于 2024-11-28 09:52:43 发布

转载最新推荐文章于 2024-11-28 09:52:43 发布 · 788 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/sy-liu/p/6632732.html

文章标签：

#json #数据库 #数据结构与算法

在项目中遇到MongoQueryException，提示查询数据超出内部限制。解决步骤包括检查查询字段是否建立索引，如果未建立，可通过db.ensureIndex()创建。建立索引可以提升查询效率，但过多的索引会影响插入速度。对于大型数据导入，若文件过大，可尝试使用GridFS，以避免超过最大BSON文档大小的限制。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Caused by: com.mongodb.MongoQueryException: Query failed with error code 17144 and error message 'Runner error: Overflow sort stage buffered data usage of 33554586 bytes exceeds internal limit

1.查看查询条件使用什么字段查询的如query.order("number")

2.查看数据库有没有为该字段建立索引 db.<collection>.getIndexes()

3. 如果没有建立索引db.ensureIndex({number:-1(或1)})

MongoDB索引使用B-tree数据结构。

建立索引的目的，提高查询效率会在query之前数据库按索引range。

弊端：索引过多会使得插入效率过低（因为每次插入（索引都会重新构造B-tree）

mongoimport -h 192.168.1.10 -d xinmeiti -c weiboAnalyse --directoryperdb weiboAnalyse.json --drop

exception:BSON representation of supplied JSON is too large: code FailedToParse: FailedToParse: Trailing number at end of input: offset:1547 of:{ "_id" : { "$oid" : "57d0c3e6fa9ae90dc6839aec" }, "updateTime" : 1514567702552, "rank_append" : { "17-10-19-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-04-01" : { "rank" : 39533, "factor" : 0.0731883945878533 }, "17-11-15-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-10-11-01" : { "rank" : 39226, "factor" : 0.0731883945878533 }, "17-12-14-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-21-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-12-20-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-11-08-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-12-04-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-30-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-10-29-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-30-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-26-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-14-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-07-01" : { "rank" : 40078, "factor" : 0.07329512825652773 }, "17-11-18-01" : { "rank" : 40189, "factor" : 0.07329512825652773 }, "17-11-06-01" : { "rank" : 39865, "factor" : 0.0731883945878533 }, "17-12-12-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-23-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-13-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-22-01" : { "rank" : 40555, "factor" : 0.07360409811643538 }, "17-11-11-01" : { "rank" : 40

原因：要导入的文件过大

结局方案：mongoimport -h 192.168.1.10 -d xinmeiti -c weiboAnalyse --directoryperdb weiboAnalyse.json --jsonArray --drop

但是 mongoimport -h 192.168.1.10 -d xinmeiti -c weiboAnalyse --directoryperdb weiboAnalyse.json --jsonArray --drop 又出现了

Invalid JSON passed to mongoimport: code FailedToParse: FailedToParse: Trailing number at end of input: offset:1547 of:{ "_id" : { "$oid" : "57d0c3e6fa9ae90dc6839aec" }, "updateTime" : 1514567702552, "rank_append" : { "17-10-19-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-04-01" : { "rank" : 39533, "factor" : 0.0731883945878533 }, "17-11-15-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-10-11-01" : { "rank" : 39226, "factor" : 0.0731883945878533 }, "17-12-14-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-21-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-12-20-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-11-08-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-12-04-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-30-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-10-29-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-30-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-26-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-14-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-07-01" : { "rank" : 40078, "factor" : 0.07329512825652773 }, "17-11-18-01" : { "rank" : 40189, "factor" : 0.07329512825652773 }, "17-11-06-01" : { "rank" : 39865, "factor" : 0.0731883945878533 }, "17-12-12-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-23-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-13-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-22-01" : { "rank" : 40555, "factor" : 0.07360409811643538 }, "17-11-11-01" : { "rank" : 40, current buffer:

{ "_id" : { "$oid" : "57d0c3e6fa9ae90dc6839aec" }, "updateTime" : 1514567702552, "rank_append" : { "17-10-19-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-04-01" : { "rank" : 39533, "factor" : 0.0731883945878533 }, "17-11-15-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-10-11-01" : { "rank" : 39226, "factor" : 0.0731883945878533 }, "17-12-14-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-21-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-12-20-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-11-08-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-12-04-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-30-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-10-29-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-30-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-26-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-14-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-07-01" : { "rank" : 40078, "factor" : 0.07329512825652773 }, "17-11-18-01" : { "rank" : 40189, "factor" : 0.07329512825652773 }, "17-11-06-01" : { "rank" : 39865, "factor" : 0.0731883945878533 }, "17-12-12-01" : { "rank" : 40795, "factor" : 0.07360409811643538 }, "17-10-23-01" : { "rank" : 39494, "factor" : 0.0731883945878533 }, "17-11-13-01" : { "rank" : 40115, "factor" : 0.07329512825652773 }, "17-11-22-01" : { "rank" : 40555, "factor" : 0.07360409811643538 }, "17-11-11-01" : { "rank" : 40

原因：最大BSON文档大小是16兆字节。最大文档大小有助于确保单个文档不能使用过量的RAM，或者在传输过程中使用过多的带宽。为了存储大于最大大小的文档，MongoDB提供了GridFS API。有关GridFS的更多信息，请参阅mongofiles和您的驱动程序的文档。

转载于:https://www.cnblogs.com/sy-liu/p/6632732.html