旁边的小伙伴昨天加班到今天的凌晨三点,睡在了公司,早晨我到工位时,和我说改了一晚上的上传接收逻辑(也是前一位离开的小伙伴交接的,属于顺位坑主),终于线上问题是修复了,但是只要一上传,整个项目前台页面就会响应不过来,一片白,拜托我帮他找一找问题,修复下,他已经连续通宵加班两天了,说完就骑着小电驴拖着疲惫的身躯回家了。
上传答题卡模块对于一个教育行业的公司来讲,还是一个重要核心模块,之前我还确实没有看过这部分的代码,我看了下核心的输出都没有,关键逻辑点耗时打印也没有,我用了二十分钟的时间,加了关键日志,重新上传分析日志如下:
一共上传了这些学生的答题卡:
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9603726, timeCosted(ms): 3972
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5889397, timeCosted(ms): 4081
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9012679, timeCosted(ms): 8555
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8204132, timeCosted(ms): 9263
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9022886, timeCosted(ms): 9450
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5419477, timeCosted(ms): 10515
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3104156, timeCosted(ms): 25085
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3323977, timeCosted(ms): 25718
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6036999, timeCosted(ms): 33981
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6801082, timeCosted(ms): 21287
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4052504, timeCosted(ms): 18791
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8567062, timeCosted(ms): 14539
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2746989, timeCosted(ms): 17491
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7809515, timeCosted(ms): 19174
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5356828, timeCosted(ms): 16425
可见耗时区间为 1秒到34秒,这个是非常慢的,但是究竟哪里慢,可以拿单独一个学生的处理逻辑进行细化跟踪:
examSubjectID:8620804909191169,xuexinID:6036999,find xsl_examrecords timeCosted(ms):19581
examSubjectID:8620804909191169,xuexinID:6036999 , 删除record记录条数:1,timeCosted(ms):2909
examSubjectID:8620804909191169,xuexinID:6036999 , 删除detail记录条数:6,timeCosted(ms):5645
examSubjectID:8620804909191169,xuexinID:6036999 , 手写识别组、提卡数据置为无效 , timeCosted(ms):41
examSubjectID:8620804909191169,xuexinID:6036999 ,operXslRPaperQuestion ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate save ,timeCosted(ms):55
examSubjectID:8620804909191169,xuexinID:6036999 ,delete old records ,timeCosted(ms):22
examSubjectID:8620804909191169,xuexinID:6036999 ,addBatchXslMcbigimage ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:6036999 ,开始上传小题数据 ,groupID = 8624130641863680 , timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:1,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):23
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:6,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:3,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:5,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):94
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:2,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999,*主观题题号:4,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:6036999 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:6036999 ,mongoTemplate.save ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:6036999 , save ks_sx_multidigit ,timeCosted(ms):3
examSubjectID:8620804909191169,xuexinID:6036999 , save ks_sx_group ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:6036999 , commit ! ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:6036999 , call hdr Service ,result: {"success":true,"resultList":null} ,timeCosted(ms):33
examSubjectID:8620804909191169,xuexinID:6036999 , updateKsGroup ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:6036999 , mongo update xsl_examrecords ,timeCosted(ms):1590
examSubjectID:8620804909191169,xuexinID:6036999 , mongo update xsl_examrecordsdetail ,timeCosted(ms):3795
这下就清楚多了,大部分的操作都是单位毫秒内,还有少数的几处逻辑直接到秒级别,甚至近20秒,查看相关的代码,发现有一个规律,就是这些代码都是操作monogo数据库的地方,很自然的就想到了是不是对应的collection没有添加索引。
- 查看索引:db.collectionName.getIndexes();
- 添加索引
db.xsl_examrecords.createIndex({"examSubjectID": 1, "xuexinID": 1});
db.xsl_examrecords.createIndex({"recordsID": 1, "isEffective": 1});
db.xsl_examrecordsdetail.createIndex({"examSubjectID": 1, "xuexinID": 1});
db.xsl_examrecordsdetail.createIndex({"recordsID": 1, "isEffective": 1,"examSubjectID": 1});
db.xsl_examrecordsdetail.createIndex({"recordsDetailID": 1, "isEffective": 1});
然后再次上传,测试添加索引后的耗时,仍然分为两部分,
第一部分:
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2249573, timeCosted(ms): 377
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5889397, timeCosted(ms): 218
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9603726, timeCosted(ms): 245
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1466901, timeCosted(ms): 157
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5864496, timeCosted(ms): 160
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8204132, timeCosted(ms): 277
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9012679, timeCosted(ms): 140
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1444464, timeCosted(ms): 343
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9022886, timeCosted(ms): 210
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6227177, timeCosted(ms): 185
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3319061, timeCosted(ms): 184
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5419477, timeCosted(ms): 240
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4002259, timeCosted(ms): 268
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3104156, timeCosted(ms): 148
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3821815, timeCosted(ms): 172
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4434870, timeCosted(ms): 196
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3323977, timeCosted(ms): 172
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6036999, timeCosted(ms): 143
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9147143, timeCosted(ms): 193
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6801082, timeCosted(ms): 165
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4052504, timeCosted(ms): 136
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9209911, timeCosted(ms): 251
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7341484, timeCosted(ms): 163
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8567062, timeCosted(ms): 153
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2746989, timeCosted(ms): 130
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7981634, timeCosted(ms): 180
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1075012, timeCosted(ms): 177
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7809515, timeCosted(ms): 222
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7721177, timeCosted(ms): 249
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6095837, timeCosted(ms): 152
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5356828, timeCosted(ms): 159
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4993338, timeCosted(ms): 137
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2249573, timeCosted(ms): 153
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6405090, timeCosted(ms): 141
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1466901, timeCosted(ms): 147
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8204132, timeCosted(ms): 143
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9012679, timeCosted(ms): 134
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2479479, timeCosted(ms): 128
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9022886, timeCosted(ms): 163
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6227177, timeCosted(ms): 130
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3319061, timeCosted(ms): 132
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4002259, timeCosted(ms): 163
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3323977, timeCosted(ms): 173
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6036999, timeCosted(ms): 187
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6801082, timeCosted(ms): 173
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4052504, timeCosted(ms): 165
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8567062, timeCosted(ms): 129
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7981634, timeCosted(ms): 149
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7809515, timeCosted(ms): 131
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6095837, timeCosted(ms): 125
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2249573, timeCosted(ms): 150
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9603726, timeCosted(ms): 141
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1466901, timeCosted(ms): 174
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:1444464, timeCosted(ms): 140
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6227177, timeCosted(ms): 166
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:5419477, timeCosted(ms): 130
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:3104156, timeCosted(ms): 125
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6036999, timeCosted(ms): 157
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9147143, timeCosted(ms): 165
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:6801082, timeCosted(ms): 122
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:4052504, timeCosted(ms): 210
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:9209911, timeCosted(ms): 129
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:8567062, timeCosted(ms): 166
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:2746989, timeCosted(ms): 131
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7981634, timeCosted(ms): 141
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7809515, timeCosted(ms): 193
Call Uploadservice- upload end , examSubjectID:8620804909191169, xuexinID:7721177, timeCosted(ms): 146
可以看到每个学生的处理都可以控制在毫秒级别,单独学生的处理细节逻辑耗时如下:
Begin call Uploadservice-uploadAndAnalyse,examSubjectID:8620804909191169,xuexinID:2249573
examSubjectID:8620804909191169,xuexinID:2249573,find mongo xsl_examrecords timeCosted(ms):5
examSubjectID:8620804909191169,xuexinID:2249573 , 删除mongo record记录条数:1,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:2249573 , 删除mongo detail记录条数:6,timeCosted(ms):9
examSubjectID:8620804909191169,xuexinID:2249573 , 手写识别组无效 , timeCosted(ms):11
examSubjectID:8620804909191169,xuexinID:2249573 , 题卡数据置为无效 , timeCosted(ms):4
examSubjectID:8620804909191169,xuexinID:2249573 ,operXslRPaperQuestion ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate save ,timeCosted(ms):32
examSubjectID:8620804909191169,xuexinID:2249573 ,delete old records ,timeCosted(ms):23
examSubjectID:8620804909191169,xuexinID:2249573 ,addBatchXslMcbigimage ,timeCosted(ms):5
examSubjectID:8620804909191169,xuexinID:2249573 ,开始上传小题数据 ,groupID = 8624264647815168 , timeCosted(ms):28
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:3,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):20
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:2,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):3
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:5,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:4,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:6,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:2249573 ,设置客观题作答记录 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573,*主观题题号:1,作答:null,答案:null,不定项:0,isCorrect:2,得分:0.0;
examSubjectID:8620804909191169,xuexinID:2249573 ,设置试题属性 ,timeCosted(ms):1
examSubjectID:8620804909191169,xuexinID:2249573 ,试题知识点关联 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,手写识别图片 ,timeCosted(ms):0
examSubjectID:8620804909191169,xuexinID:2249573 ,mongoTemplate.save ,timeCosted(ms):2
examSubjectID:8620804909191169,xuexinID:2249573 , save ks_sx_multidigit ,timeCosted(ms):8
examSubjectID:8620804909191169,xuexinID:2249573 , save ks_sx_group ,timeCosted(ms):4
examSubjectID:8620804909191169,xuexinID:2249573 , commit ! ,timeCosted(ms):3
examSubjectID:8620804909191169,xuexinID:2249573 , call hdr Service ,result: {"success":true,"resultList":null} ,timeCosted(ms):79
examSubjectID:8620804909191169,xuexinID:2249573 , updateKsGroup ,timeCosted(ms):5
examSubjectID:8620804909191169,xuexinID:2249573 , mongo update xsl_examrecords ,timeCosted(ms):5
examSubjectID:8620804909191169,xuexinID:2249573 , mongo update xsl_examrecordsdetail ,timeCosted(ms):2
可以看到之前非常慢的mongo处理逻辑的地方已经变得非常快。