干掉Config server的两个节点之后会发生什么?_failed to gather storage statistics for slow opera-优快云博客

本文链接：https://blog.youkuaiyun.com/DanChun2014/article/details/130487792

当MongoDB配置服务器(config server)的副本集(configReplSet)中有两个节点被关闭后，由于不满足多数派原则，主节点变为secondary。这导致mongos无法读取configReplSet，影响集群操作。启动其中一个从节点后，主节点恢复，集群恢复正常。整个故障恢复过程大约需要24秒。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

当前MongoDB集群的测试环境的架构有两个分片，每个分片是三个节点的副本集，一个mongos，mongoDB v4.4.20

config server也是三个节点的副本集（configReplSet），直接停掉两个节点之后，关闭步骤的返回信息相似，如下

configReplSet:SECONDARY> use admin
switched to db admin
configReplSet:SECONDARY> db.shutdownServer()
server should be down...
> exit
bye
{"t":{"$date":"2023-05-04T07:11:49.394Z"},"s":"I",  "c":"QUERY",    "id":22791,   "ctx":"js","msg":"Failed to end logical session","attr":{"lsid":{"id":{"$uuid":"f7de35d6-892f-4fdd-a78e-80d1d4932386"}},"error":{"code":9001,"codeName":"SocketException","errmsg":"socket exception [CONNECT_ERROR] server [couldn't connect to server 127.0.0.1:27019, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27019 :: caused by :: Connection refused]"}}}

查看主节点121，可以看到变成secondary

configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:PRIMARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>
configReplSet:SECONDARY>

121节点日志详细信息：

{"t":{"$date":"2023-05-04T15:11:53.400+08:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"1.1.1.122:27019"}}
{"t":{"$date":"2023-05-04T15:11:53.400+08:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"1.1.1.52:27019"}}
{"t":{"$date":"2023-05-04T15:11:53.531+08:00"},"s":"I",  "c":"REPL",     "id":21809,   "ctx":"ReplCoord-2968","msg":"Can't see a majority of the set, relinquishing primary"}
{"t":{"$date":"2023-05-04T15:11:53.531+08:00"},"s":"I",  "c":"REPL",     "id":21475,   "ctx":"ReplCoord-2968","msg":"Stepping down from primary in response to heartbeat"}
{"t":{"$date":"2023-05-04T15:11:53.532+08:00"},"s":"I",  "c":"REPL",     "id":21343,   "ctx":"RstlKillOpThread","msg":"Starting to kill user operations"}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"REPL",     "id":21344,   "ctx":"RstlKillOpThread","msg":"Stopped killing user operations"}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"REPL",     "id":21340,   "ctx":"RstlKillOpThread","msg":"State transition ops metrics","attr":{"metrics":{"lastStateTransition":"stepDown","userOpsKilled":5,"userOpsRunning":7}}}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"COMMAND",  "id":6015318, "ctx":"ReplCoord-2968","msg":"Yielding locks for prepared transactions."}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"COMMAND",  "id":6015319, "ctx":"ReplCoord-2968","msg":"Invalidating sessions for stepdown."}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"W",  "c":"COMMAND",  "id":20525,   "ctx":"conn109","msg":"Failed to gather storage statistics for slow operation","attr":{"opId":25327164,"error":"lock acquire timeout"}}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"REPL",     "id":21358,   "ctx":"ReplCoord-2968","msg":"Replica set state transition","attr":{"newState":"SECONDARY","oldState":"PRIMARY"}}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"W",  "c":"COMMAND",  "id":20525,   "ctx":"conn1732","msg":"Failed to gather storage statistics for slow operation","attr":{"opId":25327173,"error":"lock acquire timeout"}}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn109","msg":"Slow query","attr":{"type":"command","ns":"config.lockpings","command":{"findAndModify":"lockpings","query":{"_id":"db-test03:27020:1682238543:-7230616499206276199"},"update":{"$set":{"ping":{"$date":"2023-05-04T07:11:43.295Z"}}},"upsert":true,"writeConcern":{"w":"majority","wtimeout":15000},"maxTimeMS":30000,"$replData":1,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1683184301,"i":1}},"signature":{"hash":{"$binary":{"base64":"hk1EhaG1QhD+LJYlqa8rN8kD03I=","subType":"0"}},"keyId":7225138377836724247}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1683184276,"i":1}},"t":1}},"$db":"config"},"planSummary":"IDHACK","keysExamined":1,"docsExamined":1,"nMatched":1,"nModified":1,"keysInserted":1,"keysDeleted":1,"numYields":0,"reslen":944,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"writeConcern":{"w":"majority","wtimeout":15000,"provenance":"clientSupplied"},"protocol":"op_msg","durationMillis":10250}}
{"t":{"$date":"2023-05-04T15:11:53.534+08:00"},"s":"W",  "c":"COMMAND",  "id":20525,   "ctx":"conn107","msg":"Failed to gather storage statistics for slow operation","attr":{"opId":25327201,"error":"lock acquire timeout"}}
{"t":{"$date":"2023-05-04T15:11:53.534+08:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn107","msg":"Slow query","attr":{"type":"command","ns":"config.lockpings","command":{"findAndModify":"lockpings","query":{"_id":"db-test02:27020:1682238543:1003676273669917507"},"update":{"$set":{"ping":{"$date":"2023-05-04T07:11:44.867Z"}}},"upsert":true,"writeConcern":{"w":"majority","wtimeout":15000},"maxTimeMS":30000,"$replData":1,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1683184302,"i":1}},"signature":{"hash":{"$binary":{"base64":"pBveU7F8DsrnP+ci1SC8SZjcbog=","subType":"0"}},"keyId":7225138377836724247}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1683184302,"i":1}},"t":1}},"$db":"config"},"planSummary":"IDHACK","keysExamined":1,"docsExamined":1,"nMatched":1,"nModified":1,"keysInserted":1,"keysDeleted":1,"numYields":0,"reslen":943,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"writeConcern":{"w":"majority","wtimeout":15000,"provenance":"clientSupplied"},"protocol":"op_msg","durationMillis":8665}}
{"t":{"$date":"2023-05-04T15:11:53.534+08:00"},"s":"W",  "c":"COMMAND",  "id":20525,   "ctx":"conn1866","msg":"Failed to gather storage statistics for slow operation","attr":{"opId":25327224,"error":"lock acquire timeout"}}
{"t":{"$date":"2023-05-04T15:11:53.533+08:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn1732","msg":"Slow query","attr":{"type":"command","ns":"config.lockpings","command":{"findAndModify":"lockpings","query":{"_id":"db-test02:27018:1682237385:-4185141032096930406"},"update":{"$set":{"ping":{"$date":"2023-05-04T07:11:43.587Z"}}},"upsert":true,"writeConcern":{"w":"majority","wtimeout":15000},"maxTimeMS":30000,"$replData":1,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1683184297,"i":1}},"signature":{"hash":{"$binary":{"base64":"4iK6ahwdeKghfd+N3lscP/Zbil8=","subType":"0"}},"keyId":7225138377836724247}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1683184297,"i":1}},"t":1}},"$db":"config"},"planSummary":"IDHACK","keysExamined":1,"docsExamined":1,"nMatched":1,"nModified":1,"keysInserted":1,"keysDeleted":1,"numYields":0,"reslen":944,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"writeConcern":{"w":"majority","wtimeout":15000,"provenance":"clientSupplied"},"protocol":"op_msg","durationMillis":9945}}
{"t":{"$date":"2023-05-04T15:11:53.534+08:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn1866","msg":"Slow query","attr":{"type":"command","ns":"config.$cmd","command":{"update":"mongos","bypassDocumentValidation":false,"ordered":true,"updates":[{"q":{"_id":"db-test01:27017"},"u":{"$set":{"_id":"db-test01:27017","ping":{"$date":"2023-05-04T07:11:46.249Z"},"up":949382,"waiting":true,"mongoVersion":"4.4.20","advisoryHostFQDNs":[]}},"multi":false,"upsert":true}],"writeConcern":{"w":"majority","wtimeout":60000},"maxTimeMS":30000,"$replData":1,"$clusterTime":{"clusterTime":{"$timestamp":{"t":168318