本无意写这篇文章,但是之前在分析replset时有一个线程一直没弄明白其作用,后来偶然间在阅读tags时搞明白了notifierthread的作用,tags的实现过程很隐晦.不仔细阅读,很难弄明白,所以这里专门写一篇文章来分析replset tags的实现.首先来看看一份带tags的replset config.
这里的dc可以理解为datacenter或者任何自己觉得好理解的词,dc后的ny sf cloud可以理解为不同的服务器,settings下面的veryImportant中dc:3,表示在调用getLastError调用如:
db.runCommand({getLastError:1,w:"veryImportant"})
这个调用表示getLastError命令要等带上一个操作完成veryImportant指定的将上一个操作更改到3个不同的datacenter中才会返回,这里对应ny sf cloud.那么这里是怎么实现的呢,这就是这篇文章探讨的内容.下面先介绍下本文将会遇到的一些内容.
collection: local.me: 这个collection只保留一个OID,这个OID类型和mongodb中每一条数据中的_id是一样的,是自动生成的.生成了后将其保存到local.me中,以后就不需要再次生成了,读取就行了.这个OID就是来标识这台服务器的,在secondary同步另外一台服务器数据时notifierThread将作为握手协议将这个值发送给对方,用来标识自己.
collection: local.slaves: 用来保存同步自己的服务器当前同步到的操作日志时间戳.
thread: notifierThread: 告诉同步服务器自己当前同步到的操作日志时间戳.
下面简单描述一下tags的执行流程.
1. 在设置replset配置时首先会解析settings中的getLastErrorMode,和members中的tags,将其分组.
2. secondary的producethread选取一个服务器(这里姑且叫做A)同步数据,向其发出连接请求,其连接请求是不需要握手的,当这里produceThread选取了同步对象A后,notifierThread也将去连接A,并且需要握手,就是将自己的表示OID发送给对方,对方拿到自己的OID以及members中的_id将建立一个结构记录和更新这台服务器(姑且叫做B)当天同步到的时间点.
3. 当secondary向服务器请求操作日志并replay后将会更新本地的lastwriten时间,这时notifierThread就将向服务器发送请求,同样是请求数据,但是因为其之前带了握手协议,在服务端处理该请求的连接线程中保存了OID,这里服务端就会认为是notifierThread发过来的消息,表示B服务器已经同步到了notifierThread上一次请求操作日志时最后一条操作日志的时间戳.
4. 这里就可以根据OID以及_id和这个时间戳修改这台服务器同步到的时间戳,然后根据1中该服务器的tags中的分组情况,更新该服务器所在的组中的同步时间戳.
5. 调用getLastError时首先就可以得到本机的上一条操作的时间戳(这里姑且叫做H1),找到getLastError中的w,得到其中的值,根据其值得到replset config中settings.getLastErrorModes中对应的规则,然后处理其中的规则,找出其中的每一条规则(这里叫做C1....Cn)如C1,找到C1规则需要满足的分组中的时间戳(H2),找出所有的规则中的时间戳取其最小值,表示当前规则同步到的时间戳(H3),将H1与H3比较,H1<=H3表示操作已经按照要求完成(举例中已经写到了三个datacenter的服务器上),否则表示规则没满足,则getLastError命令睡眠等待,一段时间醒来检查一下,直到满足要求才返回(getLastError命令可以设置超时).
下面进入代码分析,先来看看replset config的处理,其处理代码为repl\rs_config.cpp的parseRules.
void ReplSetConfig::parseRules(const BSONObj& modes) {
map<string,TagClause> tagMap;
_populateTagMap(tagMap);//将members中的tags归类
for (BSONObj::iterator i = modes.begin(); i.more(); ) {
unsigned int primaryOnly = 0;
// ruleName : {dc : 2, m : 3}
BSONElement rule = i.next();
TagRule* r = new TagRule();
BSONObj clauseObj = rule.Obj();
for (BSONObj::iterator c = clauseObj.begin(); c.more(); ) {
BSONElement clauseElem = c.next();
// get the clause, e.g., "x.y" : 3
const char *criteria = clauseElem.fieldName();//得到每一条规则中的子句
int value = clauseElem.numberInt();
TagClause* node = new TagClause(tagMap[criteria]);
int numGroups = node->subgroups.size();//当前存在的subgroup数要大于规则中指定的数目,要不无法满足要求
uassert(14831, str::stream() << "mode " << clauseObj << " requires "
<< value << " tagged with " << criteria << ", but only "
<< numGroups << " with this tag were found", numGroups >= value);
node->name = criteria;
node->target = value;
// if any subgroups contain "me", we can decrease the target
node->actualTarget = node->target;
// then we want to add pointers between clause & subgroup
for (map<string,TagSubgroup*>::iterator sgs = node->subgroups.begin();
sgs != node->subgroups.end(); sgs++) {
bool foundMe = false;
(*sgs).second->clauses.push_back(node);//记录每一个subgroup上的规则,当subgroup时间戳更新后能够找到相应规则
// if this subgroup contains the primary, it's automatically always up-to-date
for( set<MemberCfg*>::const_iterator cfg = (*sgs).second->m.begin();
cfg != (*sgs).second->m.end();
cfg++)
{
if ((*cfg)->h.isSelf()) {
node->actualTarget--;
foundMe = true;
}
}
scoped_lock lk(groupMx);//foundme为true,表示这个subgroup已经更新了,需要检查其他
for (set<MemberCfg *>::iterator cfg = (*sgs).second->m.begin();//group是否更新,所以这里在memcfg中插入
!foundMe && cfg != (*sgs).second->m.end(); cfg++) {//TagSubgroup时是在!foundme的情况下
(*cfg)->groupsw().insert((*sgs).second);
}
}
// if all of the members of this clause involve the primary, it's always up-to-date
if (node->actualTarget == 0) {
node->last = OpTime(INT_MAX, INT_MAX);
primaryOnly++;
}
// this is a valid clause, so we want to add it to its rule
node->rule = r;
r->clauses.push_back(node);
}
// if all of the clauses are satisfied by the primary, this rule is trivially true
if (primaryOnly == r->clauses.size()) {
r->last = OpTime(INT_MAX, INT_MAX);
}
rules[rule.fieldName()] = r;
}
}
void ReplSetConfig::_populateTagMap(map<string,TagClause> &tagMap) {
// create subgroups for each server corresponding to each of
// its tags. E.g.:
// A is tagged with {"server" : "A", "dc" : "ny"}
// B is tagged with {"server" : "B", "dc" : "ny"}
// At the end of this step, tagMap will contain:
// "server" => {"A" : [A], "B" : [B]}
// "dc" => {"ny" : [A,B]}
for (unsigned i=0; i<members.size(); i++) {
MemberCfg member = members[i];
for (map<string,string>::iterator tag = member.tags.begin(); tag != member.tags.end(); tag++) {
string label = (*tag).first;
string value = (*tag).second;
TagClause& clause = tagMap[label];
clause.name = label;
TagSubgroup* subgroup;
// search for "ny" in "dc"'s clause
if (clause.subgroups.find(value) == clause.subgroups.end()) {//有同一个值如dc:ny,这里ny表示其对应的服务器在
clause.subgroups[value] = subgroup = new TagSubgroup(value);//同一个subgroup中,subgroup中一台服务器时间戳更新
}//表明这个subgroup时间戳更新.
else {
subgroup = clause.subgroups[value];
}
subgroup->m.insert(&members[i]);//记录subgroup中的成员.
}
}
}
void BackgroundSync::notifierThread() {//只干一件事情,从服务端读出操作时间戳,读到lastOpTimeWritten后就进入等待状态
Client::initThread("rsSyncNotifier");
while (!inShutdown()) {
bool clearTarget = false;
MemberState state = theReplSet->state();
if (state.primary() || state.fatal() || state.startup()) {
sleepsecs(5);
continue;
}
try {
{
boost::unique_lock<boost::mutex> lock(_lastOpMutex);
while (_consumedOpTime == theReplSet->lastOpTimeWritten) {
_lastOpCond.wait(lock);
}
}
markOplog();
}
}
cc().shutdown();
}
void BackgroundSync::markOplog() {
if (!hasCursor()) {//连接服务器,得到游标
sleepsecs(1);
return;
}
if (!_oplogMarker.moreInCurrentBatch()) {
_oplogMarker.more();
}
if (!_oplogMarker.more()) {
_oplogMarker.tailCheck();
sleepsecs(1);
return;
}
// if this member has written the op at optime T, we want to nextSafe up to and including T
while (_consumedOpTime < theReplSet->lastOpTimeWritten && _oplogMarker.more()) {//更新_consumedOpTime时间直到lastOpTimeWritten
BSONObj temp = _oplogMarker.nextSafe();
_consumedOpTime = temp["ts"]._opTime();
}
// call more() to signal the sync target that we've synced T
_oplogMarker.more();
}
bool BackgroundSync::hasCursor() {
{
// prevent writers from blocking readers during fsync
SimpleMutex::scoped_lock fsynclk(filesLockedFsync);
// we don't need the local write lock yet, but it's needed by OplogReader::connect
// so we take it preemptively to avoid deadlocking.
Lock::DBWrite lk("local");
boost::unique_lock<boost::mutex> lock(_mutex);
if (!_oplogMarkerTarget || _currentSyncTarget != _oplogMarkerTarget) {
if (!_currentSyncTarget) {
return false;
}
_oplogMarkerTarget = _currentSyncTarget;//服务端和sync时读取的服务端一样
_oplogMarker.resetConnection();
if (!_oplogMarker.connect(_oplogMarkerTarget->fullName())) {
_oplogMarkerTarget = NULL;
return false;
}
}
}
if (!_oplogMarker.haveCursor()) {//只查询其操作日志时间戳
BSONObj fields = BSON("ts" << 1);
_oplogMarker.tailingQueryGTE(rsoplog, theReplSet->lastOpTimeWritten, &fields);
}
return _oplogMarker.haveCursor();
}
BackgroundSync::BackgroundSync() : _buffer(256*1024*1024, &getSize),
_lastOpTimeFetched(0, 0),
_lastH(0),
_pause(true),
_currentSyncTarget(NULL),
_oplogMarkerTarget(NULL),
_oplogMarker(true /* doHandshake */),这notifierThread连接sync端是需要握手的
_consumedOpTime(0, 0) {
}
void BackgroundSync::produce() {//这里produceThread从sync端读取数据连接时是不需要握手的
OplogReader r(false /* doHandshake */);
getOplogReader(r);
}
bool OplogReader::connect(string hostName) {
if ( ! commonConnect(hostName) )
return false;
if ( _doHandshake && ! replHandshake(_conn.get() ) )//这里是握手部分.
return false;
return true;
}
bool replHandshake(DBClientConnection *conn) {
string myname = getHostName();
BSONObj me;
{
Lock::DBWrite l("local");
// local.me is an identifier for a server for getLastError w:2+
if ( ! Helpers::getSingleton( "local.me" , me ) ||//读取数据失败则清空collection,然后自动生成一个OID,将其存入local.me中
! me.hasField("host") ||
me["host"].String() != myname ) {
// clean out local.me
Helpers::emptyCollection("local.me");
BSONObjBuilder b;
b.appendOID( "_id" , 0 , true );
b.append( "host", myname );
me = b.obj();
Helpers::putSingleton( "local.me" , me );
}
}
BSONObjBuilder cmd;//握手将自己的member id,OID和host地址发送给同步端.
cmd.appendAs( me["_id"] , "handshake" );
if (theReplSet) {
cmd.append("member", theReplSet->selfId());
}
BSONObj res;
bool ok = conn->runCommand( "admin" , cmd.obj() , res );
return true;
}
while (_consumedOpTime == theReplSet->lastOpTimeWritten) {
_lastOpCond.wait(lock);
while (_consumedOpTime < theReplSet->lastOpTimeWritten && _oplogMarker.more()) {
BSONObj temp = _oplogMarker.nextSafe();
_consumedOpTime = temp["ts"]._opTime();
}
void BackgroundSync::notify() {//在notify中将通知上面的等待过程.
boost::unique_lock<boost::mutex> lock(s_mutex);
boost::unique_lock<boost::mutex> opLock(s_instance->_lastOpMutex);
s_instance->_lastOpCond.notify_all();
}
void _logOpObjRS(const BSONObj& op) {
if( theReplSet ) {
theReplSet->lastOpTimeWritten = ts;
theReplSet->lastH = h;
ctx.getClient()->setLastOp( ts );
replset::BackgroundSync::notify();
}
}
virtual bool run(const string& , BSONObj& cmdObj, int, string& errmsg, BSONObjBuilder& result, bool fromRepl) {
Client& c = cc();
c.gotHandshake( cmdObj );//注意的是一个连接表示一个线程,这里的Client是这个线程独有的.
return 1;
}
void Client::gotHandshake( const BSONObj& o ) {
BSONObjIterator i(o);
BSONElement id = i.next();
_remoteId = id.wrap( "_id" );//这个线程独有的OID,保存到这里
BSONObjBuilder b;
while ( i.more() )
b.append( i.next() );
b.appendElementsUnique( _handshake );//得到和之前handshake独特的数据
_handshake = b.obj();//保存handshake数据.
if (theReplSet && o.hasField("member")) {//建立结构
theReplSet->ghost->associateSlave(_remoteId, o["member"].Int());
}
}
void GhostSync::associateSlave(const BSONObj& id, const int memberId) {
const OID rid = id["_id"].OID();//得到独特的OID
rwlock lk( _lock , true );
shared_ptr<GhostSlave> &g = _ghostCache[rid];
if( g.get() == 0 ) {
g.reset( new GhostSlave() );//新建一个GhostSlave结构.
}
GhostSlave &slave = *g;
slave.slave = (Member*)rs->findById(memberId);//根据memberId将OID结构与member关联
if (slave.slave != 0)
slave.init = true;
}
下面来看GetMore数据的请求,其请求数据流程为receivedGetMore->processGetMore,在processGetMore函数中有这么一条语句:
if ( pass == 0 )//这就是我们这里关注的地点.pass目前2.2的代码一直为0
cc->updateSlaveLocation( curop );
void ClientCursor::updateSlaveLocation( CurOp& curop ) {
if ( _slaveReadTill.isNull() )
return;
mongo::updateSlaveLocation( curop , _ns.c_str() , _slaveReadTill );
}
string queryWithQueryOptimizer() {
OpTime slaveReadTill;
// Note slave's position in the oplog.
if ( pq.hasOption( QueryOption_OplogReplay ) ) {
BSONObj current = cursor->current();
BSONElement e = current["ts"];
if ( e.type() == Date || e.type() == Timestamp ) {
slaveReadTill = e._opTime();
}
}
// Save slave's position in the oplog.
if ( pq.hasOption( QueryOption_OplogReplay ) && !slaveReadTill.isNull() ) {
ccPointer->slaveReadTill( slaveReadTill );
}
}
void updateSlaveLocation( CurOp& curop, const char * ns , OpTime lastOp ) {
if ( lastOp.isNull() )
return;
Client * c = curop.getClient();
BSONObj rid = c->getRemoteID();//将自己看着primary端,则这里读取local.oplog.rs可以当作
if ( rid.isEmpty() )//是secondary在读取操作日志然后replay
return;
//这里的lastOp是上次getMore操作结束后最后一条记录的时间,
//而该函数的调用发生在这一次getmore操作的开始,表示服务
//secondary已经将数据正确的写入了自己的数据库中
slaveTracking.update( rid , curop.getRemoteString( false ) , ns , lastOp );//这个线程保存的OID,来自slave端
if (theReplSet && !theReplSet->isPrimary()) {
// we don't know the slave's port, so we make the replica set keep
// a map of rids to slaves
theReplSet->ghost->send( boost::bind(&GhostSync::percolate, theReplSet->ghost, rid, lastOp) );
}
}
void update( const BSONObj& rid , const string& host , const string& ns , OpTime last ) {
Ident ident(rid,host,ns);
scoped_lock mylk(_mutex);
_slaves[ident] = last;
_dirty = true;
if (theReplSet && theReplSet->isPrimary()) {//primary端更新secondary端同步的时间戳
theReplSet->ghost->updateSlave(ident.obj["_id"].OID(), last);
}
if ( ! _started ) {//启动线程,将每一个slave同步到的时间记录到local.slaves中
// start background thread here since we definitely need it
_started = true;
go();
}
_threadsWaitingForReplication.notify_all();
}
void GhostSync::updateSlave(const mongo::OID& rid, const OpTime& last) {
rwlock lk( _lock , false );
MAP::iterator i = _ghostCache.find( rid );//握手时保留的OID
GhostSlave& slave = *(i->second);//通过OID找到GhostSlave,找到member,然后再找到membercfg,然后membercfg与tags关联
((ReplSetConfig::MemberCfg)slave.slave->config()).updateGroups(last);
}
void updateGroups(const OpTime& last) {//membercfg中的tagSubgroup是在parseRules中插入的.
scoped_lock lk(ReplSetConfig::groupMx);//当前secondary更新则更新其所在的所有tagSubGroup的时间戳
for (set<TagSubgroup*>::const_iterator it = groups().begin(); it != groups().end(); it++)
(*it)->updateLast(last);
} void ReplSetConfig::TagSubgroup::updateLast(const OpTime& op) {
if (last < op) {//这个group的时间更新到这个时间点上
last = op;//这个group的时间戳更新则需要更新所有加在这个tagGroup上的TagClause上的时间戳.
for (vector<TagClause*>::iterator it = clauses.begin(); it < clauses.end(); it++)
(*it)->updateLast(op);
}
}
void ReplSetConfig::TagClause::updateLast(const OpTime& op) {
if (last >= op) {
return;
}
// check at least n subgroups greater than clause.last
int count = 0;
map<string,TagSubgroup*>::iterator it;//满足同一个时间点的group数大于预设置的group数目,则需要更新rule的时间戳
for (it = subgroups.begin(); it != subgroups.end(); it++) {
if ((*it).second->last >= op) {
count++;
}
}
if (count >= actualTarget) {//满足这条clause要求的target数目了
last = op;
rule->updateLast(op);
}
}
void ReplSetConfig::TagRule::updateLast(const OpTime& op) {
OpTime *earliest = (OpTime*)&op;
vector<TagClause*>::iterator it;
for (it = clauses.begin(); it < clauses.end(); it++) {
if ((*it)->last < *earliest) {//一条规则多条clause需满足,只能选取其中时间最早的
earliest = &(*it)->last;//作为这条rule的时间
}
}
// rules are simply and-ed clauses, so whatever the most-behind
// clause is at is what the rule is at
last = *earliest;
}
bool run(const string& dbname, BSONObj& _cmdObj, int, string& errmsg, BSONObjBuilder& result, bool fromRepl) {
BSONElement e = cmdObj["w"];
if ( e.ok() ) {
int timeout = cmdObj["wtimeout"].numberInt();
Timer t;
long long passes = 0;
char buf[32];
while ( 1 ) {
OpTime op(c.getLastOp());//得到上一个操作的时间戳,getLastError当然只针对上一个操作了
if ( op.isNull() ) {
if ( anyReplEnabled() ) {//只有在开启replset模式时才w才算有用
result.append( "wnote" , "no write has been done on this connection" );
}
else if ( e.isNumber() && e.numberInt() <= 1 ) {
// don't do anything
// w=1 and no repl, so this is fine
}
else {
// w=2 and no repl
result.append( "wnote" , "no replication has been enabled, so w=2+ won't work" );
result.append( "err", "norepl" );
return true;
}
break;
}
// check this first for w=0 or w=1
if ( opReplicatedEnough( op, e ) ) {//这个是我们真正关心的函数
break;
}
// if replication isn't enabled (e.g., config servers)
if ( ! anyReplEnabled() ) {
result.append( "err", "norepl" );
return true;
}//超时返回
if ( timeout > 0 && t.millis() >= timeout ) {
result.append( "wtimeout" , true );
errmsg = "timed out waiting for slaves";
result.append( "waited" , t.millis() );
result.append( "err" , "timeout" );
return true;
}
sleepmillis(1);
}
result.appendNumber( "wtime" , t.millis() );
}
result.appendNull( "err" );
return true;
} bool opReplicatedEnough( OpTime op , BSONElement w ) {
string wStr = w.String();
if (wStr == "majority") {//默认目标,要求超过半数的服务器同步到这个时间点了
// use the entire set, including arbiters, to prevent writing
// to a majority of the set but not a majority of voters
return replicatedToNum(op, theReplSet->config().getMajority());
}//根据名字查找对应的rule,比照上一条操作记录的时间戳与这条rule的当前时间戳,小于表示rule满足要求,已经同步到这个时间点了,否则表示还没有同步到这个时间点
map<string,ReplSetConfig::TagRule*>::const_iterator it = theReplSet->config().rules.find(wStr);
return op <= (*it).second->last;
}
最后来看看updateSlaveLocation部分非primary服务器的流程.非primary部分调用percolate更新slave同步时间点.
void updateSlaveLocation( CurOp& curop, const char * ns , OpTime lastOp ) {
if (theReplSet && !theReplSet->isPrimary()) {
// we don't know the slave's port, so we make the replica set keep
// a map of rids to slaves
theReplSet->ghost->send( boost::bind(&GhostSync::percolate, theReplSet->ghost, rid, lastOp) );
}
}
void GhostSync::percolate(const BSONObj& id, const OpTime& last) {
const OID rid = id["_id"].OID();
GhostSlave* slave;
{
rwlock lk( _lock , false );
MAP::iterator i = _ghostCache.find( rid );
if ( i == _ghostCache.end() ) {
return;
}
slave = i->second.get();
if (!slave->init) {
return;
}
}
const Member *target = replset::BackgroundSync::get()->getSyncTarget();
if (!target || rs->box.getState().primary()
// we are currently syncing from someone who's syncing from us
// the target might end up with a new Member, but s.slave never
// changes so we'll compare the names
|| target == slave->slave || target->fullName() == slave->slave->fullName()) {
return;
}
{
if (!slave->reader.haveCursor()) {//连接slave端读取其操作日志时间戳,因为是slave端,可以确定slave的最后一条记录的时间戳就是该slave的时间戳,到这里就更新了其时间戳.
if (!slave->reader.connect(id, slave->slave->id(), target->fullName())) {
// error message logged in OplogReader::connect
return;
}
slave->reader.ghostQueryGTE(rsoplog, last);
}
if (slave->last > last) {
return;
}
while (slave->last <= last) {
if (!slave->reader.more()) {
// we'll be back
return;
}
BSONObj o = slave->reader.nextSafe();
slave->last = o["ts"]._opTime();
}
}
}
原文链接:mongodb源码分析(十八)replication replset tags
作者: yhjj0108,杨浩
扫一扫
1819
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
