HBase 0.99 源代码分析 - Master启动过程(2)

下面继续从HMaster.finishActiveMasterInitialization方法开始分析HBase 0.99 Master启动过程。HMaster.finishActiveMasterInitialization方法主要完成如下功能：

1. 初始化Master功能组件：file system manager, server manager, assignment manager, region server tracker 等。

2. 启动服务线程：balancer,catalog janitor, executor services 等。

3. 在ZooKeeper中设置集群状态为UP

4. 等待Region server登录

5. 如果需要的话，拆分Log, 执行数据恢复

6. 分配meta, namespace和region

7. 处理集群启动或master failover

8. 调用coprocessor

HMaster.finishActiveMasterInitialization方法源代码比较长，下面我们逐步进行分析。

  private void finishActiveMasterInitialization(MonitoredTask status)
      throws IOException, InterruptedException, KeeperException, CoordinatedStateException {

    isActiveMaster = true;

    /*
     * We are active master now... go initialize components we need to run.
     * Note, there may be dross in zk from previous runs; it'll get addressed
     * below after we determine if cluster startup or failover.
     */

    status.setStatus("Initializing Master file system");

    this.masterActiveTime = System.currentTimeMillis();
    // TODO: Do this using Dependency Injection, using PicoContainer, Guice or Spring.
    this.fileSystemManager = new MasterFileSystem(this, this);

    // publish cluster ID
    status.setStatus("Publishing Cluster ID in ZooKeeper");
    ZKClusterId.setClusterId(this.zooKeeper, fileSystemManager.getClusterId());
    this.serverManager = createServerManager(this, this);

    synchronized (this) {
      if (shortCircuitConnection == null) {
        shortCircuitConnection = createShortCircuitConnection();
        metaTableLocator = new MetaTableLocator();
      }
    }

    // Invalidate all write locks held previously
    this.tableLockManager.reapWriteLocks();

    this.tableStateManager = new TableStateManager(this);
    this.tableStateManager.start();

    status.setStatus("Initializing ZK system trackers");
    initializeZKBasedSystemTrackers();

    // initialize master side coprocessors before we start handling requests
    status.setStatus("Initializing master coprocessors");
    this.cpHost = new MasterCoprocessorHost(this, this.conf);

上面代码执行了如下功能：

1. 设置本Master的isActiveMaster标志为true

2. 设置masterActiveTime为当前系统时间

3. 初始化filesystem manager类MasterFileSystem,该类包装了Master与底层文件系统交互时的一些操作，例如Log分割，检查文件系统状态等。初始化时读取配置并设置HBase跟路径，临时路径等，并初始化SplitLogManager

4. 向ZooKeeper设置cluster ID.

5. 创建server manager, server manager管理所有Region server, 包括在线和死亡的Region server, 处理Region server的启动，关闭和死亡

6. 创建到自身Regionserver的HConnection连接，存储到成员变量shortCircuitConnection中，可通过该连接绕过RPC, 执行HConnection客户端请求。

7. 初始化metaTableLocator对象, 该对象用于定位meta 表hbase:meta

8. 释放所有tablewrite lock

9. 初始化tablestate manager对象，该对象用于管理表的状态

10. 初始化所有基于ZooKeeper的system tracker, 包括：

  1). load balancer：默认实现类为 StochasticLoadBalancer, load balancer负责管理region 在regionserver之间的迁移方案。可以通过配置项hbase.master.loadbalancer.class更改实现类。对于默认实现类StochasticLoadBalancer，可以通过如下配置项控制其行为：

    hbase.master.balancer.stochastic.regionLoadCost

    hbase.master.balancer.stochastic.moveCost

    hbase.master.balancer.stochastic.tableLoadCost

    hbase.master.balancer.stochastic.localityCost

    hbase.master.balancer.stochastic.memstoreSizeCost

    hbase.master.balancer.stochastic.storefileSizeCost    

    hbase.master.balancer.stochastic.maxMoveRegions

    hbase.master.balancer.stochastic.stepsPerRegion

    hbase.master.balancer.stochastic.maxSteps

  2). load balancertracker： 基于ZooKeeper跟踪load balancer状态。

  3). assignmentmanager：负责region的分配。

  4). region servertracker： 通过ZooKeeper跟踪在线的region server

  5). draining server tracker：通过ZooKeeper管理drainingregion server。 在大型集群中, 用户有时需要把一部分region server服务器一起停止服务迁出HBase集群，这些将要被迁出的region server称为draining region server。被迁走的region server上的region需要被分配到其他region server上，为避免把一个draining server上的region迁到另一个draining server上，hbase会一次把所有draining server加入draining server列表。

  6). snapshot manager：负责管理 HBase快照功能，包括创建快照，恢复快照。快照是HBase从0.94.6开始提供的功能，创建快照时并不复制数据，因此速度很快，对region server性能基本没有影响。

11. 初始化master协处理器(coprocessor) MasterCoprocessorHost,负责管理所有master相关操作的协处理器。

继续看finishActiveMasterInitialization接下来的代码

    // start up all service threads.
    status.setStatus("Initializing master service threads");
    startServiceThreads();

    // Wake up this server to check in
    sleeper.skipSleepCycle();

    // Wait for region servers to report in
    this.serverManager.waitForRegionServers(status);
    // Check zk for region servers that are up but didn't register
    for (ServerName sn: this.regionServerTracker.getOnlineServers()) {
      // The isServerOnline check is opportunistic, correctness is handled inside
      if (!this.serverManager.isServerOnline(sn)
          && serverManager.checkAndRecordNewServer(sn, ServerLoad.EMPTY_SERVERLOAD)) {
        LOG.info("Registered server found up in zk but who has not yet reported in: " + sn);
      }
    }

上面代码执行如下功能：

1. 调用startServiceThreads方法，该方法首先按下表中的个数启动如下服务线程：

服务线程类型	配置项	默认线程个数
MASTER_OPEN_REGION	hbase.master.executor.openregion.threads	5
MASTER_CLOSE_REGION	hbase.master.executor.closeregion.threads	5
MASTER_SERVER_OPERATIONS	hbase.master.executor.serverops.threads	5
MASTER_META_SERVER_OPERATIONS	hbase.master.executor.serverops.threads	5
M_LOG_REPLAY_OPS	hbase.master.executor.logreplayops.threads	10
MASTER_TABLE_OPERATIONS	不可配置	1

 

然后读取配置项hbase.master.cleaner.interval作为log cleaner时间间隔，默认为1分钟，启动log cleaner线程，该线程每到一个时间间隔清理oldWALs目录下旧的HLog.

然后启动HFile archive清理线程，清理archive目录下的HFile archive.

置服务线程启动标志serviceStarted为true.

2. 唤醒当前server允许check in.

3. 等待region server check in. 满足如下条件之一则等待完毕：

   - master 停止

   -达到配置项hbase.master.wait.on.regionservers.maxtostart指定的region server个数

   -达到配置项hbase.master.wait.on.regionservers.mintostart指定的region server个数并且  在配置项hbase.master.wait.on.regionservers.interval指定的间隔内没有新的region server并且达到配置项hbase.master.wait.on.regionservers.timeout超时时间

4. 获取在线region server列表并进行注册。

继续看finishActiveMasterInitialization:

    // get a list for previously failed RS which need log splitting work
    // we recover hbase:meta region servers inside master initialization and
    // handle other failed servers in SSH in order to start up master node ASAP
    Set previouslyFailedServers = this.fileSystemManager
        .getFailedServersFromLogFolders();

    // remove stale recovering regions from previous run
    this.fileSystemManager.removeStaleRecoveringRegionsFromZK(previouslyFailedServers);

    // log splitting for hbase:meta server
    ServerName oldMetaServerLocation = metaTableLocator.getMetaRegionLocation(this.getZooKeeper());
    if (oldMetaServerLocation != null && previouslyFailedServers.contains(oldMetaServerLocation)) {
      splitMetaLogBeforeAssignment(oldMetaServerLocation);
      // Note: we can't remove oldMetaServerLocation from previousFailedServers list because it
      // may also host user regions
    }
    Set previouslyFailedMetaRSs = getPreviouselyFailedMetaServersFromZK();
    // need to use union of previouslyFailedMetaRSs recorded in ZK and previouslyFailedServers
    // instead of previouslyFailedMetaRSs alone to address the following two situations:
    // 1) the chained failure situation(recovery failed multiple times in a row).
    // 2) master get killed right before it could delete the recovering hbase:meta from ZK while the
    // same server still has non-meta wals to be replayed so that
    // removeStaleRecoveringRegionsFromZK can't delete the stale hbase:meta region
    // Passing more servers into splitMetaLog is all right. If a server doesn't have hbase:meta wal,
    // there is no op for the server.
    previouslyFailedMetaRSs.addAll(previouslyFailedServers);

getFailedServersFromLogFolders方法找出之前正在recovering但已down掉，却在WALs目录下仍然遗留log的server

removeStaleRecoveringRegionsFromZK从ZooKeerper删除/hbase/recovering-regions/下的不在线的节点，即上面找出的server. 此处并不是删除log, 而是删除ZooKeeper中down掉的server信息，后面会对每个region重新分配region server

如果meta表被分配给down掉的server,先分离meta log

从ZooKeeper获取之前down掉的meta server，合并down掉的meta server和region server列表。

    this.initializationBeforeMetaAssignment = true;

    // Wait for regionserver to finish initialization.
    if (BaseLoadBalancer.tablesOnMaster(conf)) {
      waitForServerOnline();
    }

    //initialize load balancer
    this.balancer.setClusterStatus(getClusterStatus());
    this.balancer.setMasterServices(this);
    this.balancer.initialize();

    // Check if master is shutting down because of some issue
    // in initializing the regionserver or the balancer.
    if(isStopped()) return;

    // Make sure meta assigned before proceeding.
    status.setStatus("Assigning Meta Region");
    assignMeta(status, previouslyFailedMetaRSs);
    // check if master is shutting down because above assignMeta could return even hbase:meta isn't
    // assigned when master is shutting down
    if(isStopped()) return;

等待region server初始化完成。

初始化load balancer。

分配meta region到region server。

    status.setStatus("Submitting log splitting work for previously failed region servers");
    // Master has recovered hbase:meta region server and we put
    // other failed region servers in a queue to be handled later by SSH
    for (ServerName tmpServer : previouslyFailedServers) {
      this.serverManager.processDeadServer(tmpServer, true);
    }

    // Fix up assignment manager status
    status.setStatus("Starting assignment manager");
    this.assignmentManager.joinCluster();

    //set cluster status again after user regions are assigned
    this.balancer.setClusterStatus(getClusterStatus());

    // Start balancer and meta catalog janitor after meta and regions have
    // been assigned.
    status.setStatus("Starting balancer and catalog janitor");
    this.clusterStatusChore = new ClusterStatusChore(this, balancer);
    Threads.setDaemonThreadRunning(clusterStatusChore.getThread());
    this.balancerChore = new BalancerChore(this);
    Threads.setDaemonThreadRunning(balancerChore.getThread());
    this.catalogJanitorChore = new CatalogJanitor(this, this);
    Threads.setDaemonThreadRunning(catalogJanitorChore.getThread());

processDeadServer会调用ServerShutdownHandler类进行日志分离。

调用assignment manager进行用户region 分配。

    status.setStatus("Starting namespace manager");
    initNamespace();

    status.setStatus("Starting quota manager");
    initQuotaManager();

    if (this.cpHost != null) {
      try {
        this.cpHost.preMasterInitialization();
      } catch (IOException e) {
        LOG.error("Coprocessor preMasterInitialization() hook failed", e);
      }
    }

    status.markComplete("Initialization successful");
    LOG.info("Master has completed initialization");
    initialized = true;
    // clear the dead servers with same host name and port of online server because we are not
    // removing dead server with same hostname and port of rs which is trying to check in before
    // master initialization. See HBASE-5916.
    this.serverManager.clearDeadServersWithSameHostNameAndPortOfOnlineServer();

    if (this.cpHost != null) {
      // don't let cp initialization errors kill the master
      try {
        this.cpHost.postStartMaster();
      } catch (IOException ioe) {
        LOG.error("Coprocessor postStartMaster() hook failed", ioe);
      }
    }

启动namespace manager和quota manager。

最后调用master启动完成协处理器。

至此master启动完成。