Hadoop NameNode启动之FSImage、edits重置(三)

最新推荐文章于 2023-12-03 18:32:45 发布

原创最新推荐文章于 2023-12-03 18:32:45 发布 · 2.1k 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

55 篇文章

订阅专栏

本文详细解析Hadoop中Namenode启动过程中的FSImage与edits日志重置操作，包括关闭旧日志文件、重命名目录、保存新的镜像文件及创建新的edits日志等关键步骤。

Namenode启动过程中，在FSImage载入后，会对FSImage和edits重置，相当于创建一组新的镜像文件和日志文件，但前提是前面的载入都已经成功了，这个操作紧跟fsImage.recoverTransitionRead之后，具体调用流程如下：

Namenode.main ->NameNode.createNameNode -> NameNode.initialize -> FSNameSystem.initialize-> FSDirecotry.loadFSImage -> fsImage.saveNamespace

这个函数内部流程很清晰，会经历如下步骤：

1、关闭edits日志文件

2、将当前current目录重命名为lastcheckpoint.tmp

3、保存新的image到current目录

4、在current目录中创建edits

5、将第二步中lastcheckpoint.tmp重命名为previous.checkpoint

6、打开fsiamge和edits文件

void  saveNamespace(boolean renewCheckpointTime) throws IOException {
    //关闭edits日志文件
    editLog.close();
    if(renewCheckpointTime)
      this.checkpointTime = FSNamesystem.now();
 
   //将当前current目录重命名为lastcheckpoint.tmp
   for(Iterator<StorageDirectory> it = dirIterator(); it.hasNext();) {
      StorageDirectory sd = it.next();
      try {
        //开始重命名操作，实际调用File.renameTo函数
        moveCurrent(sd);
      } catch(IOException ie) {
        LOG.error("Unable to move current for " + sd.getRoot(), ie);
        removeStorageDir(sd.getRoot());
      }
    }
 
    // save images into current
    for (Iterator<StorageDirectory> it =dirIterator(NameNodeDirType.IMAGE);it.hasNext();){
      StorageDirectory sd = it.next();
      try {
       //在这个函数中会创建current目录，保存新的fsimage、创建新的edits，
       //注意此时edits是初始化状态，只写入了一个LAYOUT_VERSION
       //创建VERSION fstime文件
        saveCurrent(sd);
      } catch(IOException ie) {
        LOG.error("Unable to save image for " + sd.getRoot(), ie);
        removeStorageDir(sd.getRoot());
      }
    }
 
    // -NOTE-
    // If NN has image-only and edits-onlystorage directories and
    // fails here
    // the image will have the latest namespace state.
    // During startup the image-only directories will recoverby discarding
    // lastcheckpoint.tmp, while
    // the edits-only directories will recover by fallingback
    // to the old state contained in theirlastcheckpoint.tmp.
    // The edits directories should be discarded duringstartup because their
    // checkpointTime is older than that of imagedirectories.
 
    // recreate edits in current
   for (Iterator<StorageDirectory> it =dirIterator(NameNodeDirType.EDITS);it.hasNext();) {
      StorageDirectory sd = it.next();
      try {
        saveCurrent(sd);
      } catch(IOException ie) {
        LOG.error("Unable to save edits for " + sd.getRoot(), ie);
        removeStorageDir(sd.getRoot());
      }
  }
   //将第二步中lastcheckpoint.tmp重命名为previous.checkpoint
 
   for(Iterator<StorageDirectory> it = dirIterator(); it.hasNext();) {
      StorageDirectory sd = it.next();
      try {
        moveLastCheckpoint(sd);
      } catch(IOException ie) {
        LOG.error("Unable to move last checkpoint for" + sd.getRoot(),ie);
        removeStorageDir(sd.getRoot());
      }
    }
    if(!editLog.isOpen()) editLog.open();
    ckptState = CheckpointStates.UPLOAD_DONE;
  }