spark源码阅读笔记RDD（五） RDD中的checkpoint原理

----------------------------目录----------------------------

为何需要checkpoint？

checkPoint作用

源码分析

------------------------------------------------------------

为何需要checkpoint？

大家知道checkpoint和persist都是把数据“保存起来”，persist保存的形式有：磁盘/内存/序列化/这三种，其中

tackyon功能还没有，存储类源码如下：

class StorageLevel private(
    private var _useDisk: Boolean,//磁盘
    private var _useMemory: Boolean,//内存
    private var _useOffHeap: Boolean,//tachyon或叫alluxio
    private var _deserialized: Boolean,//反序列化
    private var _replication: Int = 1)
  extends Externalizable {/*集合体*/}

（1）当persist把数据放到内存的时候，因为我们在处理大量数据的时候，我们进行persist的时候，可能把之前的数

据给挤掉。所以persist数据到内存中，虽然说非常快速，但是可以说是最不可靠的一种存储。

（2）当数据放在磁盘的时候，因为存储的数据放在磁盘的共同文件夹下，正如我们storageLevel设置默认为复制一份

也就是说当有我们存储数据的这个磁盘毁坏了，那么就说我们的数据丢失了。
（3）checkpoint就是为了解决（2）出现的问题：把数据放到HDFS中。借助hdfs的高容错、高可靠的特征来达到更

加可靠的数据持久化
note：集群模式下，hdfs存储数据一般是3份放三个节点中（a,b,c）,a在一个机架，bc在另一个机架
（一份数据复制成三份，放在三个节点二个机架中）

checkPoint作用
当checkpoint为当前RDD设置检查点的时候，该函数将会创建一个二进制的文件，并存储到checkpoint目录中，
该目录是用SparkContext.setCheckpointDir()设置的。在checkpoint的过程中，该RDD的所有依赖于父RDD中的信息

将全部被移出。对RDD进行checkpoint操作并不会马上被执行，必须执行Action操作才能触发。当需要checkpoint的

数据的时候，通过ReliableCheckpointRDD的readCheckpointFile方法来从file路径里面读出已经Checkpint的数据，然

后加以应该

源码分析

checkPoint设置的存储位置，这个存储路径必须是HDFS的路径

/**
   * 设置目录，用于存储checkpoint的RDD，如果是集群，这个目录必须是HDFS路径
   */
  def setCheckpointDir(directory: String) {
	//我们运行在集群中，如果把目录设置为本地，那么提出警告
	//另外，driver 会尝试在本地重新构建checkpoint的RDD
	//由于文件其实是在executor上的，所以会提出警告
    if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) {
      logWarning("Spark is not running in local mode, therefore the checkpoint directory " +
        s"must not be on the local filesystem. Directory '$directory' " +
        "appears to be on the local filesystem.")
    }

    checkpointDir = Option(directory).map { dir =>
      val path = new Path(dir, UUID.randomUUID().toString)//path增加一个随机码
      val fs = path.getFileSystem(hadoopConfiguration)//加载一个hadoop文件系统的配置
      fs.mkdirs(path)//创建一个hdfs下的文件
      fs.getFileStatus(path).getPath.toString//把hdfs下的文件路径通过string返回
    }
  }

RDD下的checkpoint函数：首先检查checkpoint文件目录是否为空，如果不空，那么再检查我们想要的checkpoint

的数据是否为空如果不空

  def checkpoint(): Unit = RDDCheckpointData.synchronized {
    // NOTE: we use a global lock here due to complexities downstream with ensuring
    // children RDD partitions point to the correct parent partitions. In the future
    // we should revisit this consideration.
    if (context.checkpointDir.isEmpty) {
      throw new SparkException("Checkpoint directory has not been set in the SparkContext")
    } else if (checkpointData.isEmpty) {
      checkpointData = Some(new ReliableRDDCheckpointData(this))
    }
  }

 发现它是用ReliableRDDCheckpointData(this)来把数据进行存储的，在来看ReliableRDDCheckpointData类

 /**
 * An implementation of checkpointing that writes the RDD data to reliable storage.
 * This allows drivers to be restarted on failure with previously computed state.
 */
private[spark] class ReliableRDDCheckpointData[T: ClassTag](@transient private val rdd: RDD[T])
  extends RDDCheckpointData[T](rdd) with Logging {

  // The directory to which the associated RDD has been checkpointed to
  // This is assumed to be a non-local path that points to some reliable storage
  //找到文件存储位置的名字，并用cpDir这个（key，value）的RDD存储
  private val cpDir: String =
    ReliableRDDCheckpointData.checkpointPath(rdd.context, rdd.id)
      .map(_.toString)
      .getOrElse { throw new SparkException("Checkpoint dir must be specified.") }

  /**
   * Return the directory to which this RDD was checkpointed.
   * If the RDD is not checkpointed yet, return None.
   */
   //查看我们需要checkpoint的RDD是否已经checkpoint过，如果没有checkpoint，那么返回None
  def getCheckpointDir: Option[String] = RDDCheckpointData.synchronized {
    if (isCheckpointed) {
      Some(cpDir.toString)
    } else {
      None
    }
  }

  /**
   * Materialize this RDD and write its content to a reliable DFS.
   * This is called immediately after the first action invoked on this RDD has completed.
   * 
   */
  protected override def doCheckpoint(): CheckpointRDD[T] = {
    val newRDD = ReliableCheckpointRDD.writeRDDToCheckpointDirectory(rdd, cpDir)

    // Optionally clean our checkpoint files if the reference is out of scope
    if (rdd.conf.getBoolean("spark.cleaner.referenceTracking.cleanCheckpoints", false)) {
      rdd.context.cleaner.foreach { cleaner =>
        cleaner.registerRDDCheckpointDataForCleanup(newRDD, rdd.id)
      }
    }

    logInfo(s"Done checkpointing RDD ${rdd.id} to $cpDir, new parent is RDD ${newRDD.id}")
    newRDD
  }

}

同时同名object有两个函数：checkpointPath和cleanCheckpoint

private[spark] object ReliableRDDCheckpointData extends Logging {

  /** Return the path of the directory to which this RDD's checkpoint data is written. */
  def checkpointPath(sc: SparkContext, rddId: Int): Option[Path] = {
    sc.checkpointDir.map { dir => new Path(dir, s"rdd-$rddId") }
  }

  /** Clean up the files associated with the checkpoint data for this RDD. */
  def cleanCheckpoint(sc: SparkContext, rddId: Int): Unit = {
    checkpointPath(sc, rddId).foreach { path =>
      val fs = path.getFileSystem(sc.hadoopConfiguration)
      if (fs.exists(path)) {
        if (!fs.delete(path, true)) {
          logWarning(s"Error deleting ${path.toString()}")
        }
      }
    }
  }
}

现在我们重点分析docheckpoint这个函数：通过ReliableCheckpointRDD.writeRDDToCheckpointDirectory(rdd,cpDir)

产生一个新RDD。

 /**
   * Write RDD to checkpoint files and return a ReliableCheckpointRDD representing the RDD.
   */
  def writeRDDToCheckpointDirectory[T: ClassTag](
      originalRDD: RDD[T],
      checkpointDir: String,
      blockSize: Int = -1): ReliableCheckpointRDD[T] = {

    val sc = originalRDD.sparkContext

    // Create the output path for the checkpoint
	//把checkpointDir设置我们checkpoint的目录
    val checkpointDirPath = new Path(checkpointDir)
    val fs = checkpointDirPath.getFileSystem(sc.hadoopConfiguration)
    if (!fs.mkdirs(checkpointDirPath)) {
      throw new SparkException(s"Failed to create checkpoint path $checkpointDirPath")
    }

    // Save to file, and reload it as an RDD
	//保存文件，同时把它作为一个RDD重新加载它
    val broadcastedConf = sc.broadcast(
      new SerializableConfiguration(sc.hadoopConfiguration))
    // TODO: This is expensive because it computes the RDD again unnecessarily (SPARK-8582)
	//发下這是非常消耗的，因为我们需要再次计算我们所需的这个RDD
    sc.runJob(originalRDD,
      writePartitionToCheckpointFile[T](checkpointDirPath.toString, broadcastedConf) _)

    if (originalRDD.partitioner.nonEmpty) {
      writePartitionerToCheckpointDir(sc, originalRDD.partitioner.get, checkpointDirPath)
    }

    val newRDD = new ReliableCheckpointRDD[T](
      sc, checkpointDirPath.toString, originalRDD.partitioner)
    if (newRDD.partitions.length != originalRDD.partitions.length) {
      throw new SparkException(
        s"Checkpoint RDD $newRDD(${newRDD.partitions.length}) has different " +
          s"number of partitions from original RDD $originalRDD(${originalRDD.partitions.length})")
    }
    newRDD
  }

分析：runJob重新计算我们需要的RDD，也就是说我们的RDD会再次计算，如果我们在checkpoint之前先对

这个RDD进行persisit的话，能达到更好的效果。我们再来看看runJob中的 writePartitionToCheckpointFile函数

   /**
   * Write a RDD partition's data to a checkpoint file.
   */
  def writePartitionToCheckpointFile[T: ClassTag](
      path: String,
      broadcastedConf: Broadcast[SerializableConfiguration],
      blockSize: Int = -1)(ctx: TaskContext, iterator: Iterator[T]) {
    val env = SparkEnv.get
    val outputDir = new Path(path)
    val fs = outputDir.getFileSystem(broadcastedConf.value.value)

    val finalOutputName = ReliableCheckpointRDD.checkpointFileName(ctx.partitionId())
    val finalOutputPath = new Path(outputDir, finalOutputName)
    val tempOutputPath =
      new Path(outputDir, s".$finalOutputName-attempt-${ctx.attemptNumber()}")

    if (fs.exists(tempOutputPath)) {
      throw new IOException(s"Checkpoint failed: temporary path $tempOutputPath already exists")
    }
    val bufferSize = env.conf.getInt("spark.buffer.size", 65536)

    val fileOutputStream = if (blockSize < 0) {
      fs.create(tempOutputPath, false, bufferSize)
    } else {
      // This is mainly for testing purpose
      fs.create(tempOutputPath, false, bufferSize,
        fs.getDefaultReplication(fs.getWorkingDirectory), blockSize)
    }
    val serializer = env.serializer.newInstance()
    val serializeStream = serializer.serializeStream(fileOutputStream)
    Utils.tryWithSafeFinally {
      serializeStream.writeAll(iterator)
    } {
      serializeStream.close()
    }

    if (!fs.rename(tempOutputPath, finalOutputPath)) {
      if (!fs.exists(finalOutputPath)) {
        logInfo(s"Deleting tempOutputPath $tempOutputPath")
        fs.delete(tempOutputPath, false)
        throw new IOException("Checkpoint failed: failed to save output of task: " +
          s"${ctx.attemptNumber()} and final output path does not exist: $finalOutputPath")
      } else {
        // Some other copy of this task must've finished before us and renamed it
        logInfo(s"Final output path $finalOutputPath already exists; not overwriting it")
        if (!fs.delete(tempOutputPath, false)) {
          logWarning(s"Error deleting ${tempOutputPath}")
        }
      }
    }
  }
  /**
   * Write a partitioner to the given RDD checkpoint directory. This is done on a best-effort
   * basis; any exception while writing the partitioner is caught, logged and ignored.
   */
  private def writePartitionerToCheckpointDir(
    sc: SparkContext, partitioner: Partitioner, checkpointDirPath: Path): Unit = {
    try {
      val partitionerFilePath = new Path(checkpointDirPath, checkpointPartitionerFileName)
      val bufferSize = sc.conf.getInt("spark.buffer.size", 65536)
      val fs = partitionerFilePath.getFileSystem(sc.hadoopConfiguration)
      val fileOutputStream = fs.create(partitionerFilePath, false, bufferSize)
      val serializer = SparkEnv.get.serializer.newInstance()
      val serializeStream = serializer.serializeStream(fileOutputStream)
      Utils.tryWithSafeFinally {
        serializeStream.writeObject(partitioner)
      } {
        serializeStream.close()
      }
      logDebug(s"Written partitioner to $partitionerFilePath")
    } catch {
      case NonFatal(e) =>
        logWarning(s"Error writing partitioner $partitioner to $checkpointDirPath")
    }
  }

writePartitionToCheckpointFile中的RDD加载了环境等信息。同时发RDD保存的信息放入这个RDD中。

最后我们来总结一下：

val newRDD = ReliableCheckpointRDD.writeRDDToCheckpointDirectory(rdd, cpDir)

1、把之前originalRDD信息复制到newRDD中

2、同时把这个RDD进行存储到HDFS，目录在我们设置的setCheckpointDir(directory: String)中的directory

3、同时我们会对originalRDD的hadoopConfiguration信息进行广播

4、我们在runJob的时候会再次计算我们这个RDD，也就是说我们可以对它进行缓存，這样可以得到更好的优化。

现在我们再来看看docheckpoint的最后一个环节。

if (rdd.conf.getBoolean("spark.cleaner.referenceTracking.cleanCheckpoints", false)) {
      rdd.context.cleaner.foreach { cleaner =>
        cleaner.registerRDDCheckpointDataForCleanup(newRDD, rdd.id)
      }
    }

发现我docheckpoint会对originalRDD进行清洗，也就是说，之前的计算链全部清除，只留下我们的newRDD

也就是说这个newRDD称为最原始的父RDD（找源头只能找到這里）

如何获取我们的checkpointRDD，在后面的环节再说，但是可以肯定的是都会通过ReliableCheckpointRDD的

readCheckpointFile方法来从file路径里面读出已经Checkpint的数据