spark core 2.0 ContextCleaner

最新推荐文章于 2021-12-21 23:09:53 发布

原创最新推荐文章于 2021-12-21 23:09:53 发布 · 839 阅读

0 ·

CC 4.0 BY-SA版权

spark 专栏收录该内容

158 篇文章

订阅专栏

本文深入探讨了CleanupTaskWeakReference机制，这是Spark中用于跟踪和清理资源的核心组件。通过弱引用的方式，当对象变为仅弱可达时，该机制会自动将对应的CleanupTaskWeakReference添加到引用队列中，从而触发资源的清理过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

CleanupTaskWeakReference是整个cleanup的核心，这是一个普通类，有三个成员，并且继承WeakReference。referent是一个weak reference，当指向的对象被gc时，把CleanupTaskWeakReference放到队列里，详情见：java WeekReference ReferenceQueue测试

/**
 * A WeakReference associated with a CleanupTask.
 *
 * When the referent object becomes only weakly reachable, the corresponding
 * CleanupTaskWeakReference is automatically added to the given reference queue.
 */
private class CleanupTaskWeakReference(
    val task: CleanupTask,
    referent: AnyRef,
    referenceQueue: ReferenceQueue[AnyRef])
  extends WeakReference(referent, referenceQueue)

以RDD为例，当注册一个RDD的cleanup的操作时，初始化一个对象CleanRDD(rdd.id)，这样就记住了哪个rdd被回收。

 /** Register a RDD for cleanup when it is garbage collected. */
  def registerRDDForCleanup(rdd: RDD[_]): Unit = {
    registerForCleanup(rdd, CleanRDD(rdd.id))
  }

在regitsterForCleanup里，把CleanupTaskWeakReference放到referenceBuffer里，防止被回收。

 /** Register an object for cleanup. */
  private def registerForCleanup(objectForCleanup: AnyRef, task: CleanupTask): Unit = {
    referenceBuffer.add(new CleanupTaskWeakReference(task, objectForCleanup, referenceQueue))
  }

keepClean不断从referenceQueue里取回收的对象，调用相应的方法处理。reference.get返回CleanupTaskWeakReference。

  private def keepCleaning(): Unit = Utils.tryOrStopSparkContext(sc) {
    while (!stopped) {
      try {
        val reference = Option(referenceQueue.remove(ContextCleaner.REF_QUEUE_POLL_TIMEOUT))
          .map(_.asInstanceOf[CleanupTaskWeakReference])
        // Synchronize here to avoid being interrupted on stop()
        synchronized {
          reference.map(_.task).foreach { task =>
            logDebug("Got cleaning task " + task)
            referenceBuffer.remove(reference.get)
            task match {
              case CleanRDD(rddId) =>
                doCleanupRDD(rddId, blocking = blockOnCleanupTasks)
              case CleanShuffle(shuffleId) =>
                doCleanupShuffle(shuffleId, blocking = blockOnShuffleCleanupTasks)
              case CleanBroadcast(broadcastId) =>
                doCleanupBroadcast(broadcastId, blocking = blockOnCleanupTasks)
              case CleanAccum(accId) =>
                doCleanupAccum(accId, blocking = blockOnCleanupTasks)
              case CleanCheckpoint(rddId) =>
                doCleanCheckpoint(rddId)
            }
          }
        }
      } catch {
        case ie: InterruptedException if stopped => // ignore
        case e: Exception => logError("Error in cleaning thread", e)
      }
    }
  }

doCleaRDD方法里，负责具体的清理。

  /** Perform RDD cleanup. */
  def doCleanupRDD(rddId: Int, blocking: Boolean): Unit = {
    try {
      logDebug("Cleaning RDD " + rddId)
      sc.unpersistRDD(rddId, blocking)
      listeners.asScala.foreach(_.rddCleaned(rddId))
      logInfo("Cleaned RDD " + rddId)
    } catch {
      case e: Exception => logError("Error cleaning RDD " + rddId, e)
    }
  }