Spark Streaming源码解读之JobScheduler内幕实现和深度思考

Spark Streaming JobScheduler解析

最新推荐文章于 2025-09-06 13:17:25 发布

weixin_33910460

最新推荐文章于 2025-09-06 13:17:25 发布

阅读量102

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 java runtime

原文链接：https://my.oschina.net/u/1253652/blog/675353

本文深入探讨了Apache Spark Streaming中的JobScheduler组件。从初始化到调度流程，详细分析了JobScheduler如何管理和调度任务，以及其核心组件如JobSet、JobGenerator等的工作原理。

2019独角兽企业重金招聘Python工程师标准>>>

一JobScheduler内幕实现
我们从StreamingContext start开始：

def start(): Unit = synchronized {
    state match {
      case INITIALIZED =>
        startSite.set(DStream.getCreationSite())
        StreamingContext.ACTIVATION_LOCK.synchronized {
          StreamingContext.assertNoOtherContextIsActive()
          try {
            //校验 graph是否为空 batchDuration是否为空 outputStreams是否为空即是否有输出的Dstream
            checkpoint和checkpointDuration 必须都设置（如果做checkpoint）
            validate()

            // Start the streaming scheduler in a new thread, so that thread local properties
            // like call sites and job groups can be reset without affecting those of the
            // current thread.
            //在新的线程中启动scheduler 不影响主线程
            //设置线程本地对象，类似ThreadLocal的作用
            ThreadUtils.runInNewThread("streaming-start") {
              sparkContext.setCallSite(startSite.get)
              sparkContext.clearJobGroup()
              sparkContext.setLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL, "false")
              scheduler.start()
            }
            state = StreamingContextState.ACTIVE
          } catch {
            case NonFatal(e) =>
              logError("Error starting the context, marking it as stopped", e)
              scheduler.stop(false)
              state = StreamingContextState.STOPPED
              throw e
          }
          StreamingContext.setActiveContext(this)
        }
        shutdownHookRef = ShutdownHookManager.addShutdownHook(
          StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)
        // Registering Streaming Metrics at the start of the StreamingContext
        assert(env.metricsSystem != null)
        env.metricsSystem.registerSource(streamingSource)
        uiTab.foreach(_.attach())
        logInfo("StreamingContext started")
      case ACTIVE =>
        logWarning("StreamingContext has already been started")
      case STOPPED =>
        throw new IllegalStateException("StreamingContext has already been stopped")
    }
  }

启动的时候有三种状态 INITIALIZED、ACTIVE、STOPPED，这里在主线程中启动了scheduler(scheduler.start())
我们可以看到JobScheduler实例化得时候创建了jobSets 、numConcurrentJobs（默认为1，对应一个outputstream,这里有一个性能优化点，如果有多个outputstream的话可以加大 spark.streaming.concurrentJobs这个参数，每个输出的stream不用等待）

// Use of ConcurrentHashMap.keySet later causes an odd runtime problem due to Java 7/8 diff
  // https://gist.github.com/AlainODea/1375759b8720a3f9f094
  private val jobSets: java.util.Map[Time, JobSet] = new ConcurrentHashMap[Time, JobSet]
  private val numConcurrentJobs = ssc.conf.getInt("spark.streaming.concurrentJobs", 1)
  private val jobExecutor =
    ThreadUtils.newDaemonFixedThreadPool(numConcurrentJobs, "streaming-job-executor")
  private val jobGenerator = new JobGenerator(this)
  val clock = jobGenerator.clock
  //监控Streaming作业的各种事件
  val listenerBus = new StreamingListenerBus()

  // These two are created only when scheduler starts.
  // eventLoop not being null means the scheduler has been started and not stopped
  var receiverTracker: ReceiverTracker = null
  // A tracker to track all the input stream information as well as processed record number
  var inputInfoTracker: InputInfoTracker = null

  private var eventLoop: EventLoop[JobSchedulerEvent] = null

我们继续看start方法：

def start(): Unit = synchronized {
    if (eventLoop != null) return // scheduler has already been started

    logDebug("Starting JobScheduler")
    //这里事件处理跟实际的业务处理分开，模块化，便于代码的复用和维护
    eventLoop = new EventLoop[JobSchedulerEvent]("JobScheduler") {
      override protected def onReceive(event: JobSchedulerEvent): Unit = processEvent(event)

      override protected def onError(e: Throwable): Unit = reportError("Error in job scheduler", e)
    }
    eventLoop.start()

    // attach rate controllers of input streams to receive batch completion updates
    for {
      inputDStream <- ssc.graph.getInputStreams
      rateController <- inputDStream.rateController
    } ssc.addStreamingListener(rateController)
    //启动监控线程
    listenerBus.start(ssc.sparkContext)
    receiverTracker = new ReceiverTracker(ssc)
    inputInfoTracker = new InputInfoTracker(ssc)
    receiverTracker.start()
    //jobGenerator用于生成作业
    jobGenerator.start()
    logInfo("Started JobScheduler")
  }