DT大数据梦工厂Spark定制班笔记(011)

最新推荐文章于 2017-03-17 18:10:58 发布

RY076

最新推荐文章于 2017-03-17 18:10:58 发布

阅读量459

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/RY076/article/details/51493254

本文详细介绍了Spark Streaming中ReceiverTracker的核心功能，包括启动、停止Receiver，限流及容错机制等。深入探讨了ReceiverTracker如何通过RPC机制在Executor上启动Receiver，并实现了Receiver的监控与错误汇报。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Spark Streaming源码解读之Driver中的ReceiverTracker架构设计以及具体实现彻底研究

ReceiverTracker主要的功能：

1. 在Executor上启动Receivers。

2. 停止Receivers 。

3. 更新Receiver接收数据的速率(可以实现限流)

4. 接收Receivers的运行状态，只要Receiver停止运行，就重新启动Receiver。也就是Receiver的容错功能。

5. 接受Receiver的注册。

6. 借助ReceivedBlockTracker来管理Receiver接收数据的元数据。

7. 汇报Receiver发送过来的错误信息

启动receiver

ReceiverTracker的start方法中，实例化了ReceiverTrackerEndpoint，并且在Executor上启动Receivers：

ReceiverTracker.scala(149-161)

def start(): Unit = synchronized {
if (isTrackerStarted){
    throw new SparkException("ReceiverTracker already started")
}

if (!receiverInputStreams.isEmpty) {
    endpoint = ssc.env.rpcEnv.setupEndpoint(
      "ReceiverTracker", new ReceiverTrackerEndpoint(ssc.env.rpcEnv))
    if (!skipReceiverLaunch)launchReceivers()
    logInfo("ReceiverTracker started")
    trackerState = Started
}
}

ReceiverTracker.scala(413-424)

private def launchReceivers(): Unit = {
val receivers = receiverInputStreams.map(nis => {
    val rcvr= nis.getReceiver()
    rcvr.setReceiverId(nis.id)
    rcvr
})

runDummySparkJob()

logInfo("Starting " + receivers.length +" receivers")
endpoint.send(StartAllReceivers(receivers))
}

一直追踪函数调用过程receiver被封装为RDD （详见第5讲 ReceiverTracker.scala 583-589）

val receiverRDD: RDD[Receiver[_]] =
if (scheduledLocations.isEmpty){
    ssc.sc.makeRDD(Seq(receiver),1)
} else {
    val preferredLocations= scheduledLocations.map(_.toString).distinct
    ssc.sc.makeRDD(Seq(receiver -> preferredLocations))
}