Spark内核解析2

最新推荐文章于 2023-10-05 12:25:32 发布
原创 最新推荐文章于 2023-10-05 12:25:32 发布 · 530 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文深入探讨了Spark集群的启动流程，详细解析了Master和Worker节点如何通过RPC机制进行通信，包括节点间的注册、心跳监测及故障恢复过程。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

Spark的交互流程- - -节点启动：

Master启动时首先创一个RpcEnv对象，负责管理所有通信逻辑
Master通过RpcEnv对象创建一个Endpoint，Master就是一个Endpoint，Worker可以与其进行通信
Worker启动时也是创一个RpcEnv对象
Worker通过RpcEnv对象创建一个Endpoint
Worker通过RpcEnv对，建立到Master的连接，获取到一个RpcEndpointRef对象，通过该对象可以与Master通信
Worker向Master注册，注册内容包括主机名、端口、CPU Core数量、内存数量
Master接收到Worker的注册，将注册信息维护在内存中的Table中，其中还包含了一个到Worker的RpcEndpointRef对象引用
Master回复Worker已经接收到注册，告知Worker已经注册成功
此时如果有用户提交Spark程序，Master需要协调启动Driver；而Worker端收到成功注册响应后，开始周期性向Master发送心跳

Mster

Master的main方法启动创建了一个startRpcEnvAndEndpoint

1
2
3
4
5
6
7
8
9
10
11
def main(argStrings: Array[String]) {
    // 1、初始化log对象
    Utils.initDaemon(log)
    // 2、加载SparkConf
    val conf = new SparkConf
    // 3、解析Master启动参数
    val args = new MasterArguments(argStrings, conf)  
    // 4、启动RPC框架端点
    val (rpcEnv, _, _) = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, conf)
    rpcEnv.awaitTermination()
  }

我们点击startRpcEnvAndEndpoint查看中创建了一个Master

1
2
3
4
5
6
7
8
9
10
11
12
13
  def startRpcEnvAndEndpoint(
      host: String,  
      port: Int,
      webUiPort: Int,
      conf: SparkConf): (RpcEnv, Int, Option[Int]) = {
    val securityMgr = new SecurityManager(conf)
    val rpcEnv = RpcEnv.create(SYSTEM_NAME, host, port, conf, securityMgr)
    val masterEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME,
      new Master(rpcEnv, rpcEnv.address, webUiPort, securityMgr, conf))  //创建一个Master
    val portsResponse = masterEndpoint.askWithRetry[BoundPortsResponse](BoundPortsRequest)
    (rpcEnv, portsResponse.webUIPort, portsResponse.restPort)
  }
}

在Master中我们重点关注一下onStart()方法,绑定WebUI,和定期心跳检测

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
override def onStart(): Unit = {
    logInfo("Starting Spark master at " + masterUrl)
    logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
    webUi = new MasterWebUI(this, webUiPort)
    webUi.bind()
    masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort
    if (reverseProxy) {
      masterWebUiUrl = conf.get("spark.ui.reverseProxyUrl", masterWebUiUrl)
      logInfo(s"Spark Master is acting as a reverse proxy. Master, Workers and " +
       s"Applications UIs are available at $masterWebUiUrl")
    }
    checkForWorkerTimeOutTask = forwardMessageThread.scheduleAtFixedRate(new Runnable {
      override def run(): Unit = Utils.tryLogNonFatalError {
        self.send(CheckForWorkerTimeOut)  //周期性的自己给自己发送一条消息
      }
    }, 0, WORKER_TIMEOUT_MS, TimeUnit.MILLISECONDS)  //WORKER_TIMEOUT_MS为60*1000 也就是一分钟超时时间为1分钟

    if (restServerEnabled) {
      val port = conf.getInt("spark.master.rest.port", 6066)
      restServer = Some(new StandaloneRestServer(address.host, port, conf, self, masterUrl))
    }
    restServerBoundPort = restServer.map(_.start())

    masterMetricsSystem.registerSource(masterSource)
    masterMetricsSystem.start()
    applicationMetricsSystem.start()
    // Attach the master and app metrics servlet handler to the web ui after the metrics systems are
    // started.
    masterMetricsSystem.getServletHandlers.foreach(webUi.attachHandler)
    applicationMetricsSystem.getServletHandlers.foreach(webUi.attachHandler)
 
    val serializer = new JavaSerializer(conf)
    val (persistenceEngine_, leaderElectionAgent_) = RECOVERY_MODE match { //RECOVERY_MODE是否设置
      case "ZOOKEEPER" =>
        logInfo("Persisting recovery state to ZooKeeper")
        val zkFactory =
          new ZooKeeperRecoveryModeFactory(conf, serializer)
        (zkFactory.createPersistenceEngine(), zkFactory.createLeaderElectionAgent(this))
      case "FILESYSTEM" =>
        val fsFactory =
          new FileSystemRecoveryModeFactory(conf, serializer)
        (fsFactory.createPersistenceEngine(), fsFactory.createLeaderElectionAgent(this))
      case "CUSTOM" =>
        val clazz = Utils.classForName(conf.get("spark.deploy.recoveryMode.factory"))
        val factory = clazz.getConstructor(classOf[SparkConf], classOf[Serializer])
          .newInstance(conf, serializer)
          .asInstanceOf[StandaloneRecoveryModeFactory]
        (factory.createPersistenceEngine(), factory.createLeaderElectionAgent(this))
      case _ =>
        (new BlackHolePersistenceEngine(), new MonarchyLeaderAgent(this))
    }
    persistenceEngine = persistenceEngine_
    leaderElectionAgent = leaderElectionAgent_
  }

我们在查看一下receive 这些只是受到消息,然而并没有恢复

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
override def receive: PartialFunction[Any, Unit] = {
    case ElectedLeader => // Leader选举,用在SparkHA中
      val (storedApps, storedDrivers, storedWorkers) = persistenceEngine.readPersistedData(rpcEnv)
      state = if (storedApps.isEmpty && storedDrivers.isEmpty && storedWorkers.isEmpty) {
        RecoveryState.ALIVE
      } else {
        RecoveryState.RECOVERING
      }
      logInfo("I have been elected leader! New state: " + state)
      if (state == RecoveryState.RECOVERING) {
        beginRecovery(storedApps, storedDrivers, storedWorkers)
        recoveryCompletionTask = forwardMessageThread.schedule(new Runnable {
          override def run(): Unit = Utils.tryLogNonFatalError {
            self.send(CompleteRecovery)
          }
        }, WORKER_TIMEOUT_MS, TimeUnit.MILLISECONDS)
      }

    case CompleteRecovery => completeRecovery() //成功地恢复

    case RevokedLeadership => //
      logError("Leadership has been revoked -- master shutting down.")
      System.exit(0)
	//注册应用
    case RegisterApplication(description, driver) =>
      // TODO Prevent repeated registrations from some driver
      if (state == RecoveryState.STANDBY) {
        // ignore, don't send response
      } else {
        logInfo("Registering app " + description.name)
        val app = createApplication(description, driver)
        registerApplication(app)
        logInfo("Registered app " + description.name + " with ID " + app.id)
        persistenceEngine.addApplication(app)
        driver.send(RegisteredApplication(app.id, self))
        // 启动分配Executor
        schedule()
      }
    // 检测 Executor
    case ExecutorStateChanged(appId, execId, state, message, exitStatus) =>
      val execOption = idToApp.get(appId).flatMap(app => app.executors.get(execId))
      execOption match {
        case Some(exec) =>
          val appInfo = idToApp(appId)
          val oldState = exec.state
          exec.state = state

          if (state == ExecutorState.RUNNING) {
            assert(oldState == ExecutorState.LAUNCHING,
              s"executor $execId state transfer from $oldState to RUNNING is illegal")
            appInfo.resetRetryCount()
          }

          exec.application.driver.send(ExecutorUpdated(execId, state, message, exitStatus, false))

          if (ExecutorState.isFinished(state)) {
            // Remove this executor from the worker and app
            logInfo(s"Removing executor ${exec.fullId} because it is $state")
            // If an application has already finished, preserve its
            // state to display its information properly on the UI
            if (!appInfo.isFinished) {
              appInfo.removeExecutor(exec)
            }
            exec.worker.removeExecutor(exec)

            val normalExit = exitStatus == Some(0)
            // Only retry certain number of times so we don't go into an infinite loop.
            // Important note: this code path is not exercised by tests, so be very careful when
            // changing this `if` condition.
            if (!normalExit
                && appInfo.incrementRetryCount() >= MAX_EXECUTOR_RETRIES
                && MAX_EXECUTOR_RETRIES >= 0) { // < 0 disables this application-killing path
              val execs = appInfo.executors.values
              if (!execs.exists(_.state == ExecutorState.RUNNING)) {
                logError(s"Application ${appInfo.desc.name} with ID ${appInfo.id} failed " +
                  s"${appInfo.retryCount} times; removing it")
                removeApplication(appInfo, ApplicationState.FAILED)
              }
            }
          }
          schedule()
        case None =>
          logWarning(s"Got status update for unknown executor $appId/$execId")
      }

    case DriverStateChanged(driverId, state, exception) =>
      state match {
        case DriverState.ERROR | DriverState.FINISHED | DriverState.KILLED | DriverState.FAILED =>
          removeDriver(driverId, state, exception)
        case _ =>
          throw new Exception(s"Received unexpected state update for driver $driverId: $state")
      }
	//心跳机制
    case Heartbeat(workerId, worker) =>
      idToWorker.get(workerId) match {
        case Some(workerInfo) =>
          workerInfo.lastHeartbeat = System.currentTimeMillis()
        case None =>
          if (workers.map(_.id).contains(workerId)) {
            logWarning(s"Got heartbeat from unregistered worker $workerId." +
              " Asking it to re-register.")
            worker.send(ReconnectWorker(masterUrl))
          } else {
            logWarning(s"Got heartbeat from unregistered worker $workerId." +
              " This worker was never registered, so ignoring the heartbeat.")
          }
      }

    case MasterChangeAcknowledged(appId) =>
      idToApp.get(appId) match {
        case Some(app) =>
          logInfo("Application has been re-registered: " + appId)
          app.state = ApplicationState.WAITING
        case None =>
          logWarning("Master change ack from unknown app: " + appId)
      }

      if (canCompleteRecovery) { completeRecovery() }
	//反馈Worker的运行状态
    case WorkerSchedulerStateResponse(workerId, executors, driverIds) =>
      idToWorker.get(workerId) match {
        case Some(worker) =>
          logInfo("Worker has been re-registered: " + workerId)
          worker.state = WorkerState.ALIVE

          val validExecutors = executors.filter(exec => idToApp.get(exec.appId).isDefined)
          for (exec <- validExecutors) {
            val app = idToApp.get(exec.appId).get
            val execInfo = app.addExecutor(worker, exec.cores, Some(exec.execId))
            worker.addExecutor(execInfo)
            execInfo.copyState(exec)
          }

          for (driverId <- driverIds) {
            drivers.find(_.id == driverId).foreach { driver =>
              driver.worker = Some(worker)
              driver.state = DriverState.RUNNING
              worker.drivers(driverId) = driver
            }
          }
        case None =>
          logWarning("Scheduler state from unknown worker: " + workerId)
      }

      if (canCompleteRecovery) { completeRecovery() }

    case WorkerLatestState(workerId, executors, driverIds) =>
      idToWorker.get(workerId) match {
        case Some(worker) =>
          for (exec <- executors) {
            val executorMatches = worker.executors.exists {
              case (_, e) => e.application.id == exec.appId && e.id == exec.execId
            }
            if (!executorMatches) {
              // master doesn't recognize this executor. So just tell worker to kill it.
              worker.endpoint.send(KillExecutor(masterUrl, exec.appId, exec.execId))
            }
          }

          for (driverId <- driverIds) {
            val driverMatches = worker.drivers.exists { case (id, _) => id == driverId }
            if (!driverMatches) {
              // master doesn't recognize this driver. So just tell worker to kill it.
              worker.endpoint.send(KillDriver(driverId))
            }
          }
        case None =>
          logWarning("Worker state from unknown worker: " + workerId)
      }

    case UnregisterApplication(applicationId) =>
      logInfo(s"Received unregister request from application $applicationId")
      idToApp.get(applicationId).foreach(finishApplication)
	//给积极扫那个一个消息,检测自己是否挂掉
    case CheckForWorkerTimeOut =>
      timeOutDeadWorkers()

  }

注意:上述都是没有回复的,我们需要查看一下receiveAndReply

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
  case RequestSubmitDriver(description) =>
    if (state != RecoveryState.ALIVE) {
      val msg = s"${Utils.BACKUP_STANDALONE_MASTER_PREFIX}: $state. " +
        "Can only accept driver submissions in ALIVE state."
      context.reply(SubmitDriverResponse(self, false, None, msg))
    } else {
      logInfo("Driver submitted " + description.command.mainClass)
      val driver = createDriver(description)
      persistenceEngine.addDriver(driver)
      waitingDrivers += driver
      drivers.add(driver)
      schedule()

      // TODO: It might be good to instead have the submission client poll the master to determine
      //       the current status of the driver. For now it's simply "fire and forget".

      context.reply(SubmitDriverResponse(self, true, Some(driver.id),
        s"Driver successfully submitted as ${driver.id}"))
    }

  case RequestKillDriver(driverId) =>
    if (state != RecoveryState.ALIVE) {
      val msg = s"${Utils.BACKUP_STANDALONE_MASTER_PREFIX}: $state. " +
        s"Can only kill drivers in ALIVE state."
      context.reply(KillDriverResponse(self, driverId, success = false, msg))
    } else {
      logInfo("Asked to kill driver " + driverId)
      val driver = drivers.find(_.id == driverId)
      driver match {
        case Some(d) =>
          if (waitingDrivers.contains(d)) {
            waitingDrivers -= d
            self.send(DriverStateChanged(driverId, DriverState.KILLED, None))
          } else {
            // We just notify the worker to kill the driver here. The final bookkeeping occurs
            // on the return path when the worker submits a state change back to the master
            // to notify it that the driver was successfully killed.
            d.worker.foreach { w =>
              w.endpoint.send(KillDriver(driverId))
            }
          }
          // TODO: It would be nice for this to be a synchronous response
          val msg = s"Kill request for $driverId submitted"
          logInfo(msg)
          //回复KillDriverResponse
          context.reply(KillDriverResponse(self, driverId, success = true, msg))
        case None =>
          val msg = s"Driver $driverId has already finished or does not exist"
          logWarning(msg)
          context.reply(KillDriverResponse(self, driverId, success = false, msg))
      }
    }

  case RequestDriverStatus(driverId) =>
    if (state != RecoveryState.ALIVE) {
      val msg = s"${Utils.BACKUP_STANDALONE_MASTER_PREFIX}: $state. " +
        "Can only request driver status in ALIVE state."
      context.reply(
        DriverStatusResponse(found = false, None, None, None, Some(new Exception(msg))))
    } else {
      (drivers ++ completedDrivers).find(_.id == driverId) match {
        case Some(driver) =>
          context.reply(DriverStatusResponse(found = true, Some(driver.state),
            driver.worker.map(_.id), driver.worker.map(_.hostPort), driver.exception))
        case None =>
          context.reply(DriverStatusResponse(found = false, None, None, None, None))
      }
    }

  case RequestMasterState =>
    context.reply(MasterStateResponse(
      address.host, address.port, restServerBoundPort,
      workers.toArray, apps.toArray, completedApps.toArray,
      drivers.toArray, completedDrivers.toArray, state))

  case BoundPortsRequest =>
    context.reply(BoundPortsResponse(address.port, webUi.boundPort, restServerBoundPort))

  case RequestExecutors(appId, requestedTotal) =>
    context.reply(handleRequestExecutors(appId, requestedTotal))

  case KillExecutors(appId, executorIds) =>
    val formattedExecutorIds = formatExecutorIds(executorIds)
    context.reply(handleKillExecutors(appId, formattedExecutorIds))
}

Worker

1
2
3
4
5
6
7
8
9
10
11
private[deploy] class Worker(
    override val rpcEnv: RpcEnv,
    webUiPort: Int,
    cores: Int,
    memory: Int,
    masterRpcAddresses: Array[RpcAddress],
    endpointName: String,
    workDirPath: String = null,
    val conf: SparkConf,
    val securityMgr: SecurityManager)
  extends ThreadSafeRpcEndpoint with Logging { //Worker继承了ThreadSafeRpcEndpoint,它也是一个通信端点

我们查看一下Worker的main方法

1
2
3
4
5
6
7
8
def main(argStrings: Array[String]) {
  Utils.initDaemon(log)
  val conf = new SparkConf  //创建SparkConf
  val args = new WorkerArguments(argStrings, conf) //解析参数
  val rpcEnv = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, args.cores, //创建rpcEnv
    args.memory, args.masters, args.workDir, conf = conf)
  rpcEnv.awaitTermination()
}

我们在查看一下调用的startRpcEnvAndEndpoint

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def startRpcEnvAndEndpoint(
     host: String,
     port: Int,
     webUiPort: Int,
     cores: Int,
     memory: Int,
     masterUrls: Array[String],
     workDir: String,
     workerNumber: Option[Int] = None,
     conf: SparkConf = new SparkConf): RpcEnv = {

   // The LocalSparkCluster runs multiple local sparkWorkerX RPC Environments
   val systemName = SYSTEM_NAME + workerNumber.map(_.toString).getOrElse("")
   val securityMgr = new SecurityManager(conf)
   val rpcEnv = RpcEnv.create(systemName, host, port, conf, securityMgr)
   val masterAddresses = masterUrls.map(RpcAddress.fromSparkURL(_))
   rpcEnv.setupEndpoint(ENDPOINT_NAME, new Worker(rpcEnv, webUiPort, cores, memory, //生成Worker会有很多属性.
     masterAddresses, ENDPOINT_NAME, workDir, conf, securityMgr))
   rpcEnv
 }

我们在查看一下onStart方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
override def onStart() {
    assert(!registered)
    logInfo("Starting Spark worker %s:%d with %d cores, %s RAM".format(
      host, port, cores, Utils.megabytesToString(memory)))
    logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
    logInfo("Spark home: " + sparkHome)  //打印日志
    createWorkDir()
    shuffleService.startIfEnabled()
    webUi = new WorkerWebUI(this, workDir, webUiPort) //创建WorkerWebUI
    webUi.bind()

    workerWebUiUrl = s"http://$publicAddress:${webUi.boundPort}" //8081端
    registerWithMaster() //注册点击查看
	//测量系统启动
    metricsSystem.registerSource(workerSource)
    metricsSystem.start()
    // Attach the worker metrics servlet handler to the web ui after the metrics system is started.
    metricsSystem.getServletHandlers.foreach(webUi.attachHandler)
  }

我们查看一下registerWithMaster

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
private def registerWithMaster() {
   // onDisconnected may be triggered multiple times, so don't attempt registration
   // if there are outstanding registration attempts scheduled.
   registrationRetryTimer match {  //重试机制 
     case None =>
       registered = false //未注册的
       registerMasterFutures = tryRegisterAllMasters()
       connectionAttemptCount = 0
       registrationRetryTimer = Some(forwordMessageScheduler.scheduleAtFixedRate( //
         new Runnable {
           override def run(): Unit = Utils.tryLogNonFatalError {
             Option(self).foreach(_.send(ReregisterWithMaster)) //发送
           }
         },
         INITIAL_REGISTRATION_RETRY_INTERVAL_SECONDS,
         INITIAL_REGISTRATION_RETRY_INTERVAL_SECONDS,
         TimeUnit.SECONDS))
     case Some(_) =>
       logInfo("Not spawning another attempt to register with the master, since there is an" +
         " attempt scheduled already.")
   }
 }

点击ReregisterWithMaster

1
2
case ReregisterWithMaster =>
  reregisterWithMaster()  // 调用方法
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
private def reregisterWithMaster(): Unit = {
    Utils.tryOrExit {
      connectionAttemptCount += 1
      if (registered) {
        cancelLastRegistrationRetry()
      } else if (connectionAttemptCount <= TOTAL_REGISTRATION_RETRIES) {
        logInfo(s"Retrying connection to master (attempt # $connectionAttemptCount)")
      //
        master match {
          case Some(masterRef) => //获得masterRef
            // registered == false && master != None means we lost the connection to master, so
            // masterRef cannot be used and we need to recreate it again. Note: we must not set
            // master to None due to the above comments.
            if (registerMasterFutures != null) {
              registerMasterFutures.foreach(_.cancel(true))
            }
            val masterAddress = masterRef.address
            registerMasterFutures = Array(registerMasterThreadPool.submit(new Runnable {
              override def run(): Unit = {
                try {
                  logInfo("Connecting to master " + masterAddress + "...")
                  val masterEndpoint = rpcEnv.setupEndpointRef(masterAddress, Master.ENDPOINT_NAME)
                  registerWithMaster(masterEndpoint) //重试机制
                } catch {
                  case ie: InterruptedException => // Cancelled
                  case NonFatal(e) => logWarning(s"Failed to connect to master $masterAddress", e)
                }
              }
            }))
          case None =>
            if (registerMasterFutures != null) {
              registerMasterFutures.foreach(_.cancel(true))
            }
            // We are retrying the initial registration
            registerMasterFutures = tryRegisterAllMasters()
        }
        // We have exceeded the initial registration retry threshold
        // All retries from now on should use a higher interval
        if (connectionAttemptCount == INITIAL_REGISTRATION_RETRIES) {
          registrationRetryTimer.foreach(_.cancel(true))
          registrationRetryTimer = Some(
            forwordMessageScheduler.scheduleAtFixedRate(new Runnable {
              override def run(): Unit = Utils.tryLogNonFatalError {
                self.send(ReregisterWithMaster) //重试机制
              }
            }, PROLONGED_REGISTRATION_RETRY_INTERVAL_SECONDS,
              PROLONGED_REGISTRATION_RETRY_INTERVAL_SECONDS,
              TimeUnit.SECONDS))
        }
      } else {
        logError("All masters are unresponsive! Giving up.")
        System.exit(1)
      }
    }
  }

Master

在Master中我们查看一下receiveAndReply

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
  case RegisterWorker(
      id, workerHost, workerPort, workerRef, cores, memory, workerWebUiUrl) =>
    logInfo("Registering worker %s:%d with %d cores, %s RAM".format( //正在注册
      workerHost, workerPort, cores, Utils.megabytesToString(memory)))
    if (state == RecoveryState.STANDBY) { //如果state不是主节点
      context.reply(MasterInStandby) //回复MasterInStandby
    } else if (idToWorker.contains(id)) {
      context.reply(RegisterWorkerFailed("Duplicate worker ID"))
    } else {
      val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory, //否则new 一个WorkerInfo 将接收到的消息传入WorkerInfo 注册信息包含有主机名端口 CPU数量 内存数量
        workerRef, workerWebUiUrl)
      if (registerWorker(worker)) { //点击查看registerWorker
        persistenceEngine.addWorker(worker)
        context.reply(RegisteredWorker(self, masterWebUiUrl)) //成功放入
        schedule()
      } else { //放入失败
        val workerAddress = worker.endpoint.address
        logWarning("Worker registration failed. Attempted to re-register worker at same " +
          "address: " + workerAddress)
        context.reply(RegisterWorkerFailed("Attempted to re-register worker at same address: "
          + workerAddress))
      }
    }

点击查看registerWorker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
private def registerWorker(worker: WorkerInfo): Boolean = {
  // There may be one or more refs to dead workers on this same node (w/ different ID's),
  // remove them.
  workers.filter { w =>
    (w.host == worker.host && w.port == worker.port) && (w.state == WorkerState.DEAD)
  }.foreach { w =>
    workers -= w
  }

  val workerAddress = worker.endpoint.address
  if (addressToWorker.contains(workerAddress)) {
    val oldWorker = addressToWorker(workerAddress)
    if (oldWorker.state == WorkerState.UNKNOWN) {
      // A worker registering from UNKNOWN implies that the worker was restarted during recovery.
      // The old worker must thus be dead, so we will remove it and accept the new worker.
      removeWorker(oldWorker)
    } else {
      logInfo("Attempted to re-register worker at same address: " + workerAddress)
      return false
    }
  }

  workers += worker  //这是一个HashSet[WorkerInfo]   维护的是Worker的注册信息
  idToWorker(worker.id) = worker
  addressToWorker(workerAddress) = worker
  if (reverseProxy) {
     webUi.addProxyTargets(worker.id, worker.webUiAddress)
  }
  true
}

在Master上如果我们成功了则调用context.reply(RegisteredWorker(self, masterWebUiUrl)) 告知Worker注册成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
private def handleRegisterResponse(msg: RegisterWorkerResponse): Unit = synchronized {
  msg match {
    case RegisteredWorker(masterRef, masterWebUiUrl) =>  //注册成功打印日志
      logInfo("Successfully registered with master " +   masterRef.address.toSparkURL)
      registered = true  //改变registered的状态
      changeMaster(masterRef, masterWebUiUrl)
      forwordMessageScheduler.scheduleAtFixedRate(new Runnable {
        override def run(): Unit = Utils.tryLogNonFatalError {
          self.send(SendHeartbeat) //发送心跳
        }
      }, 0, HEARTBEAT_MILLIS, TimeUnit.MILLISECONDS)
      if (CLEANUP_ENABLED) {
        logInfo(
          s"Worker cleanup enabled; old application directories will be deleted in: $workDir")
        forwordMessageScheduler.scheduleAtFixedRate(new Runnable {
          override def run(): Unit = Utils.tryLogNonFatalError {
            self.send(WorkDirCleanup)
          }
        }, CLEANUP_INTERVAL_MILLIS, CLEANUP_INTERVAL_MILLIS, TimeUnit.MILLISECONDS)
      }

      val execs = executors.values.map { e =>
        new ExecutorDescription(e.appId, e.execId, e.cores, e.state)
      }
      masterRef.send(WorkerLatestState(workerId, execs.toList, drivers.keys.toSeq))

    case RegisterWorkerFailed(message) =>
      if (!registered) {
        logError("Worker registration failed: " + message)
        System.exit(1)
      }

    case MasterInStandby =>
      // Ignore. Master not yet ready.
  }
}
1
2
3
4
5
6
7

override def receive: PartialFunction[Any, Unit] = synchronized {
  case SendHeartbeat => 					//发送的消息为workerId 和 它自身
    if (connected) { sendToMaster(Heartbeat(workerId, self)) } //sendToMaster给Master发送心跳

 ...
}

点击进入sendToMaster

1
2
3
4
5
6
7
8
private def sendToMaster(message: Any): Unit = {
  master match { 
    case Some(masterRef) => masterRef.send(message) //把消息发送给了Master
    case None =>
      logWarning(
        s"Dropping $message because the connection to master has not yet been established")
  }
}

Master中收到发送的消息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
case Heartbeat(workerId, worker) =>
  idToWorker.get(workerId) match { //Worker中是否有
    case Some(workerInfo) =>  //如果有
      workerInfo.lastHeartbeat = System.currentTimeMillis() //更新lastHeartbeat的时间
    case None => //如果没有发现
      if (workers.map(_.id).contains(workerId)) {
        logWarning(s"Got heartbeat from unregistered worker $workerId." +
          " Asking it to re-register.")
        worker.send(ReconnectWorker(masterUrl))//需要重新链接masterUrl
      } else { 
        logWarning(s"Got heartbeat from unregistered worker $workerId." +
          " This worker was never registered, so ignoring the heartbeat.")
      }
  }

我们回过头看一下Master中的Onstart调用的self.send(CheckForWorkerTimeOut) 

1
2
case CheckForWorkerTimeOut =>
  timeOutDeadWorkers() //调用
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
private def timeOutDeadWorkers() {
   // Copy the workers into an array so we don't modify the hashset while iterating through it
   val currentTime = System.currentTimeMillis()  //获取当前时间
   val toRemove = workers.filter(_.lastHeartbeat < currentTime - WORKER_TIMEOUT_MS).toArray //
   for (worker <- toRemove) { //如果被filter出来
     if (worker.state != WorkerState.DEAD) {
       logWarning("Removing %s because we got no heartbeat in %d seconds".format(
         worker.id, WORKER_TIMEOUT_MS / 1000))
       removeWorker(worker) //worker 被移除
     } else {
       if (worker.lastHeartbeat < currentTime - ((REAPER_ITERATIONS + 1) * WORKER_TIMEOUT_MS)) {
         workers -= worker // we've seen this DEAD worker in the UI, etc. for long enough; cull it
       }
     }
   }
 }
清风笑丶
关注
大数据技术之Spark内核
qq_56870570的博客
11-17 974
第 1 章 Spark 内核概述 Spark 内核泛指 Spark 的核心运行机制，包括 Spark 核心组件的运行机制、Spark 任务调度机制、Spark 内存管理机制、Spark 核心功能的运行原理等，熟练掌握 Spark 内核原理，能够帮助我们更好地完成 Spark 代码设计，并能够帮助我们准确锁定项目运行过程中出现的问题的症结所在。 1.1 Spark 核心组件回顾 1.1.1 Driver Spark 驱动器节点，用于执行 Spark 任务中的 main 方法，负责实际..
Spark知识体系完整解读
程序之道的博客
01-18 476
Spark简介 Spark是整个BDAS的核心组件，是一个大数据分布式编程框架，不仅实现了MapReduce的算子map 函数和reduce函数及计算模型，还提供更为丰富的算子，如filter、join、groupByKey等。是一个用来实现快速而同用的集群计算的平台。 Spark将分布式数据抽象为弹性分布式数据集(RDD)，实现了应用任务调度、RPC、序列化和压缩，并为运行在其上的上层组件提...
Spark内核
最新发布
weixin_39458365的博客
10-05 102
不同的管理模式下，这三部分占用的空间大小各不相同。这里还是有4个Tasks，数据类别还是分成3种类型，因为Hash算法会根据你的 Key 进行分类，在同一个进程中，无论是有多少过Task，都会把同样的Key放在同一个Buffer里，然后把Buffer中的数据写入以Core数量为单位的本地文件中，(一个Core只有一种类型的Key的数据)，每1个Task所在的进程中，分别写入共同进程中的3份本地文件，这里有4个Mapper Tasks，所以总共输出是 2个Cores x 3个分类文件 = 6个本地小文件。
Spark内核解析.docx
12-13
《Spark内核解析》 Spark作为大数据处理领域的明星框架，其内核设计的高效和灵活性使其在处理大规模数据时展现出卓越性能。Spark的核心运行机制主要包括Spark核心组件、任务调度、内存管理和核心功能的运行原理。...
Spark内核机制解析及性能调优教程（含资料）
06-18
由于Spark基于内存计算的特性，集群的任何资源都可以成为Spark程序的瓶颈:CPU，网络带宽，或者内存。通常，如果内存容得下数据，瓶颈会是网络带宽。不过有时你同样需要做些优化，例如将RDD以序列化到磁盘，来降低...
Spark技术内幕深入解析Spark内核架构设计与实现原理
12-04
《Spark技术内幕深入解析Spark内核架构设计与实现原理》这本书深入探讨了Apache Spark这一分布式计算框架的核心架构和实现机制，旨在帮助读者全面理解Spark的工作原理，并能够有效地利用其进行大数据处理。...
Spark内核机制解析及性能调优
05-23
本资料《Spark内核机制解析及性能调优》深入探讨了Spark的核心原理以及如何对其进行优化，以提升大数据处理的效率。 首先，Spark的核心组件包括Driver、Executor、RDD（弹性分布式数据集）和DAG Scheduler。Driver...
【Spark内核篇04】SparkShuffle解析1
08-04
2. **SortShuffle**: 在SortShuffle中，数据首先被写入内存数据结构，如Map（对于reduceByKey）或ArrayList（对于join），并进行局部排序。当达到一定阈值或任务结束时，内存中的数据被写入磁盘。在溢写磁盘前，数据...
Spark 内核
The Zen of Data Analysis
03-26 258
Spark内核 Spark内核泛指Spark的核心运行机制，包括Spark核心组件的运行机制、Spark任务调度机制、Spark内存管理机制、Spark核心功能的运行原理等。 Spark核心组件 Driver Spark驱动器节点，用于执行Spark任务中的main方法： 将用户程序转化为作业（job）； 在Executor之间调度任务(task)； 跟踪Executor的执行情况； 通过UI展...
Spark之Spark内核
华辰烟雨的博客
08-15 695
Spark 内核泛指 Spark 的核心运行机制，包括 Spark 核心组件的运行机制、Spark 任务调度机制、Spark 内存管理机制、Spark 核心功能的运行原理等 一、部署模式 Spark 支持多种集群管理器（Cluster Manager），分别为： 1) Standalone：独立模式，Spark 原生的简单集群管理器，自带完整的服务，可单独部署到 一个集群中，无需依赖任何其他资源管理系统，使用 Standalone 可以很方便地搭建一个 集群； ...
Spark 内核概述 部署模式
weixin_30836759的博客
07-10 234
1.Spark 内核概述 　　Spark 内核泛指 Spark 的核心运行机制，包括 Spark 核心组件的运行机制、Spark 任务调度机制、Spark 内存管理机制、Spark 核心功能的运行原理等，熟练掌握 Spark 内核原理，能够帮助我们更好地完成 Spark 代码设计，并能够帮助我们准确锁定项 目运行过程中出现的问题的症结所在。 1.1 Spark 核心...
Spark内核架构
sundujing的博客
05-10 3773
SparkContext创建：高层DAGScheduler, 底层TaskScheduler, SchedulerBackend application=driver+executor Spark的程序分成两个部分：driver和executor driver驱动executor Driver部分的源代码：SparkConf+SparkContext executor具体执行
Spark 内核解析
雾岛与鲸的博客
10-28 790
目录1、Spark内核概述1.1 Spark核心组件回顾1.1.1 Driver1.1.2 Executor1.2 Spark通用运行流程概述2、Spark通讯架构2.1 Spark通信架构概述2.2 Saprk通讯架构解析2.3 Spark集群启动3、Spark部署模式3.1 Standalone模式运行机制3.1.1 Standalone Client模式3.1.2 Standalone Cluster模式3.2 Yarn模式运行机制3.2.1 Yarn Client模式3.2.2 Yarn Clust
Spark学习(三)Spark内核
滴水穿石的博客
11-03 691
一、Spark中的一些专业术语 任务： Application：用户写的应用程序，包括Driver Program和Executor Program。 Job：一个action类算子触发执行的操作。 stage：一组任务(task)就是一个stage。 task：(thread)在集群中运行时最小的执行单元。 资源、集群： Master：资源管理的主节点。 Worker：资源管理的从节点...
Spark内核解析：大数据计算速度革命
"大数据计算平台Spark内核全面解读" Spark是一个高效、通用的大数据处理框架，起源于加州大学伯克利分校AMPLab，并于2010年开源，现已成为Apache软件基金会的重要项目。Spark的核心优势在于其快速的计算性能，尤其...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值