spark 2.1 ExecutorData and ExecutorInfo

最新推荐文章于 2020-03-29 18:01:41 发布

原创最新推荐文章于 2020-03-29 18:01:41 发布 · 744 阅读

0 ·

CC 4.0 BY-SA版权

spark 专栏收录该内容

158 篇文章

订阅专栏

本文介绍了Spark中ExecutorData类和ExecutorInfo类的作用与实现细节。ExecutorData用于存储执行器的相关信息，包括执行器端点、地址、主机名、可用核心数及总核心数等；ExecutorInfo则用于存储传递给Spark监听器的执行器信息，包含主机名、总核心数及日志URL映射等。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

ExecutorData


/**
 * Grouping of data for an executor used by CoarseGrainedSchedulerBackend.
 *
 * @param executorEndpoint The RpcEndpointRef representing this executor
 * @param executorAddress The network address of this executor
 * @param executorHost The hostname that this executor is running on
 * @param freeCores  The current number of cores available for work on the executor
 * @param totalCores The total number of cores available to the executor
 */
private[cluster] class ExecutorData(
   val executorEndpoint: RpcEndpointRef,
   val executorAddress: RpcAddress,
   override val executorHost: String,
   var freeCores: Int,
   override val totalCores: Int,
   override val logUrlMap: Map[String, String]
) extends ExecutorInfo(executorHost, totalCores, logUrlMap)

ExecutorInfo

/**
 * :: DeveloperApi ::
 * Stores information about an executor to pass from the scheduler to SparkListeners.
 */
@DeveloperApi
class ExecutorInfo(
   val executorHost: String,
   val totalCores: Int,
   val logUrlMap: Map[String, String]) {

  def canEqual(other: Any): Boolean = other.isInstanceOf[ExecutorInfo]

  override def equals(other: Any): Boolean = other match {
    case that: ExecutorInfo =>
      (that canEqual this) &&
        executorHost == that.executorHost &&
        totalCores == that.totalCores &&
        logUrlMap == that.logUrlMap
    case _ => false
  }

  override def hashCode(): Int = {
    val state = Seq(executorHost, totalCores, logUrlMap)
    state.map(_.hashCode()).foldLeft(0)((a, b) => 31 * a + b)
  }
}