long的博客

SparkEnv 源码版本2.4.7 shufflemanager实例是SortShuffleManager spark数据shuffle读写是通过 SparkEnv.get.shuffleManager控制完成的 shuffleRdd的compute是通过SparkEnv.get.shuffleManager.getReader 而shuffleMaptask的write也是通过SparkEnv.get.shuffleManager.getWriter shufflerdd->getDependen

SparkEnv 源码版本2.4.7 SparkContext有一核心属性_env即SparkEnv,持有所有spark运行实例的环境对象,序列化、RPCEnv、块管理、MapOutputTracker(shuffle输出相关的记录)等等 首先看看其基本属性 class SparkEnv ( val executorId: String, private[spark] val rpcEnv: RpcEnv, val serializer: Serializer, val cl

SparkContext 源码版本2.4.7 上一篇分析完Executor的注册启动流程,接下来看rdd.action[foreach,collect,save,…]-> sc.runJob(rdd,iter=>f) 最终调用的的SparkContext的DAGScheduler的runJob runJob->submitJob-> eventProcessLoop.post(JobSubmitted)->eventQueue eventQueue->doOnRecei

SparkContext 源码版本2.4.7 书接上回 Master通过scheule水平划分完Executor资源后,执行launchExecutor操作,通过wokerEndpoint引用向worker发送LaunchExecutor指令,顺序执行向driver发送 ExecutorAdded 消息 private def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc): Unit = { logInfo("Launching

SparkContext 源码版本2.4.7 首先我们要知道当RDD真正遇到action算子时,任务才会被触发. /** * Applies a function f to all elements of this RDD. */ def foreach(f: T => Unit): Unit = withScope { val cleanF = sc.clean(f) sc.runJob(this, (iter: Iterator[T]) => iter.f

spark-submit 源码版本2.4.7 spark-submit exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@" 主类: org.apache.spark.deploy.SparkSubmit spark-class 会启动 Moudle->launcher org.apache.spark.launcher.Main的jvm进程,构建一些JVM相关启动环境、参数 从main方法跟踪发

Worker 源码版本2.4.7 start-slave.sh 启动主类: org.apache.spark.deploy.worker.Worker 查看main startRpcEnvAndEndpoint 启动rpcEnv基础通信,与master类似,最终异步调用OnStart def main(argStrings: Array[String]) { Thread.setDefaultUncaughtExceptionHandler(new SparkUncaughtExceptionH

NettyRpcEnv 源码版本2.4.7 需要关注的是env里面几个关键的属性和方法如图: 1 dispatcher 分发 消息分发程序，负责将RPC消息路由到适当的端点 dispatcher的成员内部类: EndpointData实体 private class EndpointData( val name: String, val endpoint: RpcEndpoint, val ref: NettyRpcEndpointRef) { val in

Master 源码版本2.4.7 start-master.sh 启动主类: org.apache.spark.deploy.master.Master Master是一个伴生类 首先找到main方法 def main(argStrings: Array[String]) { Thread.setDefaultUncaughtExceptionHandler(new SparkUncaughtExceptionHandler( exitOnUncaughtException = fal

Spark三大误解