spark-core_29：blockManager.initialize（）=》NettyBlockTransferService.init()-NettyServer创建源码分析

上一节（spark-core_28：Executor初始化过程env.blockManager.initialize(conf.getAppId)- NettyBlockTransferService.init()源码分析）

分析了

a、NettyBlockRpcServer是用于打开的上传注册在BlockManager中的Block块

b,TansportConf:会通过它成员：ConfigProvider子类关联SparkConf,并按SparkTransportConf.fromSparkConf(conf,"shuffle", numCores)设置nettyserver和client的线程池的大小，并按fromSparkConf第二个参数设置key的变量值

3，TransportContext:包含创建{TransportServer:nettyServer}，{TransportClientFactory用来创建TransportClient}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道。实例化TransportContext将给成员赋值 conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联、rpcHandler:NettyBlockRpcServer、closeIdleConnections：false、同时给出站的编码器、入站解码器，赋具体实例

4，构造了TransportClientFactory:会将TransportContext、TransportConf,按IOMode的枚举类型，默认就是NIO, 得到NioSocketChannel.class基于IOMode枚举创建Netty的EventLoopGroup线程组、创建一个池化的ByteBuf分配器PooledByteBufAllocator给它的在成员变量

override def init(blockDataManager: BlockDataManager): Unit = {
  /** conf.getAppId: app-20180508234845-0000
    * serializer:    JavaSerializer()
    * blockDataManager : BlockManager实例
    * NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
    */
  val rpcHandler= new NettyBlockRpcServer(conf.getAppId, serializer, blockDataManager)
  var serverBootstrap:Option[TransportServerBootstrap] = None
  var clientBootstrap:Option[TransportClientBootstrap] = None
  if (authEnabled) {//默认是false，不开启认证
    serverBootstrap = Some(new SaslServerBootstrap(transportConf, securityManager))
    clientBootstrap = Some(new SaslClientBootstrap(transportConf, conf.getAppId, securityManager,
      securityManager.isSaslEncryptionEnabled()))
  }
 
  transportContext = new TransportContext(transportConf, rpcHandler)

  /** 没有开启ssl所以clientBootstrap是空的
    *
    * TransportClientFactory:这个工厂实例通过使用createClient创建{TransportClient}。
    * 这个工厂实例维护一个到其他主机的连接池，并应为相同的远程主机返回相同的TransportClient。 它还为所有TransportClient共享单个线程池。
    *
    * 在返回新客户端之前初始化运行给定TransportClientBootstraps的ClientFactory。Bootstraps会被同步执行，并且必须运行成功才能创建Client
    * 给这个实例TransportClientFactory:赋成员
    * context：TransportContext，
    * conf:TransportConf会通过它成员：ConfigProvider子类关联SparkConf
    * 还有初始化netty的NioSocketChannel.class、NioEventLoopGroup线程组、ByteBuf分配器PooledByteBufAllocator
    */
  clientFactory = transportContext.createClientFactory(clientBootstrap.toSeq.asJava)
  //创建一个nettySever，包括编解码，还有入站事件都加到TransportServer这个nettySever中
  server =createServer(serverBootstrap.toList)
  appId =conf.getAppId //值类似app-20180404172558-0000
  logInfo("Servercreated on " + server.getPort)

1,先看一下NettyBlockTransferService.createServer()

/** Creates andbinds the TransportServer, possibly trying multiple ports.
  * 创建并绑定TransportServer，可能尝试多个端口。*/
private def createServer(bootstraps: List[TransportServerBootstrap]):TransportServer = {
  def startService(port: Int):(TransportServer, Int) = {
    //实例化TransportContext将给成员赋值conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联 、rpcHandler:NettyBlockRpcServer、closeIdleConnections：false、同时给出站的编码器、入站解码器，赋具体实例
    val server= transportContext.createServer(port, bootstraps.asJava)
    (server, server.getPort)
  }

  val portToTry= conf.getInt("spark.blockManager.port", 0)
  //CoarseGrainedExecutorBackend的所有日志在%SPARK_HOME%/work下面
  //18/05/16 16:33:47 INFO util.Utils:Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 57010

//分配端口同时启动NettyServer
  Utils.startServiceOnPort(portToTry, startService, conf, getClass.getName)._1
}

2,TransportContext.createServer(),先实例化TransportServer，它就是NettyServer

/** Create aserver which will attempt to bind to a specific port. */
public TransportServer createServer(int port, List<TransportServerBootstrap>bootstraps) {
  /**
   * context:就是当前TransportContext，可以创建{TransportServer}，{TransportClientFactory}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道
   * hostToBind：null
   * port: 是Utils.startServiceOnPort方法随机分配的
   * rpcHandler：NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
   * bootstraps：是一个空的ArrayList
   */
  return new TransportServer(this, null, port, rpcHandler, bootstraps);
}

3,给TransportServer赋成员变量的同时，在当前节点上绑定端口

/**
 * Creates a TransportServer that bindsto the given host and the given port, or to any available
 * if 0. If you don't want to bind to anyspecial host, set "hostToBind" to null.
 * 创建一个传输服务，绑定指定主机和端口，如果端口是0，则返回任何值，如果不想绑定主机，可以设置hostToBind为null
 * context:就是当前TransportContext，可以创建{TransportServer}，{TransportClientFactory}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道
 * hostToBind：null
 * port: 是Utils.startServiceOnPort方法随机分配的
 * rpcHandler：NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
 * bootstraps：因为没有认证是一个空的List
 * */
public TransportServer(
    TransportContext context,
    String hostToBind,
    int portToBind,
    RpcHandler appRpcHandler, //NettyBlockRpcServer
    List<TransportServerBootstrap> bootstraps) {
  this.context = context;
  //conf: TransportConf会根据它的成员ConfigProvider子类关联到SparkConf
  this.conf = context.getConf();
  this.appRpcHandler= appRpcHandler;
  this.bootstraps= Lists.newArrayList(Preconditions.checkNotNull(bootstraps));

  try {
    init(hostToBind, portToBind);
  } catch(RuntimeException e) {
    JavaUtils.closeQuietly(this);
    throw e;
  }
}

4，nettyserver的引导服务，可以查看netty官网

//hostToBind： null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
  //默认得到的NIO枚举值
  IOMode ioMode = IOMode.valueOf(conf.ioMode());
  //conf.serverThreads()：CoarseGrainedExecutorBackend的core数相同
  //该方法创建一个Netty的NioEventLoopGroup线程组
  EventLoopGroup bossGroup =
    NettyUtils.createEventLoop(ioMode, conf.serverThreads(), "shuffle-server");
  EventLoopGroup workerGroup = bossGroup;
  /**
   * conf.preferDirectBufs(): 找key是:spark.shuffle.io.preferDirectBufs，查看SparkConf没有这个key则返回true
   * conf.clientThreads()的值是1
   *NettyUtils.createPooledByteBufAllocator（）：创建一个池化的ByteBuf分配器PooledByteBufAllocator
   */
  PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
    conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());
  //ServerBootstrap用于NIO服务端辅助启动类，说是降低服务端的开发复杂度
  bootstrap = new ServerBootstrap()
    .group(bossGroup, workerGroup)
    .channel(NettyUtils.getServerChannelClass(ioMode))//NioServerSocketChannel.class
    .option(ChannelOption.ALLOCATOR, allocator)
    .childOption(ChannelOption.ALLOCATOR, allocator);

  if (conf.backLog() > 0) {
    //tcp通信队列的大小
    bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
  }

  if (conf.receiveBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
  }

  if (conf.sendBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
  }

  bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    //ChannelInitializer.initChannel()会在Channel注册后被调用，该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
    @Override
    protected void initChannel(SocketChannelch) throws Exception {
      //NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
      RpcHandler rpcHandler = appRpcHandler;
      for (TransportServerBootstrap bootstrap : bootstraps) {
        rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
      }
      //context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去，给ChannelPipeline设置ChannelHander拦截器
      //返回TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
      context.initializePipeline(ch, rpcHandler);
    }
  });
  //也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server
  //由于hostToBind是Null所以就是当前worker节点，用portToBind端口
  InetSocketAddress address = hostToBind == null ?
      new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
  channelFuture = bootstrap.bind(address);
  channelFuture.syncUninterruptibly();

  port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
  logger.debug("Shuffle server started on port:" + port);
}

 

5,TransportContext.initializePipeline(ch, rpcHandler)初始化ChannelPipeline的ChannelHandler拦截器

 

/**
 * Initializes a client or server NettyChannel Pipeline which encodes/decodes messages and
 * has a {@link org.apache.spark.network.server.TransportChannelHandler}to handle request or
 * response messages.
 *
 * @param channel The channel to initialize.
 * @param channelRpcHandler The RPC handler to use for the channel.
 *
 * @return Returns the createdTransportChannelHandler, which includes a TransportClient that can
 * be used to communicate on thischannel. The TransportClient is directly associated with a
 * ChannelHandler to ensure all users ofthe same channel get the same TransportClient object.
 *
 * 初始化客户端或服务器Netty Channel Pipeline里面编码/解码消息，并返回{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息。
 *
 * initializePipeline（）返回创建的TransportChannelHandler，它包含可用于在此通道上通信的TransportClient。
 *  TransportClient直接与ChannelHandler相关联，以确保同一通道的所有用户都获得相同的TransportClient对象。
 *
 *==》TransportClient的作用：客户端获取预先协商的流的连续块。此API旨在允许大量数据的高效传输，将其分解为大小从几百KB到几MB的块。
 *   该TransportClient类用于向服务器发送请求，而{@link TransportResponseHandler}负责处理来自服务器的响应，以响应[[TransportClient]]发出的请求。
 *参数：
 * channel：SocketChannel，通过的pipeline得到ChannelPipeline
 * channelRpcHandler：NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
 */
public TransportChannelHandler initializePipeline(
    SocketChannel channel,
    RpcHandler channelRpcHandler) {
  try {
    //TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
    TransportChannelHandler channelHandler =createChannelHandler(channel, channelRpcHandler);
    channel.pipeline()
      .addLast("encoder", encoder)

//spark自定义了tcp半包解码器
      .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
      .addLast("decoder", decoder)
      .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs()/ 1000))
      // NOTE: Chunks are currently guaranteed to be returnedin the order of request, but this
      // would require more logic toguarantee if this were not part of the same event loop.
      .addLast("handler", channelHandler);
    return channelHandler;
  } catch(RuntimeException e) {
    logger.error("Error while initializing Nettypipeline", e);
    throw e;
  }
}

 

===>创建TransportContext.createChannelHandler（）用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序

/**
 * Creates the server- and client-sidehandler which is used to handle both RequestMessages and ResponseMessages. Thechannel is expected to have been successfully created, though certain properties(such as the remoteAddress()) may not be available yet.
 * 创建用于处理RequestMessages和ResponseMessages的服务器端和客户端处理程序。 预计该channel已成功创建，但某些属性（如remoteAddress（））可能还不可用。
 * 参数：
 * channel：SocketChannel，通过的pipeline得到ChannelPipeline
 * channelRpcHandler：NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
 */
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandlerrpcHandler) {
//TransportResponseHandler负责处理来自服务器的响应。以响应[[TransportClient]]发出的请求
  TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
  //该TransportClient类用于向服务器发送请求,TransportResponseHandler负责响应它的内容
  TransportClient client = new TransportClient(channel, responseHandler);
  //TransportRequestHandler:处理来自客户端的请求并将块数据写回的处理程序。
  TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
    rpcHandler);
  //{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息
  return new TransportChannelHandler(client, responseHandler, requestHandler,
    conf.connectionTimeoutMs(), closeIdleConnections);
}

===》返回TransportServer.init()，通过serverBootStrap.bind()将nettyServer的端口绑定起来

//hostToBind： null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
  。。。。。
  bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    //ChannelInitializer.initChannel()会在Channel注册后被调用，该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
    @Override
    protected void initChannel(SocketChannelch) throws Exception {
      //NettyBlockRpcServer作用： 为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
      RpcHandler rpcHandler = appRpcHandler;
      for (TransportServerBootstrap bootstrap : bootstraps) {
        rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
      }
      //context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去，给ChannelPipeline设置ChannelHander拦截器
      //返回TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
      context.initializePipeline(ch, rpcHandler);
    }
  });
  //也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server由于hostToBind是Null所以就是当前worker节点，用portToBind端口
  InetSocketAddress address = hostToBind == null ?
      new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
  channelFuture = bootstrap.bind(address);
  channelFuture.syncUninterruptibly();

  port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
  logger.debug("Shuffle server started on port:" + port);
}

==》到此NettyServer启动成功，同时整个NettyBlockTransferService.init()方法体也执行完成