spark-core_29：blockManager.initialize（）=》NettyBlockTransferService.init()-NettyServer创建源码分析

最新推荐文章于 2022-02-28 15:13:10 发布

水中舟_luyl

最新推荐文章于 2022-02-28 15:13:10 发布

阅读量509

点赞数

CC 4.0 BY-SA版权

分类专栏： spark core

本文链接：https://blog.youkuaiyun.com/luyllyl/article/details/80420171

spark 同时被 2 个专栏收录

38 篇文章

订阅专栏

core

29 篇文章

订阅专栏

本文解析了Spark中NettyServer的启动流程，包括TransportConf配置、TransportContext创建及初始化、TransportServer绑定端口等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

上一节（spark-core_28：Executor初始化过程env.blockManager.initialize(conf.getAppId)- NettyBlockTransferService.init()源码分析）

分析了

a、NettyBlockRpcServer是用于打开的上传注册在BlockManager中的Block块

b,TansportConf:会通过它成员：ConfigProvider子类关联SparkConf,并按SparkTransportConf.fromSparkConf(conf,"shuffle", numCores)设置nettyserver和client的线程池的大小，并按fromSparkConf第二个参数设置key的变量值

3，TransportContext:包含创建{TransportServer:nettyServer}，{TransportClientFactory用来创建TransportClient}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道。实例化TransportContext将给成员赋值 conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联、rpcHandler:NettyBlockRpcServer、closeIdleConnections：false、同时给出站的编码器、入站解码器，赋具体实例

4，构造了TransportClientFactory:会将TransportContext、TransportConf,按IOMode的枚举类型，默认就是NIO, 得到NioSocketChannel.class基于IOMode枚举创建Netty的EventLoopGroup线程组、创建一个池化的ByteBuf分配器PooledByteBufAllocator给它的在成员变量

override def init(blockDataManager: BlockDataManager): Unit = {
/** conf.getAppId: app-20180508234845-0000
    * serializer:    JavaSerializer()
    * blockDataManager : BlockManager实例
    * NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
    */
val rpcHandler= new NettyBlockRpcServer(conf.getAppId, serializer, blockDataManager)
var serverBootstrap:Option[TransportServerBootstrap] = None
var clientBootstrap:Option[TransportClientBootstrap] = None
if (authEnabled) {//默认是false，不开启认证
    serverBootstrap = Some(new SaslServerBootstrap(transportConf, securityManager))
    clientBootstrap = Some(new SaslClientBootstrap(transportConf, conf.getAppId, securityManager,
      securityManager.isSaslEncryptionEnabled()))
}

transportContext = new TransportContext(transportConf, rpcHandler)

/** 没有开启ssl所以clientBootstrap是空的
    *
    * TransportClientFactory:这个工厂实例通过使用createClient创建{TransportClient}。
    * 这个工厂实例维护一个到其他主机的连接池，并应为相同的远程主机返回相同的TransportClient。它还为所有TransportClient共享单个线程池。
    *
    * 在返回新客户端之前初始化运行给定TransportClientBootstraps的ClientFactory。Bootstraps会被同步执行，并且必须运行成功才能创建Client
    * 给这个实例TransportClientFactory:赋成员
    * context：TransportContext，
    * conf:TransportConf会通过它成员：ConfigProvider子类关联SparkConf
    * 还有初始化netty的NioSocketChannel.class、NioEventLoopGroup线程组、ByteBuf分配器PooledByteBufAllocator
    */
clientFactory = transportContext.createClientFactory(clientBootstrap.toSeq.asJava)
//创建一个nettySever，包括编解码，还有入站事件都加到TransportServer这个nettySever中
server =createServer(serverBootstrap.toList)
appId =conf.getAppId //值类似app-20180404172558-0000
logInfo("Servercreated on " + server.getPort)

1,先看一下NettyBlockTransferService.createServer()

/** Creates andbinds the TransportServer, possibly trying multiple ports.
* 创建并绑定TransportServer，可能尝试多个端口。*/
private def createServer(bootstraps: List[TransportServerBootstrap]):TransportServer = {
def startService(port: Int):(TransportServer, Int) = {
    //实例化TransportContext将给成员赋值conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联、rpcHandler:NettyBlockRpcServer、closeIdleConnections：false、同时给出站的编码器、入站解码器，赋具体实例
    val server= transportContext.createServer(port, bootstraps.asJava)
    (server, server.getPort)
}

val portToTry= conf.getInt("spark.blockManager.port", 0)
//CoarseGrainedExecutorBackend的所有日志在%SPARK_HOME%/work下面
//18/05/16 16:33:47 INFO util.Utils:Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 57010

//分配端口同时启动NettyServer
Utils.startServiceOnPort(portToTry, startService, conf, getClass.getName)._1
}

2,TransportContext.createServer(),先实例化TransportServer，它就是NettyServer

/** Create aserver which will attempt to bind to a specific port. */
public TransportServer createServer(int port, List<TransportServerBootstrap>bootstraps) {
/**
   * context:就是当前TransportContext，可以创建{TransportServer}，{TransportClientFactory}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道
   * hostToBind：null
   * port: 是Utils.startServiceOnPort方法随机分配的
   * rpcHandler：NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
   * bootstraps：是一个空的ArrayList
   */
return new TransportServer(this, null, port, rpcHandler, bootstraps);
}

3,给TransportServer赋成员变量的同时，在当前节点上绑定端口

/**
* Creates a TransportServer that bindsto the given host and the given port, or to any available
* if 0. If you don't want to bind to anyspecial host, set "hostToBind" to null.
* 创建一个传输服务，绑定指定主机和端口，如果端口是0，则返回任何值，如果不想绑定主机，可以设置hostToBind为null
* context:就是当前TransportContext，可以创建{TransportServer}，{TransportClientFactory}的上下文，并使用{TransportChannelHandler}设置Netty Channel管道
* hostToBind：null
* port: 是Utils.startServiceOnPort方法随机分配的
* rpcHandler：NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
* bootstraps：因为没有认证是一个空的List
* */
public TransportServer(
    TransportContext context,
    String hostToBind,
    int portToBind,
    RpcHandler appRpcHandler, //NettyBlockRpcServer
    List<TransportServerBootstrap> bootstraps) {
this.context = context;
//conf: TransportConf会根据它的成员ConfigProvider子类关联到SparkConf
this.conf = context.getConf();
this.appRpcHandler= appRpcHandler;
this.bootstraps= Lists.newArrayList(Preconditions.checkNotNull(bootstraps));

try {
    init(hostToBind, portToBind);
} catch(RuntimeException e) {
    JavaUtils.closeQuietly(this);
    throw e;
}
}

4，nettyserver的引导服务，可以查看netty官网

//hostToBind： null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
//默认得到的NIO枚举值
IOMode ioMode = IOMode.valueOf(conf.ioMode());
//conf.serverThreads()：CoarseGrainedExecutorBackend的core数相同
//该方法创建一个Netty的NioEventLoopGroup线程组
EventLoopGroup bossGroup =
    NettyUtils.createEventLoop(ioMode, conf.serverThreads(), "shuffle-server");
EventLoopGroup workerGroup = bossGroup;
/**
   * conf.preferDirectBufs(): 找key是:spark.shuffle.io.preferDirectBufs，查看SparkConf没有这个key则返回true
   * conf.clientThreads()的值是1
   *NettyUtils.createPooledByteBufAllocator（）：创建一个池化的ByteBuf分配器PooledByteBufAllocator
   */
PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
    conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());
//ServerBootstrap用于NIO服务端辅助启动类，说是降低服务端的开发复杂度
bootstrap = new ServerBootstrap()
    .group(bossGroup, workerGroup)
    .channel(NettyUtils.getServerChannelClass(ioMode))//NioServerSocketChannel.class
    .option(ChannelOption.ALLOCATOR, allocator)
    .childOption(ChannelOption.ALLOCATOR, allocator);

if (conf.backLog() > 0) {
    //tcp通信队列的大小
    bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
}

if (conf.receiveBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
}

if (conf.sendBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
}

bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    //ChannelInitializer.initChannel()会在Channel注册后被调用，该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
    @Override
    protected void initChannel(SocketChannelch) throws Exception {
      //NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
      RpcHandler rpcHandler = appRpcHandler;
      for (TransportServerBootstrap bootstrap : bootstraps) {
        rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
      }
      //context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去，给ChannelPipeline设置ChannelHander拦截器
      //返回TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
      context.initializePipeline(ch, rpcHandler);
    }
});
//也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server
//由于hostToBind是Null所以就是当前worker节点，用portToBind端口
InetSocketAddress address = hostToBind == null ?
      new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();

port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
logger.debug("Shuffle server started on port:" + port);
}

5,TransportContext.initializePipeline(ch, rpcHandler)初始化ChannelPipeline的ChannelHandler拦截器

/**
* Initializes a client or server NettyChannel Pipeline which encodes/decodes messages and
* has a {@link org.apache.spark.network.server.TransportChannelHandler}to handle request or
* response messages.
*
* @param channel The channel to initialize.
* @param channelRpcHandler The RPC handler to use for the channel.
*
* @return Returns the createdTransportChannelHandler, which includes a TransportClient that can
* be used to communicate on thischannel. The TransportClient is directly associated with a
* ChannelHandler to ensure all users ofthe same channel get the same TransportClient object.
*
* 初始化客户端或服务器Netty Channel Pipeline里面编码/解码消息，并返回{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息。
*
* initializePipeline（）返回创建的TransportChannelHandler，它包含可用于在此通道上通信的TransportClient。
*  TransportClient直接与ChannelHandler相关联，以确保同一通道的所有用户都获得相同的TransportClient对象。
*
*==》TransportClient的作用：客户端获取预先协商的流的连续块。此API旨在允许大量数据的高效传输，将其分解为大小从几百KB到几MB的块。
*   该TransportClient类用于向服务器发送请求，而{@link TransportResponseHandler}负责处理来自服务器的响应，以响应[[TransportClient]]发出的请求。
*参数：
* channel：SocketChannel，通过的pipeline得到ChannelPipeline
* channelRpcHandler：NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
*/
public TransportChannelHandler initializePipeline(
    SocketChannel channel,
    RpcHandler channelRpcHandler) {
try {
    //TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
    TransportChannelHandler channelHandler =createChannelHandler(channel, channelRpcHandler);
    channel.pipeline()
      .addLast("encoder", encoder)

//spark自定义了tcp半包解码器
      .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
      .addLast("decoder", decoder)
      .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs()/ 1000))
      // NOTE: Chunks are currently guaranteed to be returnedin the order of request, but this
      // would require more logic toguarantee if this were not part of the same event loop.
      .addLast("handler", channelHandler);
    return channelHandler;
} catch(RuntimeException e) {
    logger.error("Error while initializing Nettypipeline", e);
    throw e;
}
}

===>创建TransportContext.createChannelHandler（）用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序

/**
* Creates the server- and client-sidehandler which is used to handle both RequestMessages and ResponseMessages. Thechannel is expected to have been successfully created, though certain properties(such as the remoteAddress()) may not be available yet.
* 创建用于处理RequestMessages和ResponseMessages的服务器端和客户端处理程序。预计该channel已成功创建，但某些属性（如remoteAddress（））可能还不可用。
* 参数：
* channel：SocketChannel，通过的pipeline得到ChannelPipeline
* channelRpcHandler：NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
*/
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandlerrpcHandler) {
//TransportResponseHandler负责处理来自服务器的响应。以响应[[TransportClient]]发出的请求
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
//该TransportClient类用于向服务器发送请求,TransportResponseHandler负责响应它的内容
TransportClient client = new TransportClient(channel, responseHandler);
//TransportRequestHandler:处理来自客户端的请求并将块数据写回的处理程序。
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler);
//{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}

===》返回TransportServer.init()，通过serverBootStrap.bind()将nettyServer的端口绑定起来

//hostToBind： null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
。。。。。
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    //ChannelInitializer.initChannel()会在Channel注册后被调用，该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
    @Override
    protected void initChannel(SocketChannelch) throws Exception {
      //NettyBlockRpcServer作用：为每个请求打开或上传注册在BlockManager中的任意Block块，每一次Chunk的传输相当于一次shuffle
      RpcHandler rpcHandler = appRpcHandler;
      for (TransportServerBootstrap bootstrap : bootstraps) {
        rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
      }
      //context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去，给ChannelPipeline设置ChannelHander拦截器
      //返回TransportChannelHandler是SimpleChannelInboundHandler子类，用于处理RequestMessages和ResponseMessages（这两个都是入站要处理的对象）的服务器端和客户端处理程序
      context.initializePipeline(ch, rpcHandler);
    }
});
//也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server由于hostToBind是Null所以就是当前worker节点，用portToBind端口
InetSocketAddress address = hostToBind == null ?
      new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();

port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
logger.debug("Shuffle server started on port:" + port);
}