kafka的通信模块SocketServer代码解析

SocketServer作为kafka集群中broker的通信模块，主要用来接收socket请求，然后产生为之服务的SocketChannel对象，再通过这个SocketChannel对象来和客户端通信。SocketServer的代码如下：

class SocketServer(val brokerId: Int,
                   val host: String,
                   val port: Int,
                   val numProcessorThreads: Int,
                   val maxQueuedRequests: Int,
                   val sendBufferSize: Int,
                   val recvBufferSize: Int,
                   val maxRequestSize: Int = Int.MaxValue,
                   val maxConnectionsPerIp: Int = Int.MaxValue,
                   val connectionsMaxIdleMs: Long,
                   val maxConnectionsPerIpOverrides: Map[String, Int] ) extends Logging with KafkaMetricsGroup {
  this.logIdent = "[Socket Server on Broker " + brokerId + "], "
  private val time = SystemTime
  private val processors = new Array[Processor](numProcessorThreads)
  @volatile private var acceptor: Acceptor = null
  val requestChannel = new RequestChannel(numProcessorThreads, maxQueuedRequests)

  /* a meter to track the average free capacity of the network processors */
  private val aggregateIdleMeter = newMeter("NetworkProcessorAvgIdlePercent", "percent", TimeUnit.NANOSECONDS)
  ......
}

它主要包括三个模块：1）Acceptor主要用来监听socket请求、2）Processor主要用来转发socket他请求和相应、3）RequestChannel主要用来缓存socket请求和相应。

Acceptor的初始化流程如下：

private[kafka] class Acceptor(val host: String, 
                              val port: Int, 
                              private val processors: Array[Processor],
                              val sendBufferSize: Int, 
                              val recvBufferSize: Int,
                              connectionQuotas: ConnectionQuotas) extends AbstractServerThread(connectionQuotas) {
  //开启socket服务
  val serverChannel = openServerSocket(host, port)

  def run() {
    //注册Accept事件
    serverChannel.register(selector, SelectionKey.OP_ACCEPT);
    startupComplete()
    var currentProcessor = 0
   //监听Accept事件
    while(isRunning) {
      val ready = selector.select(500)
      if(ready > 0) {
        val keys = selector.selectedKeys()
        val iter = keys.iterator()
        while(iter.hasNext && isRunning) {
          var key: SelectionKey = null
          try {
            key = iter.next
            iter.remove()
            if(key.isAcceptable)
               accept(key, processors(currentProcessor))
            else
               throw new IllegalStateException("Unrecognized key state for acceptor thread.")

            // round robin to the next processor thread
            currentProcessor = (currentProcessor + 1) % processors.length
          } catch {
            case e: Throwable => error("Error while accepting connection", e)
          }
        }
      }
    }
    ......
  }

当监听到Accept事件时，会通过轮询的方式选择一个Processor，然后创建新的socketChannel交给这个Processor处理。Ptocessor的代码如下：

private[kafka] class Processor(val id: Int,
                               val time: Time,
                               val maxRequestSize: Int,
                               val aggregateIdleMeter: Meter,
                               val idleMeter: Meter,
                               val totalProcessorThreads: Int,
                               val requestChannel: RequestChannel,
                               connectionQuotas: ConnectionQuotas,
                               val connectionsMaxIdleMs: Long) extends AbstractServerThread(connectionQuotas) {

  private val newConnections = new ConcurrentLinkedQueue[SocketChannel]()
  private val connectionsMaxIdleNanos = connectionsMaxIdleMs * 1000 * 1000
  private var currentTimeNanos = SystemTime.nanoseconds
  private val lruConnections = new util.LinkedHashMap[SelectionKey, Long]
  private var nextIdleCloseCheckTime = currentTimeNanos + connectionsMaxIdleNanos

  override def run() {
    startupComplete()
    while(isRunning) {
      // 将新添加的socketChannel注册OP_READ事件
      configureNewConnections()
      // 处理RequestChannel中的OP_WRITE事件
      processNewResponses()
      val startSelectTime = SystemTime.nanoseconds
      val ready = selector.select(300)
      currentTimeNanos = SystemTime.nanoseconds
      val idleTime = currentTimeNanos - startSelectTime
      idleMeter.mark(idleTime)
      // We use a single meter for aggregate idle percentage for the thread pool.
      // Since meter is calculated as total_recorded_value / time_window and
      // time_window is independent of the number of threads, each recorded idle
      // time should be discounted by # threads.
      aggregateIdleMeter.mark(idleTime / totalProcessorThreads)

      trace("Processor id " + id + " selection time = " + idleTime + " ns")
      //监听selector上的OP_READ和OP_WRITE事件
      if(ready > 0) {
        val keys = selector.selectedKeys()
        val iter = keys.iterator()
        while(iter.hasNext && isRunning) {
          var key: SelectionKey = null
          try {
            key = iter.next
            iter.remove()
            if(key.isReadable)
              read(key)
            else if(key.isWritable)
              write(key)
            else if(!key.isValid)
              close(key)
            else
              throw new IllegalStateException("Unrecognized key state for processor thread.")
          } catch {
            case e: EOFException => {
              info("Closing socket connection to %s.".format(channelFor(key).socket.getInetAddress))
              close(key)
            } case e: InvalidRequestException => {
              info("Closing socket connection to %s due to invalid request: %s".format(channelFor(key).socket.getInetAddress, e.getMessage))
              close(key)
            } case e: Throwable => {
              error("Closing socket for " + channelFor(key).socket.getInetAddress + " because of error", e)
              close(key)
            }
          }
        }
      }
      maybeCloseOldestConnection
    }
    debug("Closing selector.")
    closeAll()
    swallowError(selector.close())
    shutdownComplete()
  }

其中，队列newConnections 保存了Acceptor线程转移过来的socketChannel，主要步骤如下：

1、从newConnections中取出新加入的socketChannel，将这些socketChannel注册OP_READ事件。

2、从RequestChannel中获取对应客户端的相应请求，然后产生OP_WRITE事件。

3、监听selector上的读写事件，如果是读事件，说明有新的request请求，将这些请求转移给RequestChannel的请求队列；如果是写事件，说明之前的request请求已经处理完毕，需要将它们发送给对应的客户端；如果是关闭事件，说明客户端已经断开连接，服务器也需要释放相应的资源。

RequestChannel本质上是为了解耦SocketServer和KafkaApis两个模块，内部包含requestQueue和responseQueues两个队列，分别用来存放客户端的请求和服务器给客户端的相应。requestQueue队列的个人为1，responseQueues队列的个数是numProcessors，默认是3个。Ptocessor会将读取到的OP_READ事件中的request请求放入到requestQueue队列中，KafkaRequestHandlerPool中的KafkaRequestHandler线程会从requestQueue队列取出request请求给KafkaApis模块处理，处理完成之后会将生成的Reponse放入到responseQueues的某个队列中，并触发Processor的OP_WRITE事件。最后由Processor发送给客户端。它们的关系如下：

1、客户端建立连接：

Client----》Acceptor----》Processor中的newConnections 

2、Processor从newConnections中取出socketChannel，注册OP_READ事件。同时获取RequestChannel中responseQueues队列的Response，并生成OP_WRITE事件。然后监听selector上的读写事件。将读事件中的request请求发送给RequestChannel中的requestQueue队列，将写事件中的response发送给客户端。

3、RequestChannel中的requestQueue----》KafkaRequestHandlerPool----》KafkaApis处理完成给KafkaRequestHandlerPool---》KafkaRequestHandlerPool生成response---》RequestChannel中responseQueues