从源码的角度带你一步步剖析NioEventLoop(反应器)的三步曲

Netty 中的反应器组件有多个实现类,这些实现类与其通道类型相互匹配. 对应于NioSocketChannel通道,Netty的反应器类为NioEventLoop(Nio事件轮询).

NioEventLoop 与Nio 原生selector的关系

NioEventLoop类有两个重要的成员属性：一个是Thread线程类的成员，一个是Java NIO选择器的成员属性.

一个NioEventLoop拥有一个线程Thread,主要用于轮询Nio Selector并处理IO事件,处理其他的非IO任务。Reactor 线程一般是在channel 在selector上注册之前启动的.

一个EventLoop反应器和NettyChannel通道是一对多的关系：一个反应器可以注册成千上万的通道.如图所示:

NioEventLoop 与Nio 原生channel的关系

以NioSocketChannel 类作为channel 通道的代表进行说明,

Reactor三步曲

第一步: 注册

将channel 通道的就绪事件,注册到选择器Selector.

一个Reactor 对应一个选择器Selector,一个Reactor拥有一个Selector成员属性.

以Netty的Client端为例

入口代码为:

//通道连接
 channelFuture = bootstrap.connect("127.0.0.1",8080).sync();

启动类AbstractBootstrap类的绑定端口方法

public ChannelFuture connect(String inetHost, int inetPort) {
    return this.connect(InetSocketAddress.createUnresolved(inetHost, inetPort));
}


public ChannelFuture connect(SocketAddress remoteAddress) {
    ObjectUtil.checkNotNull(remoteAddress, "remoteAddress");
    this.validate();
    return this.doResolveAndConnect(remoteAddress, this.config.localAddress());
}

private ChannelFuture doResolveAndConnect(final SocketAddress remoteAddress, final SocketAddress localAddress) {
    ChannelFuture regFuture = this.initAndRegister();
    final Channel channel = regFuture.channel();
    if (regFuture.isDone()) {
        return !regFuture.isSuccess() ? regFuture : this.doResolveAndConnect0(channel, remoteAddress, localAddress, channel.newPromise());
    } else {
        final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);
        regFuture.addListener(new ChannelFutureListener() {
            public void operationComplete(ChannelFuture future) throws Exception {
                Throwable cause = future.cause();
                if (cause != null) {
                    promise.setFailure(cause);
                } else {
                    promise.registered();
                    Bootstrap.this.doResolveAndConnect0(channel, remoteAddress, localAddress, promise);
                }

            }
        });
        return promise;
    }
}

channel注册选择器Selector 入口方法initAndRegister():

final ChannelFuture initAndRegister() {
    Channel channel = null;

    try {
        channel = this.channelFactory.newChannel();
        this.init(channel);
    } catch (Throwable var3) {
        if (channel != null) {
            channel.unsafe().closeForcibly();
            return (new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE)).setFailure(var3);
        }

        return (new DefaultChannelPromise(new FailedChannel(), GlobalEventExecutor.INSTANCE)).setFailure(var3);
    }

    ChannelFuture regFuture = this.config().group().register(channel);
    if (regFuture.cause() != null) {
        if (channel.isRegistered()) {
            channel.close();
        } else {
            channel.unsafe().closeForcibly();
        }
    }

    return regFuture;
}

MultithreadEventLoopGroup.register()方法:

 @Override
public ChannelFuture register(ChannelPromise promise) {
    return next().register(promise);
}

@Deprecated
@Override
public ChannelFuture register(Channel channel, ChannelPromise promise) {
    return next().register(channel, promise);
}

SingleThreadEventLoop.register()方法

@Override
public ChannelFuture register(final ChannelPromise promise) {
    ObjectUtil.checkNotNull(promise, "promise");
    promise.channel().unsafe().register(this, promise);
    return promise;
}

AbstractChannel.AbstractUnsafe.register()

@Override
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
    ObjectUtil.checkNotNull(eventLoop, "eventLoop");
    if (isRegistered()) {
        promise.setFailure(new IllegalStateException("registered to an event loop already"));
        return;
    }
    if (!isCompatible(eventLoop)) {
        promise.setFailure(
                new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
        return;
    }

    AbstractChannel.this.eventLoop = eventLoop;

    if (eventLoop.inEventLoop()) {
        register0(promise);
    } else {
        try {
            //如果线程没有启动,启动线程
            eventLoop.execute(new Runnable() {
                @Override
                public void run() {
                    // 通道注册到选择器,向流水线发送通道激活事件
                    register0(promise);
                }
            });
        } catch (Throwable t) {
            logger.warn(
                    "Force-closing a channel whose registration task was not accepted by an event loop: {}",
                    AbstractChannel.this, t);
            closeForcibly();
            closeFuture.setClosed();
            safeSetFailure(promise, t);
        }
    }
}

AbstractChannel.AbstractUnsafe.register0()

private void register0(ChannelPromise promise) {
    try {
       
        if (!promise.setUncancellable() || !ensureOpen(promise)) {
            return;
        }
        boolean firstRegistration = neverRegistered;
        doRegister();
        neverRegistered = false;
        registered = true;

        pipeline.invokeHandlerAddedIfNeeded();

        safeSetSuccess(promise);
        pipeline.fireChannelRegistered();
        if (isActive()) {
            if (firstRegistration) {
                pipeline.fireChannelActive();
            } else if (config().isAutoRead()) {
                beginRead();
            }
        }
    } catch (Throwable t) {
        closeForcibly();
        closeFuture.setClosed();
        safeSetFailure(promise, t);
    }
}

AbstractNioChannel.doRegister()

 @Override
protected void doRegister() throws Exception {
    boolean selected = false;
    for (;;) {
        try {
        //重点: 首先拿到原生的选择器eventLoop,在拿到原生的通道channel, 最后完成通道的注册工作,
            selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);
            return;
        } catch (CancelledKeyException e) {
            if (!selected) {
              
                eventLoop().selectNow();
                selected = true;
            } else {
                SelectionKey is still cached
                throw e;
            }
        }
    }
}

AbstractNioChannel 通道类有一个本地java通道成员ch,在AbstractNioChannel 的构造函数中,被初始化.

private final SelectableChannel ch;
protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {
    super(parent);
    this.ch = ch;
    this.readInterestOp = readInterestOp;
    try {
        ch.configureBlocking(false);
    } catch (IOException e) {
        try {
            ch.close();
        } catch (IOException e2) {
            logger.warn(
                        "Failed to close a partially initialized socket.", e2);
        }

        throw new ChannelException("Failed to enter non-blocking mode.", e);
    }
}

Netty 的channel 通过javaChannel()方法取得了java本地Channel. 它返回的是一个java NIO SocketChannel. 代码如下:

protected SelectableChannel javaChannel() {
    return ch;
}

最后,这个SocketChannel 注册到与eventLoop 关联的selector上了.

  selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);

channel.register 的第三个参数是设置selectionKey 的附加对象的,和调用selectionKey.attach(object)的效果一样.而doRegister()调用javaChannel().register()方法传递的第三个参数是this,它是一个MioSocketChannel 的实例, 意思是SocketChannel对象自身,以附加字段的方式添加到selectionKey中,供事件就绪后使用.

总结,注册时序图如下:

第二步轮询

轮询的代码,是Reactor 重要的一个组成部分,轮询选择器是否有就绪事件.

NioEventLoop 的父类SingleThreadEventExecutor中,有一个Thread thread 属性,存储了一个本地java线程,

 private volatile Thread thread;

此java 线程是在SocketChannel 注册到eventLoop的selector之前启动的,在Execute的方法中,去调用startThread()启动线程,
.AbstractChannel.AbstractUnsafe.register()中有一个方法调用 eventLoop.execute(new Runnable() ),此调用就是启动EventLoop线程的入口,execute()方法的实现在SingleThreadEventExecutor类中,具体源码如下:

private void execute(Runnable task, boolean immediate) {
    boolean inEventLoop = inEventLoop();
    //加入任务到自己的队列
    addTask(task);
    if (!inEventLoop) {
        //如果线程没有启动,则启动
        startThread();
        if (isShutdown()) {
            boolean reject = false;
            try {
                if (removeTask(task)) {
                    reject = true;
                }
            } catch (UnsupportedOperationException e) {
            }
            if (reject) {
                reject();
            }
        }
    }

    if (!addTaskWakesUp && immediate) {
        wakeup(inEventLoop);
    }
}

将通道注册的任务,添加到双向队列taskqueue中,

protected void addTask(Runnable task) {
    ObjectUtil.checkNotNull(task, "task");
    if (!offerTask(task)) {
        reject(task);
    }
}
final boolean offerTask(Runnable task) {
    if (isShutdown()) {
        reject();
    }
    return taskQueue.offer(task);
}

如果reactor的线程未启动,则调用startThread()方法启动线程,启动线程的源码如下:

private void startThread() {
    if (state == ST_NOT_STARTED) {
        if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
            boolean success = false;
            try {
                doStartThread();
                success = true;
            } finally {
                if (!success) {
                    STATE_UPDATER.compareAndSet(this, ST_STARTED, ST_NOT_STARTED);
                }
            }
        }
    }
}

STATE_UPDATER是SingleThreadEventExecutor内部维护的一个属性,它的作用是标识当前thread的状态,在初始的时候,state == ST_NOT_STARTED,因此第一次调用startThread() 方法时,就会计入到if语句中,进而调用到doStartThread()方法.doStartThread() 方法代码如下:

private void doStartThread() {
    assert thread == null;
    //通过线程池执行,通过线程工厂,每提交一个任务就会创建一个线程
    executor.execute(new Runnable() {
        @Override
        public void run() {
            //绑定第一条线程
            thread = Thread.currentThread();
            if (interrupted) {
                thread.interrupt();
            }

            boolean success = false;
            updateLastExecutionTime();
            try {
                //启动事件轮询和任务执行
                SingleThreadEventExecutor.this.run();
                success = true;
            } catch (Throwable t) {
                logger.warn("Unexpected exception from an event executor: ", t);
            } finally {
                ...
            }
        }
    });
}

接下来我们看下上述代码中的线程的创建来自于ExecuteGroup 的线程工厂,在MultithreadEventExecutorGroup类的构造函数中

 protected MultithreadEventExecutorGroup(int nThreads, Executor executor,
                                            EventExecutorChooserFactory chooserFactory, Object... args) {
        checkPositive(nThreads, "nThreads");

        if (executor == null) {
            //线程工厂,执行一个线程,创建一个线程,特点是每执行一个任务创建一个新的线程
            executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        }

        //线程数组,创建的线程数量是由装配引导bossGroup和 workerGroup时决定的
        children = new EventExecutor[nThreads];

        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
                //在eventLoop的startthread中创建线程
                children[i] = newChild(executor, args);
                success = true;
            } catch (Exception e) {
                // TODO: Think about if this is a good exception type
                throw new IllegalStateException("failed to create a child event loop", e);
            } finally {
              ....
            }
        }

到目前为止,NioEventLoop.run()就可以执行了,
总结: NioEventLoop中维护了一个线程,线程启动时会调用NioEventLoop 的run方法,执行I/O任务和非I/O任务; I/O任务即selectionKey中ready的事件,如accept,connect,read,write等,由processSelectionKey方法触发. 非I/O 任务,添加到taskQueue的任务 ,如register0,bind0等任务,由runAllTasks方法触发.

两种任务的执行时间比由变量ioRatio控制,默认为50,则表示允许非IO任务执行的时间与IO任务的执行时间相等.

NioEventLoop 事件轮询包含两种策略:

• 非阻塞的select策略
• 阻塞的select策略

事件的轮询在NioEventLoop.run()方法中,其源码如下:

@Override
    protected void run() {
        int selectCnt = 0;
        for (;;) {
            try {
                int strategy;
                try {
                    // select 策略选择
                    strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
                    switch (strategy) {
                        // 1.1 非阻塞的select策略,即重试IO循环
                    case SelectStrategy.CONTINUE:
                        continue;
                        //1.2非阻塞的新事件IO循环
                    case SelectStrategy.BUSY_WAIT:

                        //1.3 阻塞的select策略
                    case SelectStrategy.SELECT:
                        long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
                        if (curDeadlineNanos == -1L) {
                            curDeadlineNanos = NONE; // nothing on the calendar
                        }
                        nextWakeupNanos.set(curDeadlineNanos);
                        try {
                            if (!hasTasks()) {
                                strategy = select(curDeadlineNanos);
                            }
                        } finally {
                            // This update is just to help block unnecessary selector wakeups
                            // so use of lazySet is ok (no race condition)
                            nextWakeupNanos.lazySet(AWAKE);
                        }
                        // 1.4 不需要select,目前已经有可以执行的任务了
                    default:
                    }
                } catch (IOException e) {
                    // If we receive an IOException here its because the Selector is messed up. Let's rebuild
                    // the selector and retry. https://github.com/netty/netty/issues/8566
                    rebuildSelector0();
                    selectCnt = 0;
                    handleLoopException(e);
                    continue;
                }

                selectCnt++;
                //2.执行网络IO事件和任务调度
                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                boolean ranTasks;
                if (ioRatio == 100) {
                    try {
                        if (strategy > 0) {
                            //2.1 处理网络IO事件,分发入口
                            processSelectedKeys();
                        }
                    } finally {
                        // Ensure we always run tasks.
                        //2.2 处理系统Task和自定义Task
                        ranTasks = runAllTasks();
                    }
                } else if (strategy > 0) {
                    //根据ioRatio 计算非IO最多执行的时间
                    final long ioStartTime = System.nanoTime();
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        final long ioTime = System.nanoTime() - ioStartTime;
                        ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                } else {
                    //处理队列中的task任务
                    ranTasks = runAllTasks(0); // This will run the minimum number of tasks
                }

                if (ranTasks || strategy > 0) {
                    if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
                        logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
                                selectCnt - 1, selector);
                    }
                    selectCnt = 0;
                } else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
                    selectCnt = 0;
                }
            } catch (CancelledKeyException e) {
                // Harmless exception - log anyway
                if (logger.isDebugEnabled()) {
                    logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?",
                            selector, e);
                }
            } catch (Error e) {
                throw e;
            } catch (Throwable t) {
                handleLoopException(t);
            } finally {
                // Always handle shutdown even if the loop processing threw an exception.
                try {
                    //执行shutdown后的善后逻辑
                    if (isShuttingDown()) {
                        closeAll();
                        if (confirmShutdown()) {
                            return;
                        }
                    }
                } catch (Error e) {
                    throw e;
                } catch (Throwable t) {
                    handleLoopException(t);
                }
            }
        }
    }

NioEventLoop的run方法，是netty中最核心的方法，没有之一。在该方法中，完成了对已注册的channel上来自底层操作系统的socket事件的处理（在服务端时事件包括客户端的连接事件和读写事件，在客户端时是读写事件）、单线程任务队列的处理（服务端的注册事件、客户端的connect事件等），当然还包括对NIO空轮询的规避、消息的编解码等。

总结: 轮询的时序图如下

第三步:分发

将就绪事件,分发到事件附件的处理器handler中,由handler完成实际的处理.

在NioEventLoop.run()方法中,调用了processSelectedKeys()方法处理IO事件,IO事件分为两种,第一种是优化过的,第二种是普通的, Netty会尝试获取权限去优化原生Selector,如果可以,selectedKeys不为null,都是走优化过的处理方式,

private void processSelectedKeys() {
    if (selectedKeys != null) {
        //优化的selector的key
        processSelectedKeysOptimized();
    } else {
        // 原生的seletor 的key
        processSelectedKeysPlain(selector.selectedKeys());
    }
}

两种方式主要是遍历selectionKey的方式不同,具体处理事件的调用是一样的.看下processSelectedKeysPlain()方法的处理过程:

private void processSelectedKeysPlain(Set<SelectionKey> selectedKeys) {

    if (selectedKeys.isEmpty()) {
        return;
    }

    //事件迭代
    Iterator<SelectionKey> i = selectedKeys.iterator();
    for (;;) {
        final SelectionKey k = i.next();
        final Object a = k.attachment();
        i.remove();

        if (a instanceof AbstractNioChannel) {
          //处理事件
            processSelectedKey(k, (AbstractNioChannel) a);
        } else {
            @SuppressWarnings("unchecked")
            NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
            processSelectedKey(k, task);
        }

        if (!i.hasNext()) {
            break;
        }

        if (needsToSelectAgain) {
            selectAgain();
            selectedKeys = selector.selectedKeys();

            // Create the iterator again to avoid ConcurrentModificationException
            if (selectedKeys.isEmpty()) {
                break;
            } else {
                i = selectedKeys.iterator();
            }
        }
    }
}

迭代selectedKeys 获取就绪的IO事件,为每个事件都调用processSelectedKey来处理它. 在前面的Channel 注册时,将NioSocketChannel 以附加字段的方式添加到了selectionKey中,在这里,通过k.attachment()取得通道对象,然后就调用processSelectedKey来处理IO事件和通道.事件处理函数processSelectedKey()源码如下:

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
    final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
    if (!k.isValid()) {
        final EventLoop eventLoop;
        try {
            eventLoop = ch.eventLoop();
        } catch (Throwable ignored) {
            return;
        }
        if (eventLoop == this) {
            // close the channel if the key is not valid anymore
            unsafe.close(unsafe.voidPromise());
        }
        return;
    }

    try {
        int readyOps = k.readyOps();
        if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
            int ops = k.interestOps();
            ops &= ~SelectionKey.OP_CONNECT;
            k.interestOps(ops);

            unsafe.finishConnect();
        }

        if ((readyOps & SelectionKey.OP_WRITE) != 0) {
           unsafe.forceFlush();
        }
        if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
            unsafe.read();
        }
    } catch (CancelledKeyException ignored) {
        unsafe.close(unsafe.voidPromise());
    }
}

从代码中可知,处理事件有可读(OP_READ),可写(OP_WRITE) 等事件,以可读(OP_READ)为例看下通道的可读事件,会调用 unsafe.read()方法,AbstractNioByteChannel.NioByteUnsafe.read()其源码如下:

@Override
        public final void read() {
            final ChannelConfig config = config();
            if (shouldBreakReadReady(config)) {
                clearReadPending();
                return;
            }
            final ChannelPipeline pipeline = pipeline();
            final ByteBufAllocator allocator = config.getAllocator();
            final RecvByteBufAllocator.Handle allocHandle = recvBufAllocHandle();
            allocHandle.reset(config);

            ByteBuf byteBuf = null;
            boolean close = false;
            try {
                do {
                    //分配缓冲区,大小自适应
                    byteBuf = allocHandle.allocate(allocator);
                    //核心代码:从socket revbuf中接收数据.
                    allocHandle.lastBytesRead(doReadBytes(byteBuf));
                    if (allocHandle.lastBytesRead() <= 0) {
                        // nothing was read. release the buffer.
                        byteBuf.release();
                        byteBuf = null;
                        close = allocHandle.lastBytesRead() < 0;
                        if (close) {
                            // There is nothing left to read as we received an EOF.
                            readPending = false;
                        }
                        break;
                    }

                    allocHandle.incMessagesRead(1);
                    readPending = false;
                     //通过pipeline触发读事件
                    pipeline.fireChannelRead(byteBuf);
                    byteBuf = null;
                } while (allocHandle.continueReading()); //判断是否继续读

                allocHandle.readComplete();
                pipeline.fireChannelReadComplete();

                if (close) {
                    closeOnRead(pipeline);
                }
            } catch (Throwable t) {
                handleReadException(pipeline, byteBuf, t, close, allocHandle);
            } finally {
                ...
            }
        }
    }

在核心代码中,使用doReadBytes()函数从socket revbuf 读取数据,但每次读取前都需要记录缓冲区中可写区域的大小,用于判断缓冲区是否读满,继而决定是否继续读取数据.NioSocketChannel.doReadBytes()其源代码如下:

@Override
protected int doReadBytes(ByteBuf byteBuf) throws Exception {
    final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
   // 每次读取数据前,记录缓冲区中可写域大小,判断是否将缓冲区读满
    allocHandle.attemptedBytesRead(byteBuf.writableBytes());
    //重点代码: 首先通过JavaChannel()方法拿到java nio的原生通道,然后把数据读取到byteBuf.接着读取量
    // 由allocHandle.attemptedBytesRead()计算出来
    return byteBuf.writeBytes(javaChannel(), allocHandle.attemptedBytesRead());
}

总结: NioByteUnsafe.read 方法的工作:

• 分配缓冲区: 默认1024 byte,之后根据最近几次请求的数据包大小,猜测下一次数据包大小,
• 读取数据:直接调用java nio 的底层代码.
• 触发pipeline.fireChannelRead(byteBuf):开始pipeline流水线的业务处理.
• 判断是否继续读:有两个标准,一是不能超过最大的读取次数(默认16次);二是缓冲区的数据每次都要读满,比如分配2 KB byteBuf,则必须读取2KB的数据.

总结:分发的时序图如下:

到此为止NioEventLoop(反应器)的三步曲就已经完成.

EventLoop 性能高的原因是EventLoop采用了"无锁化"设计;EventLoop是一个Reactor模型的事件处理器,一个EventLoop对应一个线程,其内部会维护一个selector和taskQueue,负责处理网络IO请求和内部任务,这里的selector和taskQueue是线程内部的.

selector的IO事件处理,以及任务队列的任务处理,都在EventLoop线程串行的执行.