通过源码学Netty-发送数据

前言

上一次我们说完了Netty进行业务处理的流程，我们知道Netty处理业务逻辑的本质就是在pipeline中所有的handler执行fireChannelRead的过程，当所有的fireChannelRead都执行完成后，就要将数据写出去，这一回我们一起来看一下Netty是怎么写数据的

开始

netty写数据一共有三种形式，分别是write，flush和writeAndFlush，write方法就是把数据写到一个buffer中，flush方法把buffer中的数据发出去，而writeAndFlush就是写数据到buffer之后立刻发送出去，在write和flush之间还有一个ChannelOutBoundBuffer。Netty写数据还有两个特点，一个是当数据写不进去的时候，会停止写，注册一个OP_WRITE事件，通知什么时候可以写进去了再继续写。还有一个是当Netty一次写入数据都能成功的时候，写一次写入，Netty会尝试写入更多的数据，这主要是靠调整maxBytesPerGatheringWrite的值来实现。Netty的连续写直到写不出去的尝试次数是16（writeSpinCount）。还有，Netty待写数据太多的时候，超过了writeBufferWaterMark.high()高水位线的时候，会将可写标志置为false，让应用端自己决定要不要发送数据。

接下来，我们就来一起看一下源码，首先我们把断点加到EchoServerHandler的channelRead方法里面，然后debug启动EchoServer，再启动EchoClient，然后一路走下去来到了这里

//io.netty.channel.AbstractChannelHandlerContext#write(java.lang.Object, boolean, io.netty.channel.ChannelPromise)
private void write(Object msg, boolean flush, ChannelPromise promise) {
        ObjectUtil.checkNotNull(msg, "msg");
        try {
            if (isNotValidPromise(promise, true)) {
                ReferenceCountUtil.release(msg);
                // cancelled
                return;
            }
        } catch (RuntimeException e) {
            ReferenceCountUtil.release(msg);
            throw e;
        }
        //首先先找handler
        final AbstractChannelHandlerContext next = findContextOutbound(flush ?
                (MASK_WRITE | MASK_FLUSH) : MASK_WRITE);
        final Object m = pipeline.touch(msg, next);
        EventExecutor executor = next.executor();
        if (executor.inEventLoop()) {
            if (flush) {
                next.invokeWriteAndFlush(m, promise);
            } else {
                next.invokeWrite(m, promise);
            }
        } else {
            final WriteTask task = WriteTask.newInstance(next, m, promise, flush);
            if (!safeExecute(executor, task, promise, m, !flush)) {
                // We failed to submit the WriteTask. We need to cancel it so we decrement the pending bytes
                // and put it back in the Recycler for re-use later.
                //
                // See https://github.com/netty/netty/issues/8343.
                task.cancel();
            }
        }
    }

第一步先找handler，可以看到这里找到的next handler是headContext，然后继续往下，走到invoke方法，断点进去

 一路往下走之后来到了io.netty.channel.AbstractChannel.AbstractUnsafe#write这个方法，我们看到了ChannelOutboundBuffer，再继续往下走，走到后面的io.netty.channel.ChannelOutboundBuffer#addMessage，断点进去，看到了添加entry到队尾的过程，再往下继续走到incrementPendingOutboundBytes方法，断点进去

public void addMessage(Object msg, int size, ChannelPromise promise) {
        Entry entry = Entry.newInstance(msg, size, total(msg), promise);
        if (tailEntry == null) {
            flushedEntry = null;
        } else {
            Entry tail = tailEntry;
            tail.next = entry;
        }
        //可以看到entry加到了队尾
        tailEntry = entry;
        //如果没有数据的话这个entry就是个unflushedEntry
        if (unflushedEntry == null) {
            unflushedEntry = entry;
        }

        // increment pending bytes after adding message to the unflushed arrays.
        // See https://github.com/netty/netty/issues/1619
        incrementPendingOutboundBytes(entry.pendingSize, false);
    }

private void incrementPendingOutboundBytes(long size, boolean invokeLater) {
        if (size == 0) {
            return;
        }

        long newWriteBufferSize = TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, size);
        //如果写入的buffer大于最高水位线，就把可写标志置为false
        if (newWriteBufferSize > channel.config().getWriteBufferHighWaterMark()) {
            setUnwritable(invokeLater);
        }
    }

走到这里写入的过程就结束了，接下来就是flush准备发送了，把断点打在EchoServerHandler的channelReadComplete方法上，跟进去，来到io.netty.channel.AbstractChannelHandlerContext#flush方法

@Override
    public ChannelHandlerContext flush() {
        //先找下一个handler
        final AbstractChannelHandlerContext next = findContextOutbound(MASK_FLUSH);
        EventExecutor executor = next.executor();
        if (executor.inEventLoop()) {
            next.invokeFlush();
        } else {
            Tasks tasks = next.invokeTasks;
            if (tasks == null) {
                next.invokeTasks = tasks = new Tasks(next);
            }
            safeExecute(executor, tasks.invokeFlushTask, channel().voidPromise(), null, false);
        }

        return this;
    }

继续走到io.netty.channel.AbstractChannel.AbstractUnsafe#flush这个方法，接着往下走到io.netty.channel.ChannelOutboundBuffer#addFlush方法，打断点进入

//这个方法就是把unflush的数据转到flushed里面去
public void addFlush() {
        // There is no need to process all entries if there was already a flush before and no new messages
        // where added in the meantime.
        //
        // See https://github.com/netty/netty/issues/2577
        Entry entry = unflushedEntry;
        if (entry != null) {
            if (flushedEntry == null) {
                // there is no flushedEntry yet, so start with the entry
                flushedEntry = entry;
            }
            do {
                flushed ++;
                if (!entry.promise.setUncancellable()) {
                    // Was cancelled so make sure we free up memory and notify about the freed bytes
                    int pending = entry.cancel();
                    decrementPendingOutboundBytes(pending, false, true);
                }
                entry = entry.next;
            } while (entry != null);

            // All flushed so reset unflushedEntry
            unflushedEntry = null;
        }
    }

接着再看后面的flush0方法

protected void flush0() {
            if (inFlush0) {
                // Avoid re-entrance
                return;
            }

            final ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
            if (outboundBuffer == null || outboundBuffer.isEmpty()) {
                return;
            }

            inFlush0 = true;

            // Mark all pending write requests as failure if the channel is inactive.
            if (!isActive()) {
                try {
                    // Check if we need to generate the exception at all.
                    if (!outboundBuffer.isEmpty()) {
                        if (isOpen()) {
                            outboundBuffer.failFlushed(new NotYetConnectedException(), true);
                        } else {
                            // Do not trigger channelWritabilityChanged because the channel is closed already.
                            outboundBuffer.failFlushed(newClosedChannelException(initialCloseCause, "flush0()"), false);
                        }
                    }
                } finally {
                    inFlush0 = false;
                }
                return;
            }

            try {
                //写多次的方法
                doWrite(outboundBuffer);
            } catch (Throwable t) {
                if (t instanceof IOException && config().isAutoClose()) {
                    /**
                     * Just call {@link #close(ChannelPromise, Throwable, boolean)} here which will take care of
                     * failing all flushed messages and also ensure the actual close of the underlying transport
                     * will happen before the promises are notified.
                     *
                     * This is needed as otherwise {@link #isActive()} , {@link #isOpen()} and {@link #isWritable()}
                     * may still return {@code true} even if the channel should be closed as result of the exception.
                     */
                    initialCloseCause = t;
                    close(voidPromise(), t, newClosedChannelException(t, "flush0()"), false);
                } else {
                    try {
                        shutdownOutput(voidPromise(), t);
                    } catch (Throwable t2) {
                        initialCloseCause = t;
                        close(voidPromise(), t2, newClosedChannelException(t, "flush0()"), false);
                    }
                }
            } finally {
                inFlush0 = false;
            }
        }

断点进入dowrite方法，我们看到了这行

 这个maxBytesPerGatheringWrite就是根据之前写入的数据来算出最多可以写多少数据，而且每次都会变化。然后下面的nioBuffer数组就由这个maxBytesPerGatheringWrite来做大小控制，再接下来，就是写的过程，直接由jdk的socketChannel来实现，接着往下走到io.netty.channel.socket.nio.NioSocketChannel#adjustMaxBytesPerGatheringWrite方法，这个方法会重新分配maxBytesPerGatheringWrite大小

 private void adjustMaxBytesPerGatheringWrite(int attempted, int written, int oldMaxBytesPerGatheringWrite) {
        // By default we track the SO_SNDBUF when ever it is explicitly set. However some OSes may dynamically change
        // SO_SNDBUF (and other characteristics that determine how much data can be written at once) so we should try
        // make a best effort to adjust as OS behavior changes.
        //如果写入的和尝试写的大小一样，maxBytesPerGatheringWrite扩大一倍
        if (attempted == written) {
            if (attempted << 1 > oldMaxBytesPerGatheringWrite) {
                ((NioSocketChannelConfig) config).setMaxBytesPerGatheringWrite(attempted << 1);
            }
        } else if (attempted > MAX_BYTES_PER_GATHERING_WRITE_ATTEMPTED_LOW_THRESHOLD && written < attempted >>> 1) {
            ((NioSocketChannelConfig) config).setMaxBytesPerGatheringWrite(attempted >>> 1);
        }
    }

然后我们会看到一个io.netty.channel.ChannelOutboundBuffer#removeBytes方法，这个方法会把已经写出去的数据删掉一些，还没有写出去的数据会执行一个progess方法， 记录一下进度，随后会把可写次数减一。

由于我们测试的数据写不到16次，所以有一段代码没有走到但是还是要看一下那就是io.netty.channel.nio.AbstractNioByteChannel#incompleteWrite，这段代码会在写满16次之后还要继续写的情况下执行，它会直接新起一个task去执行写操作而不是直接注册OP_WRITE事件，当在没有写到16次的时候，要继续写的话，就要注册OP_WRITE事件，就是setOpWrite方法

protected final void incompleteWrite(boolean setOpWrite) {
        // Did not write completely.
        if (setOpWrite) {
            setOpWrite();
        } else {
            // It is possible that we have set the write OP, woken up by NIO because the socket is writable, and then
            // use our write quantum. In this case we no longer want to set the write OP because the socket is still
            // writable (as far as we know). We will find out next time we attempt to write if the socket is writable
            // and set the write OP if necessary.
            clearOpWrite();

            // Schedule flush again later so other tasks can be picked up in the meantime
            eventLoop().execute(flushTask);
        }
    }

protected final void setOpWrite() {
        final SelectionKey key = selectionKey();
        // Check first if the key is still valid as it may be canceled as part of the deregistration
        // from the EventLoop
        // See https://github.com/netty/netty/issues/2104
        if (!key.isValid()) {
            return;
        }
        final int interestOps = key.interestOps();
        if ((interestOps & SelectionKey.OP_WRITE) == 0) {
            //注册OP_WRITE事件
            key.interestOps(interestOps | SelectionKey.OP_WRITE);
        }
    }

总结

这一讲，我们看到了netty是怎么进行发送数据的，知道了三种写的方式，以及写次数的规定，高水位线的设置。