前言
上一次我们说完了Netty进行业务处理的流程,我们知道Netty处理业务逻辑的本质就是在pipeline中所有的handler执行fireChannelRead的过程,当所有的fireChannelRead都执行完成后,就要将数据写出去,这一回我们一起来看一下Netty是怎么写数据的
开始
netty写数据一共有三种形式,分别是write,flush和writeAndFlush,write方法就是把数据写到一个buffer中,flush方法把buffer中的数据发出去,而writeAndFlush就是写数据到buffer之后立刻发送出去,在write和flush之间还有一个ChannelOutBoundBuffer。Netty写数据还有两个特点,一个是当数据写不进去的时候,会停止写,注册一个OP_WRITE事件,通知什么时候可以写进去了再继续写。还有一个是当Netty一次写入数据都能成功的时候,写一次写入,Netty会尝试写入更多的数据,这主要是靠调整maxBytesPerGatheringWrite的值来实现。Netty的连续写直到写不出去的尝试次数是16(writeSpinCount)。还有,Netty待写数据太多的时候,超过了writeBufferWaterMark.high()高水位线的时候,会将可写标志置为false,让应用端自己决定要不要发送数据。
接下来,我们就来一起看一下源码,首先我们把断点加到EchoServerHandler的channelRead方法里面,然后debug启动EchoServer,再启动EchoClient,然后一路走下去来到了这里
//io.netty.channel.AbstractChannelHandlerContext#write(java.lang.Object, boolean, io.netty.channel.ChannelPromise)
private void write(Object msg, boolean flush, ChannelPromise promise) {
ObjectUtil.checkNotNull(msg, "msg");
try {
if (isNotValidPromise(promise, true)) {
ReferenceCountUtil.release(msg);
// cancelled
return;
}
} catch (RuntimeException e) {
ReferenceCountUtil.release(msg);
throw e;
}
//首先先找handler
final AbstractChannelHandlerContext next = findContextOutbound(flush ?
(MASK_WRITE | MASK_FLUSH) : MASK_WRITE);
final Object m = pipeline.touch(msg, next);
EventExecutor executor = next.executor();
if (executor.inEventLoop()) {
if (flush) {
next.invokeWriteAndFlush(m, promise);
} else {
next.invokeWrite(m, promise);
}
} else {
final WriteTask task = WriteTask.newInstance(next, m, promise, flush);
if (!safeExecute(executor, task, promise, m, !flush)) {
// We failed to submit the WriteTask. We need to cancel it so we decrement the pending bytes
// and put it back in the Recycler for re-use later.
//
// See https://github.com/netty/netty/issues/8343.
task.cancel();
}
}
}
第一步先找handler,可以看到这里找到的next handler是headContext,然后继续往下,走到invoke方法,断点进去
一路往下走之后来到了io.netty.channel.AbstractChannel.AbstractUnsafe#write这个方法,我们看到了ChannelOutboundBuffer,再继续往下走,走到后面的io.netty.channel.ChannelOutboundBuffer#addMessage,断点进去,看到了添加entry到队尾的过程,再往下继续走到incrementPendingOutboundBytes方法,断点进去
public void addMessage(Object msg, int size, ChannelPromise promise) {
Entry entry = Entry.newInstance(msg, size, total(msg), promise);
if (tailEntry == null) {
flushedEntry = null;
} else {
Entry tail = tailEntry;
tail.next = entry;
}
//可以看到entry加到了队尾
tailEntry = entry;
//如果没有数据的话这个entry就是个unflushedEntry
if (unflushedEntry == null) {
unflushedEntry = entry;
}
// increment pending bytes after adding message to the unflushed arrays.
// See https://github.com/netty/netty/issues/1619
incrementPendingOutboundBytes(entry.pendingSize, false);
}
private void incrementPendingOutboundBytes(long size, boolean invokeLater) {
if (size == 0) {
return;
}
long newWriteBufferSize = TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, size);
//如果写入的buffer大于最高水位线,就把可写标志置为false
if (newWriteBufferSize > channel.config().getWriteBufferHighWaterMark()) {
setUnwritable(invokeLater);
}
}
走到这里写入的过程就结束了,接下来就是flush准备发送了,把断点打在EchoServerHandler的channelReadComplete方法上,跟进去,来到io.netty.channel.AbstractChannelHandlerContext#flush方法
@Override
public ChannelHandlerContext flush() {
//先找下一个handler
final AbstractChannelHandlerContext next = findContextOutbound(MASK_FLUSH);
EventExecutor executor = next.executor();
if (executor.inEventLoop()) {
next.invokeFlush();
} else {
Tasks tasks = next.invokeTasks;
if (tasks == null) {
next.invokeTasks = tasks = new Tasks(next);
}
safeExecute(executor, tasks.invokeFlushTask, channel().voidPromise(), null, false);
}
return this;
}
继续走到io.netty.channel.AbstractChannel.AbstractUnsafe#flush这个方法,接着往下走到io.netty.channel.ChannelOutboundBuffer#addFlush方法,打断点进入
//这个方法就是把unflush的数据转到flushed里面去
public void addFlush() {
// There is no need to process all entries if there was already a flush before and no new messages
// where added in the meantime.
//
// See https://github.com/netty/netty/issues/2577
Entry entry = unflushedEntry;
if (entry != null) {
if (flushedEntry == null) {
// there is no flushedEntry yet, so start with the entry
flushedEntry = entry;
}
do {
flushed ++;
if (!entry.promise.setUncancellable()) {
// Was cancelled so make sure we free up memory and notify about the freed bytes
int pending = entry.cancel();
decrementPendingOutboundBytes(pending, false, true);
}
entry = entry.next;
} while (entry != null);
// All flushed so reset unflushedEntry
unflushedEntry = null;
}
}
接着再看后面的flush0方法
protected void flush0() {
if (inFlush0) {
// Avoid re-entrance
return;
}
final ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
if (outboundBuffer == null || outboundBuffer.isEmpty()) {
return;
}
inFlush0 = true;
// Mark all pending write requests as failure if the channel is inactive.
if (!isActive()) {
try {
// Check if we need to generate the exception at all.
if (!outboundBuffer.isEmpty()) {
if (isOpen()) {
outboundBuffer.failFlushed(new NotYetConnectedException(), true);
} else {
// Do not trigger channelWritabilityChanged because the channel is closed already.
outboundBuffer.failFlushed(newClosedChannelException(initialCloseCause, "flush0()"), false);
}
}
} finally {
inFlush0 = false;
}
return;
}
try {
//写多次的方法
doWrite(outboundBuffer);
} catch (Throwable t) {
if (t instanceof IOException && config().isAutoClose()) {
/**
* Just call {@link #close(ChannelPromise, Throwable, boolean)} here which will take care of
* failing all flushed messages and also ensure the actual close of the underlying transport
* will happen before the promises are notified.
*
* This is needed as otherwise {@link #isActive()} , {@link #isOpen()} and {@link #isWritable()}
* may still return {@code true} even if the channel should be closed as result of the exception.
*/
initialCloseCause = t;
close(voidPromise(), t, newClosedChannelException(t, "flush0()"), false);
} else {
try {
shutdownOutput(voidPromise(), t);
} catch (Throwable t2) {
initialCloseCause = t;
close(voidPromise(), t2, newClosedChannelException(t, "flush0()"), false);
}
}
} finally {
inFlush0 = false;
}
}
断点进入dowrite方法,我们看到了这行
这个maxBytesPerGatheringWrite就是根据之前写入的数据来算出最多可以写多少数据,而且每次都会变化。然后下面的nioBuffer数组就由这个maxBytesPerGatheringWrite来做大小控制,再接下来,就是写的过程,直接由jdk的socketChannel来实现,接着往下走到io.netty.channel.socket.nio.NioSocketChannel#adjustMaxBytesPerGatheringWrite方法,这个方法会重新分配maxBytesPerGatheringWrite大小
private void adjustMaxBytesPerGatheringWrite(int attempted, int written, int oldMaxBytesPerGatheringWrite) {
// By default we track the SO_SNDBUF when ever it is explicitly set. However some OSes may dynamically change
// SO_SNDBUF (and other characteristics that determine how much data can be written at once) so we should try
// make a best effort to adjust as OS behavior changes.
//如果写入的和尝试写的大小一样,maxBytesPerGatheringWrite扩大一倍
if (attempted == written) {
if (attempted << 1 > oldMaxBytesPerGatheringWrite) {
((NioSocketChannelConfig) config).setMaxBytesPerGatheringWrite(attempted << 1);
}
} else if (attempted > MAX_BYTES_PER_GATHERING_WRITE_ATTEMPTED_LOW_THRESHOLD && written < attempted >>> 1) {
((NioSocketChannelConfig) config).setMaxBytesPerGatheringWrite(attempted >>> 1);
}
}
然后我们会看到一个io.netty.channel.ChannelOutboundBuffer#removeBytes方法,这个方法会把已经写出去的数据删掉一些,还没有写出去的数据会执行一个progess方法, 记录一下进度,随后会把可写次数减一。
由于我们测试的数据写不到16次,所以有一段代码没有走到但是还是要看一下那就是io.netty.channel.nio.AbstractNioByteChannel#incompleteWrite,这段代码会在写满16次之后还要继续写的情况下执行,它会直接新起一个task去执行写操作而不是直接注册OP_WRITE事件,当在没有写到16次的时候,要继续写的话,就要注册OP_WRITE事件,就是setOpWrite方法
protected final void incompleteWrite(boolean setOpWrite) {
// Did not write completely.
if (setOpWrite) {
setOpWrite();
} else {
// It is possible that we have set the write OP, woken up by NIO because the socket is writable, and then
// use our write quantum. In this case we no longer want to set the write OP because the socket is still
// writable (as far as we know). We will find out next time we attempt to write if the socket is writable
// and set the write OP if necessary.
clearOpWrite();
// Schedule flush again later so other tasks can be picked up in the meantime
eventLoop().execute(flushTask);
}
}
protected final void setOpWrite() {
final SelectionKey key = selectionKey();
// Check first if the key is still valid as it may be canceled as part of the deregistration
// from the EventLoop
// See https://github.com/netty/netty/issues/2104
if (!key.isValid()) {
return;
}
final int interestOps = key.interestOps();
if ((interestOps & SelectionKey.OP_WRITE) == 0) {
//注册OP_WRITE事件
key.interestOps(interestOps | SelectionKey.OP_WRITE);
}
}
总结
这一讲,我们看到了netty是怎么进行发送数据的,知道了三种写的方式,以及写次数的规定,高水位线的设置。