mycat2.0源码分析01-接受客户端连接并发送握手报文

本文链接：https://blog.youkuaiyun.com/zhuchuangang/article/details/71486850

mycat接受客户端连接并发送握手报文
1 NIOAcceptor的实现
2 NIOReactor建立连接
3 mycat向客户端发送握手报文
4 NIOReactor写事件处理
- 41 调用Connection的doWriteQueue方法向客户端写数据

1 mycat接受客户端连接并发送握手报文

这里写图片描述

在reactor模型，acceptor线程负责接受TCP连接请求。acceptor的实现类是io.mycat.net2.NIOAcceptor。mycat将socket包装成前端连接MySQLFrontConnection，connection持有一个session，这个session中包含这个前端connection所有的后端的mysql客户端连接和sql处理器，同时mycat会将前端的connection添加到一个前端连接池中。

reactor线程负责处理IO请求，reactor的实现类是io.mycat.net2.NIOReactor。reactor获取SelectionKey所绑定的连接Connnection，调用Connnection中的asynRead()和doWriteQueue()方法，来进行读写操作。

1.1 NIOAcceptor的实现

下面讲解上图1~3的过程。

1.1.1 NIOAcceptor的初始化

public final class NIOAcceptor extends Thread {
    private static final Logger LOGGER = LoggerFactory.getLogger(NIOAcceptor.class);
    private final int port;
    private final Selector selector;
    private final ServerSocketChannel serverChannel;
    private final ConnectionFactory factory;
    private long acceptCount;
    private final NIOReactorPool reactorPool;

public NIOAcceptor(String name, String bindIp, int port,ConnectionFactory factory, NIOReactorPool reactorPool)
            throws IOException {
        //设置acceptor线程名称
        super.setName(name);
        //指定端口号
        this.port = port;
        //给acceptor分配选择器
        this.selector = Selector.open();
        //打开一个server-socket channel
        this.serverChannel = ServerSocketChannel.open();
        //server-socket channel设置为非阻塞
        this.serverChannel.configureBlocking(false);
        //设置TCP属性,重用端口
        serverChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true);
        //设置TCP属性,设置接收缓存
        serverChannel.setOption(StandardSocketOptions.SO_RCVBUF, 1024 * 16 * 2);
        // backlog=100 等待连接的最大数量
        serverChannel.bind(new InetSocketAddress(bindIp, port), 100);
        //注册OP_ACCEPT事件
        this.serverChannel.register(selector, SelectionKey.OP_ACCEPT);
        //设置连接工厂类，此处是MySQLFrontendConnectionFactory
        this.factory = factory;
        //设置reactorPool
        this.reactorPool = reactorPool;
    }

1.1.2 NIOAcceptor线程的run方法

run方法中for无限循环，selector.select(1000L)注册器设置1秒超时返回准备就绪的SelectionKey,如果SelectionKey有效且关注的是OP_ACCEPT事件，那么注册连接，否则取消SelectionKey，最后在finally块中清空所有key

@Override
    public void run() {
        final Selector selector = this.selector;
        for (;;) {
            ++acceptCount;
            try {
                selector.select(1000L);
                Set<SelectionKey> keys = selector.selectedKeys();
                try {
                    for (SelectionKey key : keys) {
                        //判断key是否有效，是否是acceptor事件
                        if (key.isValid() && key.isAcceptable()) {
                            //注册连接
                            accept();
                        } else {
                            //取消key
                            key.cancel();
                        }
                    }
                } finally {
                    //清空selectedKey
                    keys.clear();
                }
            } catch (Throwable e) {
                LOGGER.warn(getName(), e);
            }
        }
    }

1.1.3 NIOAcceptor的accept方法

通过accept()获取SocketChannel，设置SocketChannel为非阻塞模式，包装成前段连接MySQLFrontConnection，设置必要参数，获取一个reactor线程，并将前段连接MySQLFrontConnection注册到该reactor线程中进行读写操作。

private void accept() {
        SocketChannel channel = null;
        try {
            //如果有channel可以返回，accept()方法是阻塞的
            channel = serverChannel.accept();
            //设置channel为非阻塞
            channel.configureBlocking(false);
            //将channel包装成connection
            Connection c = factory.make(channel);
            c.setDirection(Connection.Direction.in);
            c.setId(ConnectIdGenerator.getINSTNCE().getId());
            InetSocketAddress remoteAddr = (InetSocketAddress) channel.getRemoteAddress();
            c.setHost(remoteAddr.getHostString());
            c.setPort(remoteAddr.getPort());
            //派发此连接到某个Reactor处理
            NIOReactor reactor = reactorPool.getNextReactor();
            //reactor注册此连接
            reactor.postRegister(c);

        } catch (Throwable e) {
            //关闭channel
            closeChannel(channel);
            LOGGER.warn(getName(), e);
        }
    }

1.1.4 NIOAcceptor的closeChannel方法

先关闭socket，再关闭channel

private static void closeChannel(SocketChannel channel) {
        if (channel == null) {
            return;
        }
        Socket socket = channel.socket();
        if (socket != null) {
            try {
                socket.close();
            } catch (IOException e) {
            }
        }
        try {
            channel.close();
        } catch (IOException e) {
        }
    }

1.2 NIOReactor建立连接

下面讲解上图4~7的过程。

1.2.1 NIOReactor的初始化

NIOReactor初始化的时候设置bufferpool和一个读写线程RWThread,所有的IO操作实际上上在这个线程中处理的。

public final class NIOReactor {
    private static final Logger LOGGER = LoggerFactory.getLogger(NIOReactor.class);
    private final String name;
    private final RWThread reactorR;
    private final SharedBufferPool shearedBufferPool;

    public NIOReactor(String name, SharedBufferPool shearedBufferPool) throws IOException {
        this.name = name;
        this.shearedBufferPool = shearedBufferPool;
        this.reactorR = new RWThread(name+"-RW");
    }

1.2.2 NIOReactor的启动

NIOReactor的启动实际上是启动RWThread线程。

final void startup() {
        reactorR.start();
    }

1.2.3 RWThread的初始化

RWThread的初始化时，会获取Selector实例，并初始化一个连接注册队列。

private final class RWThread extends Thread {
        private final Selector selector;
        private final ConcurrentLinkedQueue<Connection> registerQueue;
        private long reactCount;
        private final ReactorBufferPool myBufferPool;
        private java.util.concurrent.CopyOnWriteArrayList<Runnable> events=new CopyOnWriteArrayList<Runnable>();
        private RWThread(String name) throws IOException {
            this.setName(name);
            this.selector = Selector.open();
            myBufferPool = new ReactorBufferPool(shearedBufferPool, this, 1000);
            this.registerQueue = new ConcurrentLinkedQueue<Connection>();
        }

1.2.4 RWThread的run方法

当selector.select()方法获取的就绪事件为0，那么调用handlerEvents()方法（上图所示5）注册连接。如果有读写事件就绪，那么通过获取SelectionKey绑定的连接Connection，调用Connection的相应的读写方法。本文介绍首次建立连接，我们重点看handlerEvents()方法。

@Override
        public void run() {
            final Selector selector = this.selector;
            Set<SelectionKey> keys = null;
            int readys=0;
            for (;;) {
                ++reactCount;
                try {
                    readys=selector.select(400/(readys+1));
                    if(readys==0)
                    {
                        handlerEvents(selector);
                        continue;
                    }
                    keys = selector.selectedKeys();
                    for (final SelectionKey key : keys) {
                        Connection con = null;
                        try {
                            final Object att = key.attachment();
                            LOGGER.debug("select-key-readyOps = {}, attachment = {}", key.readyOps(), att);
                            if (att != null && key.isValid()) {
                                con = (Connection) att;
                                if (key.isReadable()) {
                                    try {
                                        con.asynRead();
                                    } catch (Throwable e) {
                                        if (!(e instanceof java.io.IOException)) {
                                            LOGGER.warn("caught err: " + con, e);
                                        }
                                        con.close("program err:" + e.toString());
                                        continue;
                                    }
                                }
                                // "key" may be cancelled in asynRead()!
                                // @author little-pan
                                // @since 2016-09-29
                                if(key.isValid() == false){
                                    LOGGER.debug("select-key cancelled");
                                    continue;
                                }
                                if (key.isWritable()) {
                                    con.doWriteQueue();
                                }
                            } else {
                                key.cancel();
                            }
                        } catch (final Throwable e) {
                            if (e instanceof CancelledKeyException) {
                                if (LOGGER.isDebugEnabled()) {
                                    LOGGER.debug(con + " socket key canceled");
                                }
                            } else {
                                LOGGER.warn(con + " " + e);
                            }

                        }

                    }

                } catch (Throwable e) {
                    LOGGER.warn(name, e);
                } finally {
                    if (keys != null) {
                        keys.clear();
                    }
                }
                //handler other events
                handlerEvents(selector);
            }
        }

1.2.4 RWThread的handlerEvents方法

当selector.select()方法获取的就绪事件为0，那么调用handlerEvents()方法（上图所示5）注册连接。在register方法(上图所示6)中，循环注册队列registerQueue中的连接。

private void handlerEvents(Selector selector)
        {
            try
            {
            processEvents();
            register(selector);
            }catch(Exception e)
            {
                LOGGER.warn("caught user event err:",e);
            }
        }

private void register(Selector selector) {

            if (registerQueue.isEmpty()) {
                return;
            }
            Connection c = null;
            while ((c = registerQueue.poll()) != null) {
                try {
                    c.register(selector, myBufferPool);
                } catch (Throwable e) {
                    LOGGER.warn("register error ", e);
                    c.close("register err");
                }
            }
        }

1.3 mycat向客户端发送握手报文

1.3.1 调用Connection的register方法

如上图7，Connection类的register方法给绑定的channel注册读事件，并且将该前段连接添加到连接池。为这个连接类初始化读/写buffer，最后调用MySQLFrontConnectionHandler的onConnected()方法发送握手报文。

public void register(Selector selector, ReactorBufferPool myBufferPool) throws IOException {
        processKey = channel.register(selector, SelectionKey.OP_READ, this);
        NetSystem.getInstance().addConnection(this);
       // boolean isLinux=GenelUtil.isLinuxSystem();
        //String maprFileName=isLinux? "/dev/zero":id+".rtmp";
        //String mapwFileName=isLinux? "/dev/zero":id+".wtmp";
        String maprFileName=id+".rtmp";
        String mapwFileName=id+".wtmp";
        LOGGER.info("connection bytebuffer mapped "+maprFileName);

        //TODO
        /**使用MyCatMemoryAllocator分配Direct Buffer,再进行SocketChannel通信时候，
         * 网络读写都会减少一次数据的拷贝,而使用FileChanel与SocketChannel数据交换时
         * 底层最终还是生成一个临时的Direct Buffer，用临时Direct Buffer写入或者读SocketChannel中
         * 后面考虑会使用netty中ByteBuf中的DirectBuffer进行网络IO通信。效率更高
         * */
        this.readDataBuffer =new MappedFileConDataBuffer(maprFileName); // 2 ,3
        this.writeDataBuffer=new MappedFileConDataBuffer3(mapwFileName);
        //存在bug暂不启用，以后统一是ByteBuf作为buffer进行NIO网络通信。
       // this.readDataBuffer = new  ByteBufConDataBuffer(4096,16*1024*1024);
       // this.writeDataBuffer = new ByteBufConDataBuffer(4096,16*1024*1024);
        //新的client进来后，处理Server发送Client的handshake init packet
        this.handler.onConnected(this);
    }

1.3.2 调用MySQLFrontConnectionHandler的onConnected方法

如上图8，前段连接的session设置登录验证回调，并且发送客户端发送握手报文。

    public void onConnected(MySQLFrontConnection con) throws IOException {
        LOGGER.debug("onConnected(): {}", con);
        con.getSession().changeCmdHandler(loginCmdHandler);
        con.sendAuthPackge();
    }

1.3.3 调用MySQLFrontConnection的sendAuthPackge方法

如上图10，MySQLFrontConnection的sendAuthPackge方法组装HandshakePacket报文，调用MySQLFrontConnection的writeMsqlPackage()方法把报文发送给客户端。

public void sendAuthPackge() throws IOException {
            // 生成认证数据
            byte[] rand1 = RandomUtil.randomBytes(8);
            byte[] rand2 = RandomUtil.randomBytes(12);

            // 保存认证数据
            byte[] seed = new byte[rand1.length + rand2.length];
            System.arraycopy(rand1, 0, seed, 0, rand1.length);
            System.arraycopy(rand2, 0, seed, rand1.length, rand2.length);
            this.seed = seed;

            // 发送握手数据包
            HandshakePacket hs = new HandshakePacket();
            hs.packetId = 0;
            hs.protocolVersion = Versions.PROTOCOL_VERSION;
            hs.serverVersion = Versions.SERVER_VERSION;
            hs.threadId = id;
            hs.seed = rand1;
            hs.serverCapabilities = getServerCapabilities();
            // hs.serverCharsetIndex = (byte) (charsetIndex & 0xff);
            hs.serverStatus = 2;
            hs.restOfScrambleBuff = rand2;
            this.writeMsqlPackage(hs);
            // asynread response
           // this.asynRead();
        }

1.3.4 调用MySQLFrontConnection的writeMsqlPackage方法

如上图11~13，将握手报文写入MySQLFrontConnection的writeDataBuffer，并调用this.enableWrite(true)唤醒NIOReactor的selector，处理写事件。

MySQLFrontConnection的writeMsqlPackage方法：

    public void writeMsqlPackage(MySQLPacket pkg) throws IOException
    {
        int pkgSize=pkg.calcPacketSize();
        ByteBuffer buf=getWriteDataBuffer().beginWrite(pkgSize+MySQLPacket.packetHeaderSize);
        pkg.write(buf,pkgSize);
        getWriteDataBuffer().endWrite(buf);
        this.enableWrite(true);
    }

Connection的enableWrite方法：

public void enableWrite(boolean wakeup) {
        boolean needWakeup = false;
        try {
            SelectionKey key = this.processKey;
            key.interestOps(key.interestOps() | SelectionKey.OP_WRITE);
            needWakeup = true;
        } catch (Exception e) {
            LOGGER.warn("can't enable write " + e);

        }
        if (needWakeup && wakeup) {
            processKey.selector().wakeup();
        }
    }

1.4 NIOReactor写事件处理

NIOReactor类的RWThread类中调用con.doWriteQueue()放写数据

1.4.1 调用Connection的doWriteQueue()方法向客户端写数据

如上图14~17，NIOReactor的RWThread的run中，当有写事件触发时，调用doWriteQueue()方法向客户端写数据。

@Override
        public void run() {
            final Selector selector = this.selector;
            Set<SelectionKey> keys = null;
            int readys=0;
            for (;;) {
                ++reactCount;
                try {
                    readys=selector.select(400/(readys+1));
                    if(readys==0)
                    {
                        handlerEvents(selector);
                        continue;
                    }
                    keys = selector.selectedKeys();
                    for (final SelectionKey key : keys) {
                        Connection con = null;
                        try {
                            final Object att = key.attachment();
                            LOGGER.debug("select-key-readyOps = {}, attachment = {}", key.readyOps(), att);
                            if (att != null && key.isValid()) {
                                con = (Connection) att;
                                if (key.isReadable()) {
                                    try {
                                        con.asynRead();
                                    } catch (Throwable e) {
                                        if (!(e instanceof java.io.IOException)) {
                                            LOGGER.warn("caught err: " + con, e);
                                        }
                                        con.close("program err:" + e.toString());
                                        continue;
                                    }
                                }
                                // "key" may be cancelled in asynRead()!
                                // @author little-pan
                                // @since 2016-09-29
                                if(key.isValid() == false){
                                    LOGGER.debug("select-key cancelled");
                                    continue;
                                }
                                if (key.isWritable()) {
                                    con.doWriteQueue();
                                }
                            } else {
                                key.cancel();
                            }
                        } catch (final Throwable e) {
                            if (e instanceof CancelledKeyException) {
                                if (LOGGER.isDebugEnabled()) {
                                    LOGGER.debug(con + " socket key canceled");
                                }
                            } else {
                                LOGGER.warn(con + " " + e);
                            }

                        }

                    }

                } catch (Throwable e) {
                    LOGGER.warn(name, e);
                } finally {
                    if (keys != null) {
                        keys.clear();
                    }
                }
                //handler other events
                handlerEvents(selector);
            }
        }

Connection的doWriteQueue()方法：

public void doWriteQueue() {
        try {
            boolean noMoreData = write0();
            lastWriteTime = TimeUtil.currentTimeMillis();
            if (noMoreData) {
                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) != 0)) {
                    disableWrite();
                }

            } else {

                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) == 0)) {
                    enableWrite(false);
                }
            }

        } catch (IOException e) {
            if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("caught err:", e);
            }
            close("err:" + e);
        }

    }

Connection的write0()方法：

private boolean write0() throws IOException {
        final NetSystem nets = NetSystem.getInstance();
        final ConDataBuffer buffer = this.writeDataBuffer;
        final int written = buffer.transferTo(this.channel);
        final int remains = buffer.writingPos() - buffer.readPos();
        netOutBytes += written;
        nets.addNetOutBytes(written);
        // trace-protocol
        // @author little-pan
        // @since 2016-09-29
        if(nets.getNetConfig().isTraceProtocol()){
            final String hexs = StringUtil.dumpAsHex(buffer, buffer.readPos() - written, written);
            LOGGER.info("C#{}B#{}: last writed = {} bytes, remain to write = {} bytes, written bytes\n{}", 
                getId(), buffer.hashCode(), written, remains, hexs);
        }
        return (remains == 0);
    }