netty开发tcp服务器最好不要用分隔符DelimiterBasedFrameDecoder这种分包方式

• 我们都知道，tcp通讯属于流传输，对于上面承载的业务协议栈是不做分包处理的，所以大量客户端给服务器发送数据，就会有黏包现象，所以必须分包，反之，服务器给客户端发数据，也会黏包。
• netty提供了很多decoder用来分包，目前个人觉得效率最高，最好的方式还是LengthFieldBasedFrameDecoder，没有之一。
• 很多人刚开始做开发经验不足，按照教科书上的指点，使用了分隔符作为分包机制，其实这种方式效率非常低，比较“愚蠢”，不管是服务器还是客户端，因为你作为接收端，不知道tcp流里面什么时候分隔符到来，所以必须一个字节一个字节去和分隔符做对比，有的人说，那有什么，netty都做好了我们只管用就好了，既然你说这种方式不好，为什么netty还有这种方式呢？这就是杠精附体，根本不懂所以然的那些人。
• 不信可以参考一下DelimiterBasedFrameDecoder里面的decode实现，看看netty是不是用了for循环，是不是用了indexOf在一个字节一个字节和分隔符做对比，有遍历，有对比，就消耗CPU时间，消耗服务器性能，在今天网络如此发达，不只是一个客户端，可是成千上万的客户端。

    protected Object decode(ChannelHandlerContext ctx, ByteBuf buffer) throws Exception {
        if (lineBasedDecoder != null) {
            return lineBasedDecoder.decode(ctx, buffer);
        }
        // Try all delimiters and choose the delimiter which yields the shortest frame.
        int minFrameLength = Integer.MAX_VALUE;
        ByteBuf minDelim = null;
        for (ByteBuf delim: delimiters) {
            int frameLength = indexOf(buffer, delim);
            if (frameLength >= 0 && frameLength < minFrameLength) {
                minFrameLength = frameLength;
                minDelim = delim;
            }
        }

        if (minDelim != null) {
            int minDelimLength = minDelim.capacity();
            ByteBuf frame;

            if (discardingTooLongFrame) {
                // We've just finished discarding a very large frame.
                // Go back to the initial state.
                discardingTooLongFrame = false;
                buffer.skipBytes(minFrameLength + minDelimLength);

                int tooLongFrameLength = this.tooLongFrameLength;
                this.tooLongFrameLength = 0;
                if (!failFast) {
                    fail(tooLongFrameLength);
                }
                return null;
            }

            if (minFrameLength > maxFrameLength) {
                // Discard read frame.
                buffer.skipBytes(minFrameLength + minDelimLength);
                fail(minFrameLength);
                return null;
            }

            if (stripDelimiter) {
                frame = buffer.readRetainedSlice(minFrameLength);
                buffer.skipBytes(minDelimLength);
            } else {
                frame = buffer.readRetainedSlice(minFrameLength + minDelimLength);
            }

            return frame;
        } else {
            if (!discardingTooLongFrame) {
                if (buffer.readableBytes() > maxFrameLength) {
                    // Discard the content of the buffer until a delimiter is found.
                    tooLongFrameLength = buffer.readableBytes();
                    buffer.skipBytes(buffer.readableBytes());
                    discardingTooLongFrame = true;
                    if (failFast) {
                        fail(tooLongFrameLength);
                    }
                }
            } else {
                // Still discarding the buffer since a delimiter is not found.
                tooLongFrameLength += buffer.readableBytes();
                buffer.skipBytes(buffer.readableBytes());
            }
            return null;
        }
    }

• 既然分隔符这么多问题，为什么netty还保留有分隔符这种分包机制呢，原因就是这种分包方式非常适合那种每次通讯就十几个字节的这种业务，问题是现在很多tcp业务数据量动辄至少都是几个KB，还有几MB的那种，还用分隔符这种方式效率极低。
• 再看看LengthFieldBasedFrameDecoder分包是怎么实现的，他是使用了包头+包体这种分包方式，tcp接收端，无论服务端还是客户端，收到固定长度的包头，就从里面拿出了包体的长度，之后直接从netty的ByteBuf里面取出包体的长度内容即可，因为接收端是可以预知到包有多长的，什么时候读取完毕，分隔符分包，你并不知道接下来什么时候分隔符到来，只能一个字节一个字节去对比。
• 那有人说，就算你用这种包头+包体的方式，那发送端发送的数据包格式错误怎么办，会不会导致接收端出问题，那是肯定的了，计算机是机器，不是人，你要故意给它投毒，那它只能死，所以既然是通讯协议，双方必须按照统一规范来做，不是随意就去发数据。
• 举例，比如要设计一种客户端和服务端的tcp通讯协议，包头2个字节的业务代码，4个字节的长度，4个字节就有4GB的容量了，足够任何tcp业务使用了吧，应该没有人一次发送4GB的tcp包。无论是客户端还是服务器，每次接收数据，先接收固定的6个字节，解析出包体长度，之后再从tcp流中读取指定的长度即可，有人就会有疑问，那不需要CRC校验么，其实不需要，因为TCP的可靠性，不会像UART那样会受到电磁干扰导致数据出错。

单个tcp 数据包格式：

• 解释：上面展示了一种tcp通讯数据包的格式，完整的一个tcp数据包由包头和包体组成

单个tcp数据包包头格式：

• 解释：单个数据包的包头，由2 Byte的业务代码和4 Byte的包体长度组成，2个字节的业务码最多有65535种组合，可以满足大多数的应用需要了，可以表示登录认证、心跳、服务响应等等，接收端接收到固定的6字节包头，然后拿出业务码进行判断，如果符合要求再拿出4字节的包体长度，之后根据这个包体长度从缓冲区里面读取响应的长度即可，完全不需要像分隔符那样要一个字节一个字节的查找遍历。

多个tcp数据包在ByteBufer接收缓冲区中的排列方式是一个挨着一个的

• 解释：多个tcp数据包在tcp/ip协议栈里面是按照这种方式存储的，发送方一个接着一个发包，接收方的缓冲区里面是FIFO的方式进行存储，更不可能出现乱序的问题，因为tcp的丢包重传是协议栈内部已经实现了，用户层无需关心，那有人就会这样担心，万一因为网络原因导致数据包不完整怎么办，这种情况不太可能，因为比如接收端接收到一半的时候，后续的数据包还没来，这时候tcp断掉了，那么缓冲区里面的数据肯定会丢掉的，上层接收也没收到一个完整的数据包更不会处理。

• 下面解释一下怎么使用netty的LengthFieldBasedFrameDecoder作为解码器和LengthFieldPrepender作为编码器，另外，不是所有的客户端都是和服务器一样是netty，很多资料解释都是拿netty的客户端和服务端做演示，我们拿mbedtls的客户端做演示，实现netty服务器和mbedtls客户端实现TLS加密通讯(为简单起见，当前展示的是没有证书验证的实现，需要证书双向验证的，可以参考其他教程)

先上netty的代码

• 以我们上面展示的这种tcp分包方式，由于标准的LengthFieldPrepender编码器有点傻，不能满足我们的要求，所以我们直接拷贝LengthFieldPrepender代码出来重命名，然后修改一下里面的encode的实现，之后加到pipeline里面让netty去调用
• 另外解码器LengthFieldBasedFrameDecoder我们就不需要继承，直接使用即可，很多教程都是演示怎么继承然后重新实现，不必这么费劲。
• netty server某奏云链接

再上mbedtls客户端代码

• 客户端给netty服务器发送10MB的字符串，然后netty将收到的字符串打印出来，之后回复给客户端，然后客户端收到并打印出来
• 下载mbedtls最新代码，直接替换进去，用VS2010编译在windows上测试即可

ssl_client1.c

/*
 *  SSL client demonstration program
 *
 *  Copyright (C) 2006-2015, ARM Limited, All Rights Reserved
 *  SPDX-License-Identifier: Apache-2.0
 *
 *  Licensed under the Apache License, Version 2.0 (the "License"); you may
 *  not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *  http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 *  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 *
 *  This file is part of mbed TLS (https://tls.mbed.org)
 */

#if !defined(MBEDTLS_CONFIG_FILE)
#include "mbedtls/config.h"
#else
#include MBEDTLS_CONFIG_FILE
#endif

#define MESSAGE_COUNT				(1024*1024*10)

#define BREAK_UINT32( var, ByteNum ) \
          (unsigned char)((unsigned int)(((var) >>((ByteNum) * 8)) & 0x00FF))

#define BUILD_UINT32(Byte0, Byte1, Byte2, Byte3) \
          ((unsigned int)((unsigned int)((Byte0) & 0x00FF) \
          + ((unsigned int)((Byte1) & 0x00FF) << 8) \
          + ((unsigned int)((Byte2) & 0x00FF) << 16) \
          + ((unsigned int)((Byte3) & 0x00FF) << 24)))

#define BUILD_UINT16(loByte, hiByte) \
          ((short)(((loByte) & 0x00FF) + (((hiByte) & 0x00FF) << 8)))

#define HI_UINT16(a) (((a) >> 8) & 0xFF)
#define LO_UINT16(a) ((a) & 0xFF)

#define BUILD_UINT8(hiByte, loByte) \
          ((unsigned char)(((loByte) & 0x0F) + (((hiByte) & 0x0F) << 4)))

#define HI_UINT8(a) (((a) >> 4) & 0x0F)
#define LO_UINT8(a) ((a) & 0x0F)

#if defined(MBEDTLS_PLATFORM_C)
#include "mbedtls/platform.h"
#else
#include <stdio.h>
#include <stdlib.h>
#define mbedtls_time            time
#define mbedtls_time_t          time_t
#define mbedtls_fprintf         fprintf
#define mbedtls_printf          printf
#define mbedtls_exit            exit
#define MBEDTLS_EXIT_SUCCESS    EXIT_SUCCESS
#define MBEDTLS_EXIT_FAILURE    EXIT_FAILURE
#endif /* MBEDTLS_PLATFORM_C */

#if !defined(MBEDTLS_BIGNUM_C) || !defined(MBEDTLS_ENTROPY_C) ||  \
    !defined(MBEDTLS_SSL_TLS_C) || !defined(MBEDTLS_SSL_CLI_C) || \
    !defined(MBEDTLS_NET_C) || !defined(MBEDTLS_RSA_C) ||         \
    !defined(MBEDTLS_CERTS_C) || !defined(MBEDTLS_PEM_PARSE_C) || \
    !defined(MBEDTLS_CTR_DRBG_C) || !defined(MBEDTLS_X509_CRT_PARSE_C)
int main( void )
{
    mbedtls_printf("MBEDTLS_BIGNUM_C and/or MBEDTLS_ENTROPY_C and/or "
           "MBEDTLS_SSL_TLS_C and/or MBEDTLS_SSL_CLI_C and/or "
           "MBEDTLS_NET_C and/or MBEDTLS_RSA_C and/or "
           "MBEDTLS_CTR_DRBG_C and/or MBEDTLS_X509_CRT_PARSE_C "
           "not defined.\n");
    mbedtls_exit( 0 );
}
#else

#include "mbedtls/net_sockets.h"
#include "mbedtls/debug.h"
#include "mbedtls/ssl.h"
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/error.h"
#include "mbedtls/certs.h"

#include <string.h>

#define SERVER_PORT "8992"
#define SERVER_NAME "localhost"

#define DEBUG_LEVEL 1

static void my_debug( void *ctx, int level,
                      const char *file, int line,
                      const char *str )
{
    ((void) level);

    mbedtls_fprintf( (FILE *) ctx, "%s:%04d: %s", file, line, str );
    fflush(  (FILE *) ctx  );
}

int main( void )
{
    int ret = 1, len;
	unsigned char *buffer=NULL;
	unsigned char *json=NULL;
    int exit_code = MBEDTLS_EXIT_FAILURE;
    mbedtls_net_context server_fd;
    uint32_t flags;
	unsigned char msgBuf[6];
	int packSize=0;
	int packCount=0;
	int packRemain=0;
    const char *pers = "ssl_client1";
	unsigned char *buf=NULL;
	short msgHeader=0;
	int index=0,number=0;
	int msgLength=0;
    mbedtls_entropy_context entropy;
    mbedtls_ctr_drbg_context ctr_drbg;
    mbedtls_ssl_context ssl;
    mbedtls_ssl_config conf;
    mbedtls_x509_crt cacert;

#if defined(MBEDTLS_DEBUG_C)
    mbedtls_debug_set_threshold( DEBUG_LEVEL );
#endif

    /*
     * 0. Initialize the RNG and the session data
     */
    mbedtls_net_init( &server_fd );
    mbedtls_ssl_init( &ssl );
    mbedtls_ssl_config_init( &conf );
    mbedtls_x509_crt_init( &cacert );
    mbedtls_ctr_drbg_init( &ctr_drbg );

    mbedtls_printf( "\n  . Seeding the random number generator..." );
    fflush( stdout );

    mbedtls_entropy_init( &entropy );
    if( ( ret = mbedtls_ctr_drbg_seed( &ctr_drbg, mbedtls_entropy_func, &entropy,
                               (const unsigned char *) pers,
                               strlen( pers ) ) ) != 0 )
    {
        mbedtls_printf( " failed\n  ! mbedtls_ctr_drbg_seed returned %d\n", ret );
        goto exit;
    }

    mbedtls_printf( " ok\n" );

    /*
     * 0. Initialize certificates
     */
    mbedtls_printf( "  . Loading the CA root certificate ..." );
    fflush( stdout );

    ret = mbedtls_x509_crt_parse( &cacert, (const unsigned char *) mbedtls_test_cas_pem,
                          mbedtls_test_cas_pem_len );
    if( ret < 0 )
    {
        mbedtls_printf( " failed\n  !  mbedtls_x509_crt_parse returned -0x%x\n\n", (unsigned int) -ret );
        goto exit;
    }

    mbedtls_printf( " ok (%d skipped)\n", ret );
    /*
     * 1. Start the connection
     */
    mbedtls_printf( "  . Connecting to tcp/%s/%s...", SERVER_NAME, SERVER_PORT );
    fflush( stdout );

    if( ( ret = mbedtls_net_connect( &server_fd, SERVER_NAME,
                                         SERVER_PORT, MBEDTLS_NET_PROTO_TCP ) ) != 0 )
    {
        mbedtls_printf( " failed\n  ! mbedtls_net_connect returned %d\n\n", ret );
        goto exit;
    }

    mbedtls_printf( " ok\n" );

    /*
     * 2. Setup stuff
     */
    mbedtls_printf( "  . Setting up the SSL/TLS structure..." );
    fflush( stdout );

    if( ( ret = mbedtls_ssl_config_defaults( &conf,
                    MBEDTLS_SSL_IS_CLIENT,
                    MBEDTLS_SSL_TRANSPORT_STREAM,
                    MBEDTLS_SSL_PRESET_DEFAULT ) ) != 0 )
    {
        mbedtls_printf( " failed\n  ! mbedtls_ssl_config_defaults returned %d\n\n", ret );
        goto exit;
    }

    mbedtls_printf( " ok\n" );

    /* OPTIONAL is not optimal for security,
     * but makes interop easier in this simplified example */
    mbedtls_ssl_conf_authmode( &conf, MBEDTLS_SSL_VERIFY_NONE );
    mbedtls_ssl_conf_ca_chain( &conf, &cacert, NULL );
    mbedtls_ssl_conf_rng( &conf, mbedtls_ctr_drbg_random, &ctr_drbg );
    mbedtls_ssl_conf_dbg( &conf, my_debug, stdout );

    if( ( ret = mbedtls_ssl_setup( &ssl, &conf ) ) != 0 )
    {
        mbedtls_printf( " failed\n  ! mbedtls_ssl_setup returned %d\n\n", ret );
        goto exit;
    }

    if( ( ret = mbedtls_ssl_set_hostname( &ssl, SERVER_NAME ) ) != 0 )
    {
        mbedtls_printf( " failed\n  ! mbedtls_ssl_set_hostname returned %d\n\n", ret );
        goto exit;
    }

    mbedtls_ssl_set_bio( &ssl, &server_fd, mbedtls_net_send, mbedtls_net_recv, NULL );

    /*
     * 4. Handshake
     */
    mbedtls_printf( "  . Performing the SSL/TLS handshake...\n" );
    fflush( stdout );

    while( ( ret = mbedtls_ssl_handshake( &ssl ) ) != 0 )
    {
        if( ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE )
        {
            mbedtls_printf( " failed\n  ! mbedtls_ssl_handshake returned -0x%x\n\n", (unsigned int) -ret );
            goto exit;
        }
    }

    mbedtls_printf( " ok\n" );

    /*
     * 5. Verify the server certificate
     */
    mbedtls_printf( "  . Verifying peer X.509 certificate..." );

    /* In real life, we probably want to bail out when ret != 0 */
    if( ( flags = mbedtls_ssl_get_verify_result( &ssl ) ) != 0 )
    {
        char vrfy_buf[512];

        mbedtls_printf( " failed\n" );

        mbedtls_x509_crt_verify_info( vrfy_buf, sizeof( vrfy_buf ), "  ! ", flags );

        mbedtls_printf( "%s\n", vrfy_buf );
		goto exit;
    }
    else
        mbedtls_printf( " ok\n" );

    /*
     * 3. Write the GET request
     */
    mbedtls_printf( "  > Write to server:\n" );
    fflush( stdout );

	buffer=(unsigned char *)mbedtls_calloc(1,MESSAGE_COUNT);
	if(buffer == NULL)
	{
		mbedtls_printf( "memory malloc failure\n" );
		goto exit;
	}

	memset(buffer,0,MESSAGE_COUNT);

	msgHeader=0x2345;
	number='A';
	for(index=0;index<(MESSAGE_COUNT-1);index++)
	{
		buffer[index]=(unsigned char )number;
		number++;
		if(number>'Z')
		{
			number='A';
		}
	}

	msgLength=strlen((char *)buffer);
	msgBuf[0]=HI_UINT16(msgHeader);
	msgBuf[1]=LO_UINT16(msgHeader);
	msgBuf[2]=BREAK_UINT32(msgLength,3);
	msgBuf[3]=BREAK_UINT32(msgLength,2);
	msgBuf[4]=BREAK_UINT32(msgLength,1);
	msgBuf[5]=BREAK_UINT32(msgLength,0);

	mbedtls_printf( "write package header 0x%X %d bytes\n", msgHeader,msgLength);
	ret = mbedtls_ssl_write( &ssl,(unsigned char *) msgBuf, sizeof(msgBuf));
	if(ret <=0)
	{
            mbedtls_printf( " failed\n  ! mbedtls_ssl_write returned %d\n\n", ret );
            goto exit;
	}
	len =0;

	packSize=1024;
	packCount=msgLength/packSize;
	packRemain=msgLength%packSize;
	mbedtls_printf( "write package body %d bytes packCount=%d packRemain=%d\n", msgLength,packCount,packRemain);
	if(packCount > 0)
	{
		do
		{
			ret = mbedtls_ssl_write( &ssl, buffer+len, packSize);
			if( ret < 0 )
			{
				mbedtls_printf( " failed\n  ! mbedtls_ssl_write returned %d\n\n", ret );
				goto exit;
			}
			len+=ret;
		}while(len <= packCount*packSize);
	}

	mbedtls_printf( " mbedtls_ssl_write %d bytes\n", len);

	if(packRemain > 0)
	{
			ret = mbedtls_ssl_write( &ssl, buffer+len, packRemain);
			if( ret < 0 )
			{
				mbedtls_printf( " failed\n  ! mbedtls_ssl_write returned %d\n\n", ret );
				goto exit;
			}
	}

    mbedtls_printf( " %d bytes written\n", len);
	mbedtls_free(buffer);

    do
    {
        ret = mbedtls_ssl_read( &ssl, msgBuf, sizeof(msgBuf));
		if(ret <= 0)
		{
			continue;
		}

		msgHeader=BUILD_UINT16(msgBuf[1],msgBuf[0]);
		msgLength=BUILD_UINT32(msgBuf[5],msgBuf[4],msgBuf[3],msgBuf[2]);
		len=0;
		buf=(unsigned char *)mbedtls_calloc(1,msgLength+1);
		if(buf == NULL)
		{
			mbedtls_printf( "memory failure\n" );
			goto exit;
		}

		do 
		{
			ret = mbedtls_ssl_read( &ssl, buf+len, 1024);
			if(ret > 0)
			{
				len+=ret;
			}else{
				mbedtls_printf( "\n\nEOF\n\n" );
				break;
			}
		}while(len <msgLength);

        mbedtls_printf( "0x%X %d bytes read\n\n%s\n", msgHeader,msgLength,(char *)buf);
		break;
    }while( 1 );

    mbedtls_ssl_close_notify( &ssl );

    exit_code = MBEDTLS_EXIT_SUCCESS;

exit:

#ifdef MBEDTLS_ERROR_C
    if( exit_code != MBEDTLS_EXIT_SUCCESS )
    {
        char error_buf[100];
        mbedtls_strerror( ret, error_buf, 100 );
        mbedtls_printf("Last error was: %d - %s\n\n", ret, error_buf );
    }
#endif

    mbedtls_net_free( &server_fd );

    mbedtls_x509_crt_free( &cacert );
    mbedtls_ssl_free( &ssl );
    mbedtls_ssl_config_free( &conf );
    mbedtls_ctr_drbg_free( &ctr_drbg );
    mbedtls_entropy_free( &entropy );

#if defined(_WIN32)
    mbedtls_printf( "  + Press Enter to exit this program.\n" );
    fflush( stdout ); getchar();
#endif

    mbedtls_exit( exit_code );
	system("pause");
}
#endif /* MBEDTLS_BIGNUM_C && MBEDTLS_ENTROPY_C && MBEDTLS_SSL_TLS_C &&
          MBEDTLS_SSL_CLI_C && MBEDTLS_NET_C && MBEDTLS_RSA_C &&
          MBEDTLS_CERTS_C && MBEDTLS_PEM_PARSE_C && MBEDTLS_CTR_DRBG_C &&
          MBEDTLS_X509_CRT_PARSE_C */