HADOOP datanode SendBlock毛刺现象分析

这几天研究HDFS的datanode 和 client端的数据通信,发现目前datanode是使用NIO来做底层通信的.在datanode的DataXceiver的readBlock方法就是将datanode的block中的数据发送到客户端或者其他的datanode中去.我们可以看以下几段代码.

  try {
      try {
        blockSender = new BlockSender(block, startOffset, length,
            true, true, false, datanode, clientTraceFmt);
      } catch(IOException e) {
        out.writeShort(DataTransferProtocol.OP_STATUS_ERROR);
        throw e;
      }

      out.writeShort(DataTransferProtocol.OP_STATUS_SUCCESS); // send op status
      long read = blockSender.sendBlock(out, baseStream, null); // send data

readBlock 在读取client的包头之后就开始发送相关的数据,发送是通过2个流,out就是普通的socket流,baseSteam是用于NIO的.在sendBlock中做了一些checksum计算和偏移操作后就调用sendchunks发送客户端所需要的data.此处如果transferToAllowed在配置文件中设置就是用sendfile方法加快发送流程,不然直接socket流来发送.

      if (transferToAllowed && !verifyChecksum && 
          baseStream instanceof SocketOutputStream && 
          blockIn instanceof FileInputStream) {
        
        FileChannel fileChannel = ((FileInputStream)blockIn).getChannel();
        
        // blockInPosition also indicates sendChunks() uses transferTo.
        blockInPosition = fileChannel.position();
        streamForSendChunks = baseStream;
......
 while (endOffset > offset) {
        long len = sendChunks(pktBuf, maxChunksPerPacket, 
                              streamForSendChunks);
        offset += len;
        totalRead += len + ((len + bytesPerChecksum - 1)/bytesPerChecksum*
                            checksumSize);
        seqno++;

在sendChunks方法中我们发现如果使用transferTo()的方法来NIO发送数据时候必须做一个wait 动作,注释的解释是此处避免一个jre的bug.

          //first write the packet
          sockOut.write(buf, 0, dataOff);
          // no need to flush. since we know out is not a buffered stream.
          sockOut.transferToFully(fileChannel, blockInPosition, len);
        }
---------------------------------------------
 public void transferToFully(FileChannel fileCh, long position, int count) 
                              throws IOException {
    
    while (count > 0) {
      /* 
       * Ideally we should wait after transferTo returns 0. But because of
       * a bug in JRE on Linux (http://bugs.sun.com/view_bug.do?bug_id=5103988),
       * which throws an exception instead of returning 0, we wait for the
       * channel to be writable before writing to it. If you ever see 
       * IOException with message "Resource temporarily unavailable" 
       * thrown here, please let us know.
       * 
       * Once we move to JAVA SE 7, wait should be moved to correct place.
       */
      waitForWritable();
      int nTransfered = (int) fileCh.transferTo(position, count, getChannel());

也就是这个waitForWritable();方法我们在用jstack dump信息时发现很多线程都block在这个地方,原因是由于下面file的transferTo动作在写入时要wait到这个channel是否可写.底层有个register和select的方法来做底层的等待,并有超时判断.

    /**
     * Waits on the channel with the given timeout using one of the 
     * cached selectors. It also removes any cached selectors that are
     * idle for a few seconds.
     * 
     * @param channel
     * @param ops
     * @param timeout
     * @return
     * @throws IOException
     */
    int select(SelectableChannel channel, int ops, long timeout) 
                                                   throws IOException {
     
      SelectorInfo info = get(channel);
      
      SelectionKey key = null;
      int ret = 0;
      
      try {
        while (true) {
          long start = (timeout == 0) ? 0 : System.currentTimeMillis();

          key = channel.register(info.selector, ops);// Step1
          ret = info.selector.select(timeout);//Step2
          
          if (ret != 0) {
            return ret;
          }
          
          /* Sometimes select() returns 0 much before timeout for 
           * unknown reasons. So select again if required.
           */
          if (timeout > 0) {
            timeout -= System.currentTimeMillis() - start;
            if (timeout <= 0) {
              return 0;
            }
          }

奇怪的是Step1和Step2之间等待的时间极度不均匀,在用btrace跟踪后发生分布的很散列.

                   timecost(千分之一毫秒)------- Distribution ------------- count
                    128 |                                         0
                    256 |                                         51895
                    512 |@@                                       243717
                   1024 |@@@@@@@@@@@@@                            1410085
                   2048 |@@@@@@@@@@@@@@@@@@@@                     2135480
                   4096 |@@                                       283689
                   8192 |                                         8247
                  16384 |                                         6856
                  32768 |                                         6741
                  65536 |                                         8390
                 131072 |                                         2019
                 262144 |                                         171
                 524288 |                                         21
                1048576 |                                         3
                2097152 |                                         0

有大量的毛刺产生.目前HDFS这个select的超时时间设置过长.默认为8分钟.可有下列参数改小到100毫秒做个实验.

<property> 
     <name>dfs.datanode.socket.write.timeout</name> 
     <value>1000</value> 
</property>

可以从以下直方图看出毛刺可以消除.

                  value  ------------- Distribution ------------- count
                    128 |                                         0
                    256 |                                         221
                    512 |                                         6896
                   1024 |@@@@@@@@@@@@@                            98509
                   2048 |@@@@@@@@@@@@@@@@@@@@@                    154922
                   4096 |@@@                                      23579
                   8192 |                                         1490
                  16384 |                                         438
                  32768 |                                         85
                  65536 |                                         5
                 131072 |                                         0

---------------------------------------------