Hadoop RPC 源码阅读2

前面分析了ipc.Server是如何处理客户端的RPC请求，下面来分析客户端如何向服务端发送请求。

ipc.Client源码分析

ipc.Client有几个内部类，包括Call、Connection、ConnectionID等

首先看Client类的call方法：

  
  /** Make a call, passing <code>param</code>, to the IPC server defined by
   * <code>remoteId</code>, returning the value.  
   * Throws exceptions if there are network problems or if the remote code 
   * threw an exception. */
  public Writable call(Writable param, ConnectionId remoteId)  
                       throws InterruptedException, IOException {
    Call call = new Call(param);              // 将请求传入的数据封装为一个Call对象
    Connection connection = getConnection(remoteId, call); // 为该客户端创建一个连接，后面对代码具体分析
    connection.sendParam(call);                 // send the parameter 向服务端发送call对象，后面对代码继续分析
    boolean interrupted = false;                 // 标志是否因中断异常而终止，是为TRUE，否则为FALSE
    synchronized (call) {
      while (!call.done) {
        try {
          call.wait();// wait for the result请求对象在此阻塞，当Client收到服务端的返回值时被唤醒，见后文对Connection.receiveResponse方法的分析
        } catch (InterruptedException ie) {
          // save the fact that we were interrupted
          interrupted = true;       // 中断异常而终止
        }
      }

      if (interrupted) {
        // set the interrupt flag now that we are done waiting
        Thread.currentThread().interrupt();
      }

      if (call.error != null) {
        if (call.error instanceof RemoteException) {
          call.error.fillInStackTrace();
          throw call.error;
        } else { // local exception 本地异常
          // use the connection because it will reflect an ip change, unlike
          // the remoteId
          throw wrapException(connection.getRemoteAddress(), call.error);
        }
      } else {
        return call.value; // 返回结果数据
      }
    }
  }

Client的getConnection（）方法：

  /** Get a connection from the pool, or create a new one and add it to the
   * pool.  Connections to a given ConnectionId are reused. */
   private Connection getConnection(ConnectionId remoteId,
                                   Call call)
                                   throws IOException, InterruptedException {
    if (!running.get()) { // running：AtomicBoolean,以原子方式更新的Boolean值，get方法返回当前值，FALSE则表示客户端stop了
      // the client is stopped
      throw new IOException("The client is stopped");
    }
    Connection connection;
    /* we could avoid this allocation for each RPC by having a  
     * connectionsId object and with set() method. We need to manage the
     * refs for keys in HashMap properly. For now its ok.
     */
    do {
      synchronized (connections) {
        connection = connections.get(remoteId);
        if (connection == null) { // 如果没有该客户端的连接，则为其创建一个新的连接，并将这个连接放进hashTable里；如果有则直接从hashTable中get到
          connection = new Connection(remoteId);   // 注意：这里只是储存了remoteId的信息，并没有真正和服务端建立连接
          connections.put(remoteId, connection);
        }
      }
    } while (!connection.addCall(call));// 将这个call对象放入connection的call池中（hashTable），并且notify服务端的一个listener
    
    //we don't invoke the method below inside "synchronized (connections)"
    //block above. The reason for that is if the server happens to be slow,
    //it will take longer to establish a connection and that will slow the
    //entire system down.
    connection.setupIOstreams(); // 真正和服务端建立连接
    return connection;
  }

一个call获得了一个connection，并且加入了connection的call队列中去之后，这个connection如何和服务端建立真正的连接呢？

Client.Connection的setupIOstreams()方法，Connection是Client的一个内部类，继承自Thread：

    /** Connect to the server and set up the I/O streams. It then sends
     * a header to the server and starts
     * the connection thread that waits for responses.
     */
    private synchronized void setupIOstreams() throws InterruptedException {
      if (socket != null || shouldCloseConnection.get()) {
        return;
      }
     
      try {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Connecting to "+server);
        }
        short numRetries = 0;
        final short maxRetries = 15;
        Random rand = null;
        while (true) {
          setupConnection(); // 建立连接
          InputStream inStream = NetUtils.getInputStream(socket); // 获取输入流
          OutputStream outStream = NetUtils.getOutputStream(socket); // 获取输出流
          writeRpcHeader(outStream);
          if (useSasl) {
            final InputStream in2 = inStream;
            final OutputStream out2 = outStream;
            UserGroupInformation ticket = remoteId.getTicket();
            if (authMethod == AuthMethod.KERBEROS) {
              if (ticket.getRealUser() != null) {
                ticket = ticket.getRealUser();
              }
            }
            boolean continueSasl = false;
            try { 
              continueSasl = 
                ticket.doAs(new PrivilegedExceptionAction<Boolean>() {
                  @Override
                  public Boolean run() throws IOException {
                    return setupSaslConnection(in2, out2);
                  }
                }); 
            } catch (Exception ex) {
              if (rand == null) {
                rand = new Random();
              }
              handleSaslConnectionFailure(numRetries++, maxRetries, ex, rand,
                   ticket);
              continue;
            }
            if (continueSasl) {
              // Sasl connect is successful. Let's set up Sasl i/o streams.
              inStream = saslRpcClient.getInputStream(inStream);
              outStream = saslRpcClient.getOutputStream(outStream);
            } else {
              // fall back to simple auth because server told us so.
              authMethod = AuthMethod.SIMPLE;
              header = new ConnectionHeader(header.getProtocol(),
                  header.getUgi(), authMethod);
              useSasl = false;
            }
          }
          this.in = new DataInputStream(new BufferedInputStream
              (new PingInputStream(inStream))); // 封装输入流
          this.out = new DataOutputStream
          (new BufferedOutputStream(outStream)); // 封装输出流
          writeHeader();

          // update last activity time
          touch(); 

          // start the receiver thread after the socket connection has been set up
          start(); // socket连接建立起来之后，启动接收者线程。
          return;
        }
      } catch (Throwable t) {
        if (t instanceof IOException) {
          markClosed((IOException)t);
        } else {
          markClosed(new IOException("Couldn't set up IO streams", t));
        }
        close();
      }
    }

看一下steupConnection（）如何建立一个连接：

    
    private synchronized void setupConnection() throws IOException {
      short ioFailures = 0;
      short timeoutFailures = 0;
      while (true) {
        try {
          this.socket = socketFactory.createSocket(); // socketFactory创建一个socket
          this.socket.setTcpNoDelay(tcpNoDelay);
          
          /*
           * Bind the socket to the host specified in the principal name of the
           * client, to ensure Server matching address of the client connection
           * to host name in principal passed.
           */
          if (UserGroupInformation.isSecurityEnabled()) {
            KerberosInfo krbInfo = 
              remoteId.getProtocol().getAnnotation(KerberosInfo.class);
            if (krbInfo != null && krbInfo.clientPrincipal() != null) {
              String host = 
                SecurityUtil.getHostFromPrincipal(remoteId.getTicket().getUserName());
              
              // If host name is a valid local address then bind socket to it
              InetAddress localAddr = NetUtils.getLocalInetAddress(host);
              if (localAddr != null) {
                this.socket.bind(new InetSocketAddress(localAddr, 0));
              }
            }
          }
          
          // connection time out is 20s 设置连接超时时间：20s
          NetUtils.connect(this.socket, server, 20000); 
          if (rpcTimeout > 0) {
            pingInterval = rpcTimeout;  // rpcTimeout overwrites pingInterval
          }

          this.socket.setSoTimeout(pingInterval);
          return;
        } catch (SocketTimeoutException toe) {
          /* Check for an address change and update the local reference.
           * Reset the failure counter if the address was changed
           */
          if (updateAddress()) {
            timeoutFailures = ioFailures = 0;
          }
          /* The max number of retries is 45, 最多连接重试45次，共20s*45 = 15min重试时间
           * which amounts to 20s*45 = 15 minutes retries.
           */
          handleConnectionFailure(timeoutFailures++, 45, toe);
        } catch (IOException ie) {
          if (updateAddress()) {
            timeoutFailures = ioFailures = 0;
          }
          handleConnectionFailure(ioFailures++, maxRetries, ie);
        }
      }
    }

上面的分析可知，客户端为请求建立了一个连接，连接建立好后，发送请求数据，Connection.sendParam()方法：

    /** Initiates a call by sending the parameter to the remote server.
     * Note: this is not called from the Connection thread, but by other
     * threads.
     */
    public void sendParam(Call call) {
      if (shouldCloseConnection.get()) {
        return;
      }

      DataOutputBuffer d=null;
      try {
        synchronized (this.out) {
          if (LOG.isDebugEnabled())
            LOG.debug(getName() + " sending #" + call.id);
          
          //for serializing the
          //data to be written
          d = new DataOutputBuffer();
          d.writeInt(call.id);
          call.param.write(d);
          byte[] data = d.getData();
          int dataLength = d.getLength();
          out.writeInt(dataLength);      //first put the data length 向服务端发送数据的长度
          out.write(data, 0, dataLength);//write the data 向服务端发送数据
          out.flush();
        }
      } catch(IOException e) {
        markClosed(e);
      } finally {
        //the buffer is just an in-memory buffer, but it is still polite to
        // close early
        IOUtils.closeStream(d);
      }
    }  

至此，客户端将向服务端的请求发送出去了。下面看，如何接收服务端的响应。

由源码值Connection是继承自Thread，在Connection.setupIOstreams()方法中，在封装好输入输出流，准备发送数据前，调用了start()方法，启动了这个connection线程，

会执行Connection的run方法，在run方法中：

public void run() {
      if (LOG.isDebugEnabled())
        LOG.debug(getName() + ": starting, having connections " 
            + connections.size());

      while (waitForWork()) {//wait here for work - read or close connection 在此等待服务端的回应
        receiveResponse(); // 对回应具体的处理操作
      }
      
      close();
      
      if (LOG.isDebugEnabled())
        LOG.debug(getName() + ": stopped, remaining connections "
            + connections.size());
    }

Connection的receiveResponse方法：

    /* Receive a response.
     * Because only one receiver, so no synchronization on in.
     */
    private void receiveResponse() {
      if (shouldCloseConnection.get()) {
        return;
      }
      touch();
      
      try {
        int id = in.readInt();                    // try to read an id 读取id

        if (LOG.isDebugEnabled())
          LOG.debug(getName() + " got value #" + id);

        Call call = calls.get(id); // 根据id从call池中找到发送时的那个call对象

        int state = in.readInt();     // read call status 读取call的状态
        if (state == Status.SUCCESS.state) {
          Writable value = ReflectionUtils.newInstance(valueClass, conf);
          value.readFields(in);                 // read value 读取数据
          call.setValue(value); //将读取的数据值赋给那个call对象，同时唤醒Client等待线程（setValue中调用allCompete方法，allCompete中使用notify）
          calls.remove(id); // 从call池中将已经处理过的call对象删除掉
        } else if (state == Status.ERROR.state) {
          call.setException(new RemoteException(WritableUtils.readString(in),
                                                WritableUtils.readString(in)));
          calls.remove(id);
        } else if (state == Status.FATAL.state) {
          // Close the connection
          markClosed(new RemoteException(WritableUtils.readString(in), 
                                         WritableUtils.readString(in)));
        }
      } catch (IOException e) {
        markClosed(e);
      }
    }

Call中的setValue和callCompete方法，Call是Client的一个内部类：

    
    /** Set the return value when there is no error. 
     * Notify the caller the call is done.
     * 
     * @param value return value of the call.
     */
    public synchronized void setValue(Writable value) {
      this.value = value;
      callComplete();
    }

    /** Indicate when the call is complete and the
     * value or error are available.  Notifies by default.  */
    protected synchronized void callComplete() {
      this.done = true;
      notify();                                 // notify caller 唤醒
    }

总结：

Client将请求传入的数据封装成一个Call对象，并为其建立一个连接connection，然后使用这个connection将这个请求对象call发送给服务端。一个Client中可以建立多个connection，每个connection有一个唯一的connectionID, 使用hashtable来管理。一个连接connection可以发送多个请求Call对象，使用hashtable来管理。

当Client为一个call建立一个connection并将call发送给服务端之后，会阻塞，等待服务端的响应。connection在发送一个call请求时会启动一个接收者线程来接收服务端的回应。当connection收到服务端对这个call请求的响应数据之后，将响应数据赋给这个call，同时唤醒Client。

       这样客户端就获取了服务端响应返回的数据。

///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

用户的请求是如何传给Client的？在前文提到ipc.Server实际上是一个抽象类，他的实现在ipc.RPC.Server中，并已经分析了RPC.Server的部分源码，现在回到RPC类中继续分析。