Hadoop RPC源码分析

最新推荐文章于 2025-08-10 16:33:07 发布

转载最新推荐文章于 2025-08-10 16:33:07 发布 · 402 阅读

文章标签：

#hadoop #server #服务器 #thread #socket #header

本文深入探讨了Hadoop RPC的工作机制，包括客户端和服务器端的交互流程，详细介绍了连接建立、数据传输及响应处理的过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

[java] view plain copy

Hadoop RPC ：与传统的RPC程序相似，分为Server端和Client端二个部分，下面将分别介绍其工作机制
1.Client端分析：
|-->Client(Class<? extends Writable> valueClass, Configuration conf, SocketFactory factory)
|-->分别设置 maxIdleTime、tcpNoDelay、pingInterval值
|-->this.conf = conf; ｜设置configuation
|-->this.socketFactory = factory; ｜设定socketFactory仓库，用于socket连接
|-->call(Writable param, InetSocketAddress addr,Class<?> protocol, UserGroupInformation ticket) |与服务器建立连接
|-->Call call = new Call(param); ｜new一个Call对象，Call对象对于在RPC中作为基础的传输单元
|-->Connection connection = getConnection(addr, protocol, ticket, call);
|-->connection.sendParam(call); ｜发送call数据
|--> while (!call.done)
|--> call.wait(); |client端此处作同步处理，Server端实际为异步处理，在Response的setValue后被notify()唤醒
1.1 获取连接方法
|-->getConnection |从连接池里面获取一个连接对象，没有则创建一个
|-->ConnectionId remoteId = new ConnectionId(addr, protocol, ticket); ｜创建连接id，避免重复分配connectionID
|-->connection = new Connection(remoteId); ｜利用conncetionId创建Connectin连接
|-->connections.put(remoteId, connection); ｜缓存进连接池当中,Hashtable予以保存
|-->connection.setupIOstreams(); ｜在这此真正建立连接
1.1.1 connection.setupIOstreams() ｜与服务器建立连接
|-->this.socket = socketFactory.createSocket();
|-->this.socket.setTcpNoDelay(tcpNoDelay);
|-->NetUtils.connect(this.socket, remoteId.getAddress(), 20000); ｜用NIO的方式建立连接
|-->SocketChannel ch = socket.getChannel(); ｜获取通道
|-->socket.connect(endpoint, timeout); ｜建立连接
|-->this.in = new DataInputStream(new BufferedInputStream ｜分别设置输入输出流
(new PingInputStream(NetUtils.getInputStream(socket))));
|-->this.out = new DataOutputStream
(new BufferedOutputStream(NetUtils.getOutputStream(socket)));
|-->writeHeader(); ｜写Connection的头信息
|-->touch(); ｜更新最后活动时间，同步等待responser进行处理，client模式为同步机制
|-->start(); ｜connection继承了Thread，实际上执行了run()方法
|-->run()
|-->while (waitForWork()) ｜等待服务器连接端响应，或者关闭连接
|-->receiveResponse();
|-->close() ｜最后关闭
1.2 connection.sendParam(call); ｜发送数据
|-->d = new DataOutputBuffer();
|-->d.writeInt(call.id); ｜首先写个call_id,用于标致唯一Call
|-->call.param.write(d); ｜将Call中的Writable对象写入DataoutputStream中
|-->byte[] data = d.getData();
|-->int dataLength = d.getLength(); ｜获取data的长度
|-->out.writeInt(dataLength); ｜往通道中先写长度，再写数据
|-->out.write(data, 0, dataLength);
|-->out.flush(); ｜flush，使写入生效
|-->IOUtils.closeStream(d); ｜利用IOUtils关闭流
1.3 receiveResponse() ｜接收服务器端响应，处理请求。
|-->int id = in.readInt();
|-->Call call = calls.get(id);
|-->int state = in.readInt(); ｜如果为success状态，则读取in流
|-->Writable value = ReflectionUtils.newInstance(valueClass, conf); |利用反射生成一个Writable对象
|-->value.readFields(in);
|-->this.value = value;
|-->callComplete();
|-->this.done = true;
|-->notify(); |此时用于唤醒在client端call()调用时的call.wait()
|-->call.setValue(value);
|-->calls.remove(id);
2.Server端分析：
2.1:namenode初始化Server端
|-->initialize() |开启RPC的服务端
|-->this.server = RPC.getServer(this, socAddr.getHostName(), socAddr.getPort(),
handlerCount, false, conf);
|-->this.server.start();
2.1.1:RPC.getServer |RPC的静态类Server是Server服务器类的子类
|-->new Server(instance, conf, bindAddress, port, numHandlers, verbose);
|-->super(bindAddress, port, Invocation.class, numHandlers, conf, classNameBase(instance.getClass().getName()));
2.1.1.1:构建Server端类
|-->listener = new Listener(); ｜用于监听client端信息
|-->responder = new Responder(); ｜用于处理监听请求
2.2:this.server.start() ｜启动server端服务，分别启动srepsonder,listener和handler类
|-->responder.start(); ｜注意启动顺序，确保监听器开始时即可处理响应请求
|-->listener.start();
|-->handlers = new Handler[handlerCount]; ｜handlerCount为handler线程个数
|-->for (int i = 0; i < handlerCount; i++)
|-->handlers[i] = new Handler(i);
|-->handlers[i].start();
2.3:Listener监听器
2.3.1：构建Listener()
|-->address = new InetSocketAddress(bindAddress, port);
|-->acceptChannel = ServerSocketChannel.open();
|-->acceptChannel.configureBlocking(false);
|-->bind(acceptChannel.socket(), address, backlogLength);
|-->selector= Selector.open();
|-->acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
|-->this.setDaemon(true); ｜设置为后台守护进程
注：上述构建过程实际是简单的NIO server端启动过程，当client端请求过多时会出现瓶颈，后续版本增加了reader
|-->readers = new Reader[readThreads];
|-->readPool = Executors.newFixedThreadPool(readThreads); |设定reader线程池大小
|-->for (int i = 0; i < readThreads; i++)
|-->Selector readSelector = Selector.open();
|-->Reader reader = new Reader(readSelector);
|-->readers[i] = reader;
|-->readPool.execute(reader);
2.3.2:Listener线程run()执行
|-->while (running) |无限轮循
|-->selector.select();
|-->terator<SelectionKey> iter = selector.selectedKeys().iterator();
|-->while (iter.hasNext())
|-->key = iter.next();
|-->iter.remove();
|-->if (key.isAcceptable())
|-->doAccept(key); ｜往通道里面写数据
|-->else if (key.isReadable())
|-->doRead(key); ｜从通道里面读取数据
2.3.3:doAccept(key) ｜接受client请求
|-->ServerSocketChannel server = (ServerSocketChannel) key.channel();
|-->for (int i=0; i<10; i++) ｜开启最多10个线程进行处理
|-->SocketChannel channel = server.accept();
|-->channel.configureBlocking(false);
|-->SelectionKey readKey = channel.register(selector, SelectionKey.OP_READ); |此channel重新注册至selector中，进行读取操作
|-->c = new Connection(readKey, channel, System.currentTimeMillis());
|-->readKey.attach(c); ｜将connection附加到key当中,在is_readable中继续处理
2.3.4:doRead(key) ｜处理client请求
|-->Connection c = (Connection)key.attachment(); ｜在accept时作为attach封装了Connection信息
|-->count = c.readAndProcess();
|-->dataLengthBuffer |如果未满或已经为空，则读取了一个RPC，否则继续读
|-->channelRead(channel, dataLengthBuffer);
|-->int version = versionBuffer.get(0); ｜读取版本信息
|-->data = ByteBuffer.allocate(dataLength); ｜获取rpc大小
|-->if (headerRead) ｜判断headerRead是否读取，默认都会先发送header文件
|-->processData();
|-->else |先处理header头信息
|-->processHeader();
|-->closeConnection(c);
2.3.4.1 processHeader()
|-->header.readFields(in);
|-->String protocolClassName = header.getProtocol(); ｜从头文件中获取protocl的处理class
|-->protocol = getProtocolClass(header.getProtocol(), conf);
|-->user = SecurityUtil.getSubject(header.getUgi()); ｜获取user信息，作为安全策略
2.3.4.2 processData()
|-->int id = dis.readInt();
|-->Writable param = ReflectionUtils.newInstance(paramClass, conf); |paramClass此时为namenode的class
|-->param.readFields(dis);
|-->Call call = new Call(id, param, this); |在服务器端构建Call单元，并加入等待处理队列
|-->callQueue.put(call); ｜加入callQueue队列后，工作将交由handler进行处理
2.4:Handler处理器
2.4.1 构建Handler处理器 ,简单设置了Thread信息
|-->this.setDaemon(true);
|-->this.setName("IPC Server handler "+ instanceNumber + " on " + port);
2.4.2 run()线程执行
|-->while (running)
|-->final Call call = callQueue.take();
|-->CurCall.set(call); ｜确保当前处理的call唯一
|-->value = Subject.doAs(call.connection.user, new PrivilegedExceptionAction<Writable>() {
@Override
public Writable run() throws Exception
return call(call.connection.protocol, call.param, call.timestamp);｝
|-->setupResponse(buf, call, (error == null) ? Status.SUCCESS : Status.ERROR,
value, errorClass, error); ｜此时设置call的response信息
|-->responder.doRespond(call); ｜调用responder响应处理完成的call请求
2.4.3 执行call方法调用
call(call.connection.protocol, call.param, call.timestamp)
|-->Invocation call = (Invocation)param; |此时param对象预先读取了method方法名及其参数
|-->Method method = protocol.getMethod(call.getMethodName(),
call.getParameterClasses());
|-->method.setAccessible(true);
|-->Object value = method.invoke(instance, call.getParameters()); ｜利用反射获取方法调用的值
|-->return new ObjectWritable(method.getReturnType(), value); ｜最后返回一个ObjectWritable的处理结果
2.4.4 responder.doRespond(call) ｜属于responder的处理策略，提前分析
|-->call.connection.responseQueue.addLast(call);
|-->call.connection.responseQueue == 1 ｜call的responseQueue队列为1，则处理call单元
|-->processResponse(call.connection.responseQueue, true); ｜下节分析
2.5 Responser处理器
2.5.1 processResponse(responseQueue，inHandler) |handler处理完call数据后，传递给Responser
|-->call = responseQueue.removeFirst();
|-->SocketChannel channel = call.connection.channel;
|-->int numBytes = channelWrite(channel, call.response);
|-->if (!call.response.hasRemaining()) |此时response中已经不存在call对象了
|-->call.connection.decRpcCount();
|-->else
|-->call.connection.responseQueue.addFirst(call);
|-->writeSelector.wakeup(); ｜激活响应通道
|-->channel.register(writeSelector, SelectionKey.OP_WRITE, call);
2.5.2 run() ｜Thread执行方法
|-->waitPending(); ｜等待wake up唤醒
|-->writeSelector.select(PURGE_INTERVAL); ｜设置超时
|-->Iterator<SelectionKey> iter = writeSelector.selectedKeys().iterator();
|-->while (iter.hasNext())
|-->SelectionKey key = iter.next();
|-->iter.remove();
|--> doAsyncWrite(key);
|-->calls = new ArrayList<Call>(writeSelector.keys().size());
|-->while (iter.hasNext())
|-->Call call = (Call)key.attachment();
|-->calls.add(call);
|--> for(Call call : calls)
|-->doPurge(call, now); ｜清除响应队列中长久未响应的call
2.5.3 doAsyncWrite(key)
|-->Call call = (Call)key.attachment();
|-->if (processResponse(call.connection.responseQueue, false)) ｜调用processResponse进行响应