本文讨论的原理是,假设dubbo框架使用protocol是"dubbo",server和client底层传输数据是"netty"。
原理机制
远程调用时,发送请求的封装对象
com.alibaba.dubbo.remoting.exchange.Request
远程调用时,接收返回的封装对象
com.alibaba.dubbo.remoting.exchange.Response
请求发出后,返回一个com.alibaba.dubbo.remoting.exchange.ResponseFuture对象,
ResponseFuture的默认实现类com.alibaba.dubbo.remoting.exchange.support.DefaultFuture
在我们的业务层调用dubbo封装的service时,dubbo底层会调用com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker<T>的doInvoke方法:
public interface Invocation {
String getMethodName();
Class<?>[] getParameterTypes();
Object[] getArguments();
Map<String, String> getAttachments();
String getAttachment(String key);
String getAttachment(String key, String defaultValue);
Invoker<?> getInvoker();
}
可以看出,Invocation对象封装了消费者调用提供者的
类名: getInvoker()返回Invoker可以获得接口名;
方法名: getMethodName();
参数类型列表: getParameterTypes();
参数值列表: getArguments();
@Override
protected Result doInvoke(final Invocation invocation) throws Throwable {
RpcInvocation inv = (RpcInvocation) invocation;
final String methodName = RpcUtils.getMethodName(invocation);
inv.setAttachment(Constants.PATH_KEY, getUrl().getPath());
inv.setAttachment(Constants.VERSION_KEY, version);
ExchangeClient currentClient;
if (clients.length == 1) {
currentClient = clients[0];
} else {
currentClient = clients[index.getAndIncrement() % clients.length];
}
try {
boolean isAsync = RpcUtils.isAsync(getUrl(), invocation);
boolean isOneway = RpcUtils.isOneway(getUrl(), invocation);
int timeout = getUrl().getMethodParameter(methodName, Constants.TIMEOUT_KEY,Constants.DEFAULT_TIMEOUT);
if (isOneway) {
boolean isSent = getUrl().getMethodParameter(methodName, Constants.SENT_KEY, false);
currentClient.send(inv, isSent);
RpcContext.getContext().setFuture(null);
return new RpcResult();
} else if (isAsync) { //11处,调用服务的结果,异步返回模式。
ResponseFuture future = currentClient.request(inv, timeout) ; //12处,向提供者发送消息
RpcContext.getContext().setFuture(new FutureAdapter<Object>(future));
return new RpcResult();
} else {
RpcContext.getContext().setFuture(null);
return (Result) currentClient.request(inv, timeout).get(); //13处,调用服务的结果,同步返回结果
}
} catch (TimeoutException e) {
throw new RpcException(RpcException.TIMEOUT_EXCEPTION, "Invoke remote method timeout. method: " + invocation.getMethodName() + ", provider: " + getUrl() + ", cause: " + e.getMessage(), e);
} catch (RemotingException e) {
throw new RpcException(RpcException.NETWORK_EXCEPTION, "Failed to invoke remote method: " + invocation.getMethodName() + ", provider: " + getUrl() + ", cause: " + e.getMessage(), e);
}
}
如果调用服务,是异步返回结果,则是"11处",其中"12处"是向提供者发送服务消息。
如果调用服务,是同步返回结果,则是"13处",currentClient.request(inv, timeout)会向提供者发送消息,并返回ResponseFuture对象,然后调用ResponseFuture的get方法,返回结果;
这里的currentClient.request(inv, timeout),其实就是调用对象ReferenceCountExchangeClient.request()方法;
下面是调用dubbo服务的轨迹:
at com.alibaba.dubbo.remoting.transport.AbstractClient.send(AbstractClient.java:268)
at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request(HeaderExchangeChannel.java:112)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeClient.request(HeaderExchangeClient.java:91)
at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:81)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
追随轨迹,看到com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request方法:
public ResponseFuture request(Object request, int timeout) throws RemotingException {
if (closed) {
throw new RemotingException(this.getLocalAddress(), null, "Failed to send request " + request + ", cause: The channel " + this + " is closed!");
}
// create request.
Request req = new Request(); //14处,创建一个Request对象
req.setVersion("2.0.0");
req.setTwoWay(true);
req.setData(request);
DefaultFuture future = new DefaultFuture(channel, req, timeout); //15处,创建一个结果封装对象DefaultFuture
try{
channel.send(req);
}catch (RemotingException e) {
future.cancel();
throw e;
}
return future;
}
可以看到,"14处"会先创建一个Request对象,然后"15处"将Request对象放进DefaultFuture对象,DefaultFuture对象负责等待处理服务调用的返回消息。
先看看Request创建时的逻辑:
public class Request {
private static final AtomicLong INVOKE_ID = new AtomicLong(0);
private final long mId;
public Request() {
mId = newId();
}
private static long newId() {
// getAndIncrement()增长到MAX_VALUE时,再增长会变为MIN_VALUE,负数也可以做为ID
return INVOKE_ID.getAndIncrement();
}
}
而每个Request对象都会生成一个id,也就是说每个dubbo服务请求都对应一个id。
再看看DefaultFuture的内容结构:
下面看看ResponseFuture的实现类DefaultFuture
public class DefaultFuture implements ResponseFuture {
private static final Map<Long, Channel> CHANNELS = new ConcurrentHashMap<Long, Channel>();
private static final Map<Long, DefaultFuture> FUTURES = new ConcurrentHashMap<Long, DefaultFuture>();
private final Lock lock = new ReentrantLock();
private final Condition done = lock.newCondition();
private final Request request;
private volatile Response response;
public DefaultFuture(Channel channel, Request request, int timeout){
this.channel = channel;
this.request = request;
this.id = request.getId();
this.timeout = timeout > 0 ? timeout : channel.getUrl().getPositiveParameter(Constants.TIMEOUT_KEY, Constants.DEFAULT_TIMEOUT);
// put into waiting map.
FUTURES.put(id, this); //16处,将Request id和Request对象关联起来
CHANNELS.put(id, channel);
}
public Object get() throws RemotingException {
return get(timeout);
}
public Object get(int timeout) throws RemotingException {
if (timeout <= 0) {
timeout = Constants.DEFAULT_TIMEOUT;
}
if (! isDone()) {
long start = System.currentTimeMillis();
lock.lock();
try {
while (! isDone()) {
done.await(timeout, TimeUnit.MILLISECONDS); //17处,如果调用服务,提供者还没返回消息,则一直阻塞时长timeout。
if (isDone() || System.currentTimeMillis() - start > timeout) {
break;
}
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
lock.unlock();
}
if (! isDone()) {
throw new TimeoutException(sent > 0, channel, getTimeoutMessage(false));
}
}
return returnFromResponse(); //18处,返回dubbo服务调用结果
}
private Object returnFromResponse() throws RemotingException {
Response res = response;
if (res == null) {
throw new IllegalStateException("response cannot be null");
}
if (res.getStatus() == Response.OK) {
return res.getResult(); //19处,服务调用状态是OK,返回提供者返回的结果。
}
if (res.getStatus() == Response.CLIENT_TIMEOUT || res.getStatus() == Response.SERVER_TIMEOUT) { //20处,提供者一直没返回结果,则状态会是超时。
throw new TimeoutException(res.getStatus() == Response.SERVER_TIMEOUT, channel, res.getErrorMessage());
}
throw new RemotingException(channel, res.getErrorMessage());
}
}
消费者发送了请求消息后,会调用DefaultFuture的get方法,直到有结果返回或超时,如果还没结果返回,看"17处",提供者还没返回消息,则一直阻塞时长timeout。
提供者返回了结果,则服务调用状态是OK,则是"19处";提供者一直没返回结果,则状态会是超时,则是"20处"。
这里,会有个问题,就是返回消息中,dubbo如何识别返回消息对应是哪个请求消息?
下面继续看看返回消息时,dubbo是如何处理的。
有消息返回时,调用DefaultFuture的静态方法received:
public static void received(Channel channel, Response response) {
try {
DefaultFuture future = FUTURES.remove(response.getId()); //1处,根据消息的id,可以获得消息对应的DefaultFuture
if (future != null) {
future.doReceived(response); //2处,调用DefaultFuture方法doReceived,设置返回对象Response
} else {
logger.warn("The timeout response finally returned at "
+ (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS").format(new Date()))
+ ", response " + response
+ (channel == null ? "" : ", channel: " + channel.getLocalAddress()
+ " -> " + channel.getRemoteAddress()));
}
} finally {
CHANNELS.remove(response.getId());
}
}
从1处看,根据消息的id,可以获得消息对应的DefaultFuture,然后在2处,调用DefaultFuture方法doReceived,设置返回对象Response
private void doReceived(Response res) {
lock.lock();
try {
response = res; //3处
if (done != null) {
done.signal(); //4处
}
} finally {
lock.unlock();
}
if (callback != null) {
invokeCallback(callback); //5处
}
}
从3处看,就是设置返回对象Response,并且4处,释放,则get方法就可以不再阻塞,返回远程调用的结果;如果调用结果是异步返回的,则会在"5处",调用callback方法。
总结:
dubbo底层使用一个长连接,不断传输client和server的数据,client的Request和server的Response是用一个请求id(即是Request的id属性)来绑定对应关系的,所以不管长连接传输的数据是否按照顺序的,都没关系,因为可以根据id找到对应的client的Request。
线程派发
接着,我们来看看dubbo的提供者对这一长连接上的数据如何处理。
对一个长连接会有几个事件:
1.建立连接
2.断开连接
3.发送消息
4.接收消息
5.出现异常
dubbo会使用线程池来处理这几个事件,也即是,可能对一个事件派发线程来处理的,根据派发线程方式不同,分为几种:
all, direct, message, execution, connection
即是通过标签<dubbo:protocol dispatcher="all"/>的dispatcher来设置的,默认是"all"
1).all,在事件1.2.4.5,都会派发线程处理,如果根据用户配置的线程池创建方式得到线程池为null,但会有默认的线程池Executors.newCachedThreadPool(),
dubbo会创建com.alibaba.dubbo.remoting.transport.dispatcher.all.AllDispatcher
2)direct,不派发线程,对应com.alibaba.dubbo.remoting.transport.dispatcher.direct.DirectDispatcher
3)message,在事件4,派发线程处理,对应com.alibaba.dubbo.remoting.transport.dispatcher.message.MessageOnlyDispatcher
4)connection,在1.2.4.5,派发线程,并且对1.2事件会保证顺序,对应com.alibaba.dubbo.remoting.transport.dispatcher.connection.ConnectionOrderedDispatcher
5)execution,除发送全部使用线程池处理,对应com.alibaba.dubbo.remoting.transport.dispatcher.execution.ExecutionDispatcher
线程池
下面看看dubbo是如何创建线程池
使用标签指定线程池:
<dubbo:provider threadpool="fixed"/>或者<dubbo:protocol threadpool="fixed"/>
可配置:
fixed=com.alibaba.dubbo.common.threadpool.support.fixed.FixedThreadPool
cached=com.alibaba.dubbo.common.threadpool.support.cached.CachedThreadPool
limited=com.alibaba.dubbo.common.threadpool.support.limited.LimitedThreadPool
下面是三种线程池的源码
/**
* 此线程池可伸缩,线程空闲一分钟后回收,新请求重新创建线程,来源于:<code>Executors.newCachedThreadPool()</code>
*
* @see java.util.concurrent.Executors#newCachedThreadPool()
* @author william.liangf
*/
public class CachedThreadPool implements ThreadPool {
public Executor getExecutor(URL url) {
String name = url.getParameter(Constants.THREAD_NAME_KEY, Constants.DEFAULT_THREAD_NAME);
int cores = url.getParameter(Constants.CORE_THREADS_KEY, Constants.DEFAULT_CORE_THREADS);
int threads = url.getParameter(Constants.THREADS_KEY, Integer.MAX_VALUE);
int queues = url.getParameter(Constants.QUEUES_KEY, Constants.DEFAULT_QUEUES);
int alive = url.getParameter(Constants.ALIVE_KEY, Constants.DEFAULT_ALIVE); //default_alive:60000
return new ThreadPoolExecutor(cores, threads, alive, TimeUnit.MILLISECONDS,
queues == 0 ? new SynchronousQueue<Runnable>() :
(queues < 0 ? new LinkedBlockingQueue<Runnable>()
: new LinkedBlockingQueue<Runnable>(queues)),
new NamedThreadFactory(name, true), new AbortPolicyWithReport(name, url));
}
}
/**
* 此线程池启动时即创建固定大小的线程数,不做任何伸缩,来源于:<code>Executors.newFixedThreadPool()</code>
*
* @see java.util.concurrent.Executors#newFixedThreadPool(int)
* @author william.liangf
*/
public class FixedThreadPool implements ThreadPool {
public Executor getExecutor(URL url) {
String name = url.getParameter(Constants.THREAD_NAME_KEY, Constants.DEFAULT_THREAD_NAME);
int threads = url.getParameter(Constants.THREADS_KEY, Constants.DEFAULT_THREADS);
int queues = url.getParameter(Constants.QUEUES_KEY, Constants.DEFAULT_QUEUES);
return new ThreadPoolExecutor(threads, threads, 0, TimeUnit.MILLISECONDS,
queues == 0 ? new SynchronousQueue<Runnable>() :
(queues < 0 ? new LinkedBlockingQueue<Runnable>()
: new LinkedBlockingQueue<Runnable>(queues)),
new NamedThreadFactory(name, true), new AbortPolicyWithReport(name, url));
}
}
/**
* 此线程池一直增长,直到上限,增长后不收缩。
*
* @author <a href="mailto:gang.lvg@alibaba-inc.com">kimi</a>
*/
public class LimitedThreadPool implements ThreadPool {
public Executor getExecutor(URL url) {
String name = url.getParameter(Constants.THREAD_NAME_KEY, Constants.DEFAULT_THREAD_NAME);
int cores = url.getParameter(Constants.CORE_THREADS_KEY, Constants.DEFAULT_CORE_THREADS);
int threads = url.getParameter(Constants.THREADS_KEY, Constants.DEFAULT_THREADS);
int queues = url.getParameter(Constants.QUEUES_KEY, Constants.DEFAULT_QUEUES);
return new ThreadPoolExecutor(cores, threads, Long.MAX_VALUE, TimeUnit.MILLISECONDS,
queues == 0 ? new SynchronousQueue<Runnable>() :
(queues < 0 ? new LinkedBlockingQueue<Runnable>()
: new LinkedBlockingQueue<Runnable>(queues)),
new NamedThreadFactory(name, true), new AbortPolicyWithReport(name, url));
}
}
自己写了个RPC:
可以给个star,^0^.