文章目录
Trace模块原理图
- TraceMoudleProvider启动负责构建trace的Grpc handler
- TraceMoudleProvider启动负责构建segmentProducer
- TraceSegmentReportServiceHandler负责接收agent上报的trace
- segmentProducer负责处理handler接收的请求并发送给SegmentParse进行解析
- SegmentParse负责解析agent上报Segment数据并存储于SegmentParse的SegmentCoreInfo
- 每一个消息都会构建一个新的SegmentParse
- SegmentParse内部会创建SpanListener负责解析Segment中的span数据并存储于SpanListener的属性中[如MultiScopesSpanListener的sourceBuilder属性]
- 每一个消息都会构建一个新的SegmentParse,每一个SegmentParse都会构建一个新的SpanListener集合
- 解析的结果如sourceBuilder或SegmentCoreInfo会被SpanListener.build方法构建成Source数据对象
- Source数据对象发往sourceReceiver处理
- sourceReceiver将Source对象转StorageData对象,存储到es
为什么每一个消息都要构建新的SegmentParse和spanListeners
每一个UpstreamSegment都会创建一个解析器 [SegmentParseV2本身是资源轻量级对象,同时其内部segmentCoreInfo存储解析相关结果,所以一个请求一个解析器,segmentCoreInfo不可以存储多个请求的trace信息]【spanlisteners同理】
- Trace存储在es上的数据主要包含三类
- MultiScopesSpanListener处理最终存储的Metrics对象
- SegmentSpanListener处理最终存储的(Segment)Record对象
- MultiScopesSpanListener处理最终存储的TopN(Record)对象
源码分析一TraceModuleProvider
原理图
- 构建GrpcHandler: TraceSegmentServiceHandler和TraceSegmentReportServiceHandler
- 初始化segmentProducer和segmentProducerV2
- listenerManager初始化spanListener工厂,包含四个处理器工厂[如上图]
只需要关注TraceSegmentReportServiceHandler和segmentProducerV2
TraceSegmentReportServiceHandler以及jetty的TraceSegmentServletHandler为老版本实现
public class TraceModuleProvider extends ModuleProvider {
private final TraceServiceModuleConfig moduleConfig;
private SegmentParse.Producer segmentProducer;
private SegmentParseV2.Producer segmentProducerV2;
private DBLatencyThresholdsAndWatcher thresholds;
private UninstrumentedGatewaysConfig uninstrumentedGatewaysConfig;
public TraceModuleProvider() {
this.moduleConfig = new TraceServiceModuleConfig();
}
@Override public String name() {
return "default";
}
@Override public Class<? extends ModuleDefine> module() {
return TraceModule.class;
}
@Override public ModuleConfig createConfigBeanIfAbsent() {
return moduleConfig;
}
@Override public void prepare() throws ServiceNotProvidedException {
配置topn延迟的阈值 MultiScopesSpanListener会根据阈值算出慢请求 交给TopNStreamProcessor处理
thresholds = new DBLatencyThresholdsAndWatcher(moduleConfig.getSlowDBAccessThreshold(), this);
uninstrumentedGatewaysConfig = new UninstrumentedGatewaysConfig(this);
moduleConfig.setDbLatencyThresholdsAndWatcher(thresholds);
moduleConfig.setUninstrumentedGatewaysConfig(uninstrumentedGatewaysConfig);
segmentProducer = new SegmentParse.Producer(getManager(), listenerManager(), moduleConfig);
初始化SegmentParseV2 主要用于解析TraceSegment数据 初始化spanListener工厂Factory [解析阶段根据工厂创建spanListener集合]
segmentProducerV2 = new SegmentParseV2.Producer(getManager(), listenerManager(), moduleConfig);
this.registerServiceImplementation(ISegmentParserService.class, new SegmentParserServiceImpl(segmentProducerV2));
}
public SegmentParserListenerManager listenerManager() {
SegmentParserListenerManager listenerManager = new SegmentParserListenerManager();
if (moduleConfig.isTraceAnalysis()) {
listenerManager.add(new MultiScopesSpanListener.Factory());
listenerManager.add(new ServiceMappingSpanListener.Factory());
listenerManager.add(new ServiceInstanceMappingSpanListener.Factory());
}
listenerManager.add(new SegmentSpanListener.Factory(moduleConfig.getSampleRate()));
return listenerManager;
}
@Override public void start() throws ModuleStartException {
DynamicConfigurationService dynamicConfigurationService = getManager().find(ConfigurationModule.NAME).provider().getService(DynamicConfigurationService.class);
GRPCHandlerRegister grpcHandlerRegister = getManager().find(SharingServerModule.NAME).provider().getService(GRPCHandlerRegister.class);
JettyHandlerRegister jettyHandlerRegister = getManager().find(SharingServerModule.NAME).provider().getService(JettyHandlerRegister.class);
try {
dynamicConfigurationService.registerConfigChangeWatcher(thresholds);
dynamicConfigurationService.registerConfigChangeWatcher(uninstrumentedGatewaysConfig);
为grpc server添加请求的处理[TraceSegmentServiceHandler]
grpcHandlerRegister.addHandler(new TraceSegmentServiceHandler(segmentProducer));
新版本采用grpc上报
grpcHandlerRegister.addHandler(new TraceSegmentReportServiceHandler(segmentProducerV2, getManager()));
老版本TraceSegment支持http协议上报
jettyHandlerRegister.addHandler(new TraceSegmentServletHandler(segmentProducer));
SegmentStandardizationWorker standardizationWorker = new SegmentStandardizationWorker(getManager(), segmentProducer, moduleConfig.getBufferPath() + "v5", moduleConfig.getBufferOffsetMaxFileSize(), moduleConfig.getBufferDataMaxFileSize(), moduleConfig.isBufferFileCleanWhenRestart(), false);
segmentProducer.setStandardizationWorker(standardizationWorker);
辅助工具写入文件以及外部遥测模块[比如Prometheus]
SegmentStandardizationWorker standardizationWorkerV2 = new SegmentStandardizationWorker(getManager(), segmentProducerV2, moduleConfig.getBufferPath(), moduleConfig.getBufferOffsetMaxFileSize(), moduleConfig.getBufferDataMaxFileSize(), moduleConfig.isBufferFileCleanWhenRestart(), true);
segmentProducerV2.setStandardizationWorker(standardizationWorkerV2);
} catch (IOException e) {
throw new ModuleStartException(e.getMessage(), e);
}
}
源码分析一Segment上报
源码分析一TraceSegmentReportServiceHandler
- 转SegmentParseV2.Producer 处理UpstreamSegment数据
public class TraceSegmentReportServiceHandler extends TraceSegmentReportServiceGrpc.TraceSegmentReportServiceImplBase implements GRPCHandler {
private final SegmentParseV2.Producer segmentProducer;
接收grpc请求 [agent端 TraceSegmentServiceClient负责发送数据]
@Override public StreamObserver<UpstreamSegment> collect(StreamObserver<Commands> responseObserver) {
return new StreamObserver<UpstreamSegment>() {
@Override public void onNext(UpstreamSegment segment) {
HistogramMetrics.Timer timer = histogram.createTimer();
try {
segmentProducer处理请求
segmentProducer.send(segment, SegmentSource.Agent);
} finally {
timer.finish();
}
}
};
}
}
源码分析一SegmentParseV2.Producer
- 交给解析器segmentParse处理
public void send(UpstreamSegment segment, SegmentSource source) {
// 为每一个UpstreamSegment创建一个解析器 [SegmentParseV2本身是资源轻量级对象,同时其内部segmentCoreInfo存储解析相关结果,所以一个请求一个解析器]
SegmentParseV2 segmentParse = new SegmentParseV2(moduleManager, listenerManager, config);
segmentParse.setStandardizationWorker(standardizationWorker);
segmentParse.parse(new BufferData<>(segment), source);
}
源码分析一SegmentParseV2.parse
- 1 解析前先初始化SpanListeners
- 2 构建SegmentObject并通过Decorator包装屏蔽版本差异【SegmentParseV2或SegmentParse】
- 3 preBuild预构建将SegmentDecorator数据的span信息解析到spanListener的属性中
- notifyListenerToBuild 处理spanListener解析后的数据以及SegmentParseV2解析后的数据segmentCoreInfo,通过StreamProcessor流式处理后持久化
public class SegmentParseV2 {
解析过程中不同的span交由不同的listener处理,每一个消息都有一个SegmentParseV2实例和spanListeners实例集合
private final List<SpanListener> spanListeners;
private final SegmentParserListenerManager listenerManager;
解析过程中生成的segment核心数据都会记录到这里
private final SegmentCoreInfo segmentCoreInfo;
public boolean parse(BufferData<UpstreamSegment> bufferData, SegmentSource source) {
解析前先初始化SpanListeners
createSpanListeners();
try {
UpstreamSegment upstreamSegment = bufferData.getMessageType();
获取traceIds集合 [注意只有traceids[0]作为全局链路id]
UniqueId也就是DistributedTraceId
List<UniqueId> traceIds = upstreamSegment.getGlobalTraceIdsList();
if (bufferData.getV2Segment() == null) {
bufferData.setV2Segment(parseBinarySegment(upstreamSegment));
}
获取traceSegment对象 [包含全部的span信息 traceid信息 serviceid Serviceinstanceid]
SegmentObject segmentObject = bufferData.getV2Segment();
...... 删除其他代码
封装成SegmentObject[屏蔽版本差异]
SegmentDecorator segmentDecorator = new SegmentDecorator(segmentObject);
前置构建完成segmentCoreInfo处理
if (!preBuild(traceIds, segmentDecorator)) {
...... 删除其他代码
return false;
} else {
...... 删除其他代码
将采样的segment进行stream流式作业
notifyListenerToBuild();
return true;
}
} catch (Throwable e) {
...... 删除其他代码
return true;
}
}
}
源码分析一preBuild
- 链路traceId构建
- notifyGlobalsListener通知MultiScopesSpanListener和SegmentSpanListener处理
- 设置数据到segmentCoreInfo[流式处理Segment都是copy该属性获取数据]
- 进行exchanged
- 根据span类型通知不同的spanListener处理
exchang知识介绍
1 agent 上报为节约带宽会将ip等信息映射成一个唯一id,存储在agent的字典
2 上报数据只上报id
但字典不一定有数据[比如agent刚启动未更新字典],此时上报的为ip等字符串,这里则通过exchange将字符串转id
private boolean preBuild(List<UniqueId> traceIds, SegmentDecorator segmentDecorator) {
StringBuilder segmentIdBuilder = new StringBuilder();
TraceSegmentId为链路traceId 等于traceIds.get(0)
for (int i = 0; i < segmentDecorator.getTraceSegmentId().getIdPartsList().size(); i++) {
if (i == 0) {
segmentIdBuilder.append(segmentDecorator.getTraceSegmentId().getIdPartsList().get(i));
} else {
segmentIdBuilder.append(".").append(segmentDecorator.getTraceSegmentId().getIdPartsList().get(i));
}
}
for (UniqueId uniqueId : traceIds) {
通知GlobalTraceIdsListener 处理
MultiScopesSpanListener 对traceId进行采样
SegmentSpanListener 如果被采样设置内部属性segment信息
notifyGlobalsListener(uniqueId);
}
构建traceSegment相关信息
segmentCoreInfo.setSegmentId(segmentIdBuilder.toString());
segmentCoreInfo.setServiceId(segmentDecorator.getServiceId());
segmentCoreInfo.setServiceInstanceId(segmentDecorator.getServiceInstanceId());
segmentCoreInfo.setDataBinary(segmentDecorator.toByteArray());
segmentCoreInfo.setV2(true);
boolean exchanged = true;
exchanged完成span中的字典转换
for (int i = 0; i < segmentDecorator.getSpansCount(); i++) {
SpanDecorator spanDecorator = segmentDecorator.getSpans(i);
完成Component peer operationName 的字符串转id映射 【agent可能没有进行字典转换,则需要转换】
将字符串改成字典key,字符串置为空
if (!SpanExchanger.getInstance(moduleManager).exchange(spanDecorator, segmentCoreInfo.getServiceId())) {
exchanged = false;
} else {
for (int j = 0; j < spanDecorator.getRefsCount(); j++) {
ReferenceDecorator referenceDecorator = spanDecorator.getRefs(j);
完成Endpoint ParentEndpoint 和NetworkAddress转换
if (!ReferenceIdExchanger.getInstance(moduleManager).exchange(referenceDecorator, segmentCoreInfo.getServiceId())) {
exchanged = false;
}
}
}
if (segmentCoreInfo.getStartTime() > spanDecorator.getStartTime()) {
segmentCoreInfo.setStartTime(spanDecorator.getStartTime());
}
if (segmentCoreInfo.getEndTime() < spanDecorator.getEndTime()) {
segmentCoreInfo.setEndTime(spanDecorator.getEndTime());
}
segmentCoreInfo.setError(spanDecorator.getIsError() || segmentCoreInfo.isError());
}
成功将所有字符串转数值映射
if (exchanged) {
long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(segmentCoreInfo.getStartTime());
segmentCoreInfo.setMinuteTimeBucket(minuteTimeBucket);
for (int i = 0; i < segmentDecorator.getSpansCount(); i++) {
SpanDecorator屏蔽版本差异
SpanDecorator spanDecorator = segmentDecorator.getSpans(i);
根据span类型调用不同的spanListener处理
if (spanDecorator.getSpanId() == 0) {
仅处理第一个span
notifyFirstListener(spanDecorator);
}
根据span类型调用不同的spanListener处理
if (SpanType.Exit.equals(spanDecorator.getSpanType())) {
notifyExitListener(spanDecorator);
} else if (SpanType.Entry.equals(spanDecorator.getSpanType())) {
notifyEntryListener(spanDecorator);
} else if (SpanType.Local.equals(spanDecorator.getSpanType())) {
notifyLocalListener(spanDecorator);
} else {
logger.error("span type value was unexpected, span type name: {}", spanDecorator.getSpanType().name());
}
}
}
return exchanged;
}
源码分析一SegmentSpanListener.build
- notifyListenerToBuild会调用所有spanlistener的build
private void notifyListenerToBuild() {
spanListeners.forEach(SpanListener::build);
}
- 调用RecordStreamProcessor进行流式处理,存储至es
@Override public void build() {
...... 删除其他代码
将segment发给SegmentDispatcher处理,SegmentDispatcher
SegmentDispatcher调用RecordStreamProcessor进行流式处理
RecordStreamProcessor直接调用es client-api进行存储[不同于MetricsStreamProcessor的L1,L2聚合]
sourceReceiver.receive(segment);
}
源码分析一MultiScopesSpanListener.build
- prebuild构建了entrySourceBuilders,exitSourceBuilders和slowDatabaseAccesses数据
- 基于entrySourceBuilders和exitSourceBuilders生成大量Source对象
- sourceReceiver根据这些Source对象进行颁发[Dispatcher为oal引擎生成]
- entrySourceBuilders和exitSourceBuilders生成的source最终交给MetricsStreamProcessor处理
- preBuild构建了slowDatabaseAccesses对象,
- sourceReceiver转发给DatabaseStatementDispatcher
- DatabaseStatementDispatcher构建TopNDatabaseStatement
- TopNStreamProcessor进行TopNDatabaseStatement慢查询对象处理和存储
@Override public void build() {
entrySourceBuilders.forEach(entrySourceBuilder -> {
entrySourceBuilder.setTimeBucket(minuteTimeBucket);
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toAll());
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toService());
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toServiceInstance());
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toEndpoint());
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toServiceRelation());
动态生成的Dispatcher 交给MetricsStreamProcessor处理
sourceReceiver.receive(entrySourceBuilder.toServiceInstanceRelation());
EndpointRelation endpointRelation = entrySourceBuilder.toEndpointRelation();
if (endpointRelation != null) {
sourceReceiver.receive(endpointRelation);
}
});
动态生成的Dispatcher 交给MetricsStreamProcessor处理
exitSourceBuilders.forEach(exitSourceBuilder -> {
if (nonNull(entrySpanDecorator)) {
exitSourceBuilder.setSourceEndpointId(entrySpanDecorator.getOperationNameId());
} else {
exitSourceBuilder.setSourceEndpointId(Const.USER_ENDPOINT_ID);
}
exitSourceBuilder.setSourceEndpointName(endpointInventoryCache.get(exitSourceBuilder.getSourceEndpointId()).getName());
exitSourceBuilder.setTimeBucket(minuteTimeBucket);
sourceReceiver.receive(exitSourceBuilder.toServiceRelation());
sourceReceiver.receive(exitSourceBuilder.toServiceInstanceRelation());
if (RequestType.DATABASE.equals(exitSourceBuilder.getType())) {
sourceReceiver.receive(exitSourceBuilder.toDatabaseAccess());
}
});
交给TopNStreamProcessor处理topn慢查询
slowDatabaseAccesses.forEach(sourceReceiver::receive);
}
总结
- notifyFirstListener和notifyExitListener等notify源码主要用于解析span对象
- 放置于SpanListener的属性中,将来根据这些解析后初始化的属性进行Stream流式处理和存储
- SegmentSpanListener主要存储TraceSegment,Record数据到es
- MultiScopesSpanListener主要基于SourceBuilder构建大量Metrics度量对象存储到es
- MultiScopesSpanListener同时通过slowDatabaseAccesses调用TopNStreamProcessor存储慢查询到es【默认慢查询阈值200ms】
- ServiceInstanceMappingSpanListener和ServiceMappingSpanListenery用于更新注册信息[读者可自行阅读]
备注
- MultiScopesSpanListener的Stream流式作业调用MetricsStreamProcessor
- 因此Trace的度量存储和告警分析流程同JVMModule[参见第十篇]