skywalking源码分析第十一篇一server-receiver-plugin之trace链路追踪插件模块启动

本文深入剖析了一款分布式系统追踪组件的Trace模块,从TraceModuleProvider的启动、Segment上报、TraceSegmentReportServiceHandler的处理到SegmentParseV2的解析过程。讲解了如何构建GrpcHandler、初始化segmentProducer以及如何处理agent上报的trace数据,详细解析了源码流程,涉及SegmentParseV2的构建、解析和存储策略。此外,还介绍了MultiScopesSpanListener和SegmentSpanListener在处理Metrics和Record对象中的作用,以及TopN慢查询的处理机制。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Trace模块原理图

  • TraceMoudleProvider启动负责构建trace的Grpc handler
  • TraceMoudleProvider启动负责构建segmentProducer
  • TraceSegmentReportServiceHandler负责接收agent上报的trace
  • segmentProducer负责处理handler接收的请求并发送给SegmentParse进行解析
  • SegmentParse负责解析agent上报Segment数据并存储于SegmentParse的SegmentCoreInfo
  • 每一个消息都会构建一个新的SegmentParse
  • SegmentParse内部会创建SpanListener负责解析Segment中的span数据并存储于SpanListener的属性中[如MultiScopesSpanListener的sourceBuilder属性]
  • 每一个消息都会构建一个新的SegmentParse,每一个SegmentParse都会构建一个新的SpanListener集合
  • 解析的结果如sourceBuilder或SegmentCoreInfo会被SpanListener.build方法构建成Source数据对象
  • Source数据对象发往sourceReceiver处理
  • sourceReceiver将Source对象转StorageData对象,存储到es

为什么每一个消息都要构建新的SegmentParse和spanListeners

每一个UpstreamSegment都会创建一个解析器 [SegmentParseV2本身是资源轻量级对象,同时其内部segmentCoreInfo存储解析相关结果,所以一个请求一个解析器,segmentCoreInfo不可以存储多个请求的trace信息]【spanlisteners同理】

  • Trace存储在es上的数据主要包含三类
  • MultiScopesSpanListener处理最终存储的Metrics对象
  • SegmentSpanListener处理最终存储的(Segment)Record对象
  • MultiScopesSpanListener处理最终存储的TopN(Record)对象

在这里插入图片描述

源码分析一TraceModuleProvider

原理图

在这里插入图片描述

  • 构建GrpcHandler: TraceSegmentServiceHandler和TraceSegmentReportServiceHandler
  • 初始化segmentProducer和segmentProducerV2
  • listenerManager初始化spanListener工厂,包含四个处理器工厂[如上图]

只需要关注TraceSegmentReportServiceHandler和segmentProducerV2
TraceSegmentReportServiceHandler以及jetty的TraceSegmentServletHandler为老版本实现


public class TraceModuleProvider extends ModuleProvider {

    private final TraceServiceModuleConfig moduleConfig;
    private SegmentParse.Producer segmentProducer;
    private SegmentParseV2.Producer segmentProducerV2;
    private DBLatencyThresholdsAndWatcher thresholds;
    private UninstrumentedGatewaysConfig uninstrumentedGatewaysConfig;

    public TraceModuleProvider() {
        this.moduleConfig = new TraceServiceModuleConfig();
    }

    @Override public String name() {
        return "default";
    }

    @Override public Class<? extends ModuleDefine> module() {
        return TraceModule.class;
    }

    @Override public ModuleConfig createConfigBeanIfAbsent() {
        return moduleConfig;
    }

    @Override public void prepare() throws ServiceNotProvidedException {
        配置topn延迟的阈值 MultiScopesSpanListener会根据阈值算出慢请求 交给TopNStreamProcessor处理
        thresholds = new DBLatencyThresholdsAndWatcher(moduleConfig.getSlowDBAccessThreshold(), this);

        uninstrumentedGatewaysConfig = new UninstrumentedGatewaysConfig(this);

        moduleConfig.setDbLatencyThresholdsAndWatcher(thresholds);
        moduleConfig.setUninstrumentedGatewaysConfig(uninstrumentedGatewaysConfig);

        segmentProducer = new SegmentParse.Producer(getManager(), listenerManager(), moduleConfig);
        初始化SegmentParseV2 主要用于解析TraceSegment数据 初始化spanListener工厂Factory [解析阶段根据工厂创建spanListener集合]
        segmentProducerV2 = new SegmentParseV2.Producer(getManager(), listenerManager(), moduleConfig);

        this.registerServiceImplementation(ISegmentParserService.class, new SegmentParserServiceImpl(segmentProducerV2));
    }

    public SegmentParserListenerManager listenerManager() {
        SegmentParserListenerManager listenerManager = new SegmentParserListenerManager();
        if (moduleConfig.isTraceAnalysis()) {
            listenerManager.add(new MultiScopesSpanListener.Factory());
            listenerManager.add(new ServiceMappingSpanListener.Factory());
            listenerManager.add(new ServiceInstanceMappingSpanListener.Factory());
        }
        listenerManager.add(new SegmentSpanListener.Factory(moduleConfig.getSampleRate()));

        return listenerManager;
    }

    @Override public void start() throws ModuleStartException {
        DynamicConfigurationService dynamicConfigurationService = getManager().find(ConfigurationModule.NAME).provider().getService(DynamicConfigurationService.class);
        GRPCHandlerRegister grpcHandlerRegister = getManager().find(SharingServerModule.NAME).provider().getService(GRPCHandlerRegister.class);
        JettyHandlerRegister jettyHandlerRegister = getManager().find(SharingServerModule.NAME).provider().getService(JettyHandlerRegister.class);
        try {
            dynamicConfigurationService.registerConfigChangeWatcher(thresholds);
            dynamicConfigurationService.registerConfigChangeWatcher(uninstrumentedGatewaysConfig);
            为grpc server添加请求的处理[TraceSegmentServiceHandler]
            grpcHandlerRegister.addHandler(new TraceSegmentServiceHandler(segmentProducer));
            新版本采用grpc上报
            grpcHandlerRegister.addHandler(new TraceSegmentReportServiceHandler(segmentProducerV2, getManager()));

            老版本TraceSegment支持http协议上报
            jettyHandlerRegister.addHandler(new TraceSegmentServletHandler(segmentProducer));

            SegmentStandardizationWorker standardizationWorker = new SegmentStandardizationWorker(getManager(), segmentProducer, moduleConfig.getBufferPath() + "v5", moduleConfig.getBufferOffsetMaxFileSize(), moduleConfig.getBufferDataMaxFileSize(), moduleConfig.isBufferFileCleanWhenRestart(), false);
            segmentProducer.setStandardizationWorker(standardizationWorker);
            辅助工具写入文件以及外部遥测模块[比如Prometheus]
            SegmentStandardizationWorker standardizationWorkerV2 = new SegmentStandardizationWorker(getManager(), segmentProducerV2, moduleConfig.getBufferPath(), moduleConfig.getBufferOffsetMaxFileSize(), moduleConfig.getBufferDataMaxFileSize(), moduleConfig.isBufferFileCleanWhenRestart(), true);
            segmentProducerV2.setStandardizationWorker(standardizationWorkerV2);
        } catch (IOException e) {
            throw new ModuleStartException(e.getMessage(), e);
    }

}

源码分析一Segment上报

源码分析一TraceSegmentReportServiceHandler

  • 转SegmentParseV2.Producer 处理UpstreamSegment数据

public class TraceSegmentReportServiceHandler extends TraceSegmentReportServiceGrpc.TraceSegmentReportServiceImplBase implements GRPCHandler {
    
    private final SegmentParseV2.Producer segmentProducer;
    
    接收grpc请求 [agent端 TraceSegmentServiceClient负责发送数据]
    @Override public StreamObserver<UpstreamSegment> collect(StreamObserver<Commands> responseObserver) {
        return new StreamObserver<UpstreamSegment>() {
            @Override public void onNext(UpstreamSegment segment) {
                HistogramMetrics.Timer timer = histogram.createTimer();
                try {
                    segmentProducer处理请求
                    segmentProducer.send(segment, SegmentSource.Agent);
                } finally {
                    timer.finish();
                }
            }

        };
    }
}

源码分析一SegmentParseV2.Producer

  • 交给解析器segmentParse处理
public void send(UpstreamSegment segment, SegmentSource source) {
    // 为每一个UpstreamSegment创建一个解析器 [SegmentParseV2本身是资源轻量级对象,同时其内部segmentCoreInfo存储解析相关结果,所以一个请求一个解析器]
    SegmentParseV2 segmentParse = new SegmentParseV2(moduleManager, listenerManager, config);
    segmentParse.setStandardizationWorker(standardizationWorker);
    segmentParse.parse(new BufferData<>(segment), source);
}

源码分析一SegmentParseV2.parse

在这里插入图片描述

  • 1 解析前先初始化SpanListeners
  • 2 构建SegmentObject并通过Decorator包装屏蔽版本差异【SegmentParseV2或SegmentParse】
  • 3 preBuild预构建将SegmentDecorator数据的span信息解析到spanListener的属性中
  • notifyListenerToBuild 处理spanListener解析后的数据以及SegmentParseV2解析后的数据segmentCoreInfo,通过StreamProcessor流式处理后持久化

public class SegmentParseV2 {
    解析过程中不同的span交由不同的listener处理,每一个消息都有一个SegmentParseV2实例和spanListeners实例集合
    private final List<SpanListener> spanListeners; 
    private final SegmentParserListenerManager listenerManager;
    解析过程中生成的segment核心数据都会记录到这里
    private final SegmentCoreInfo segmentCoreInfo;

    
    public boolean parse(BufferData<UpstreamSegment> bufferData, SegmentSource source) {
        解析前先初始化SpanListeners
        createSpanListeners();
        try {
            UpstreamSegment upstreamSegment = bufferData.getMessageType();
            获取traceIds集合 [注意只有traceids[0]作为全局链路id]
            UniqueId也就是DistributedTraceId
            List<UniqueId> traceIds = upstreamSegment.getGlobalTraceIdsList();

            if (bufferData.getV2Segment() == null) {
                bufferData.setV2Segment(parseBinarySegment(upstreamSegment));
            }
            获取traceSegment对象 [包含全部的span信息 traceid信息  serviceid Serviceinstanceid]
            SegmentObject segmentObject = bufferData.getV2Segment();
            ...... 删除其他代码
            封装成SegmentObject[屏蔽版本差异]
            SegmentDecorator segmentDecorator = new SegmentDecorator(segmentObject);
            前置构建完成segmentCoreInfo处理
            if (!preBuild(traceIds, segmentDecorator)) {
                ...... 删除其他代码
                return false;
            } else {
               ...... 删除其他代码
                将采样的segment进行stream流式作业
                notifyListenerToBuild();
                return true;
            }
        } catch (Throwable e) {
           ...... 删除其他代码

            return true;
        }
    }
}

源码分析一preBuild

  • 链路traceId构建
  • notifyGlobalsListener通知MultiScopesSpanListener和SegmentSpanListener处理
  • 设置数据到segmentCoreInfo[流式处理Segment都是copy该属性获取数据]
  • 进行exchanged
  • 根据span类型通知不同的spanListener处理

exchang知识介绍

1 agent 上报为节约带宽会将ip等信息映射成一个唯一id,存储在agent的字典
2 上报数据只上报id
但字典不一定有数据[比如agent刚启动未更新字典],此时上报的为ip等字符串,这里则通过exchange将字符串转id


 private boolean preBuild(List<UniqueId> traceIds, SegmentDecorator segmentDecorator) {
    StringBuilder segmentIdBuilder = new StringBuilder();
    TraceSegmentId为链路traceId 等于traceIds.get(0)
    for (int i = 0; i < segmentDecorator.getTraceSegmentId().getIdPartsList().size(); i++) {
        if (i == 0) {
            segmentIdBuilder.append(segmentDecorator.getTraceSegmentId().getIdPartsList().get(i));
        } else {
            segmentIdBuilder.append(".").append(segmentDecorator.getTraceSegmentId().getIdPartsList().get(i));
        }
    }

    for (UniqueId uniqueId : traceIds) {
        通知GlobalTraceIdsListener 处理
        MultiScopesSpanListener 对traceId进行采样
        SegmentSpanListener 如果被采样设置内部属性segment信息
        notifyGlobalsListener(uniqueId);
    }
    构建traceSegment相关信息
    segmentCoreInfo.setSegmentId(segmentIdBuilder.toString());
    segmentCoreInfo.setServiceId(segmentDecorator.getServiceId());
    segmentCoreInfo.setServiceInstanceId(segmentDecorator.getServiceInstanceId());
    segmentCoreInfo.setDataBinary(segmentDecorator.toByteArray());
    segmentCoreInfo.setV2(true);

    boolean exchanged = true;

    exchanged完成span中的字典转换
    for (int i = 0; i < segmentDecorator.getSpansCount(); i++) {
        SpanDecorator spanDecorator = segmentDecorator.getSpans(i);
        完成Component peer operationName 的字符串转id映射 【agent可能没有进行字典转换,则需要转换】
        将字符串改成字典key,字符串置为空
        if (!SpanExchanger.getInstance(moduleManager).exchange(spanDecorator, segmentCoreInfo.getServiceId())) {
            exchanged = false;
        } else {
            for (int j = 0; j < spanDecorator.getRefsCount(); j++) {
                ReferenceDecorator referenceDecorator = spanDecorator.getRefs(j);
                完成Endpoint ParentEndpoint 和NetworkAddress转换
                if (!ReferenceIdExchanger.getInstance(moduleManager).exchange(referenceDecorator, segmentCoreInfo.getServiceId())) {
                    exchanged = false;
                }
            }
        }

        if (segmentCoreInfo.getStartTime() > spanDecorator.getStartTime()) {
            segmentCoreInfo.setStartTime(spanDecorator.getStartTime());
        }
        if (segmentCoreInfo.getEndTime() < spanDecorator.getEndTime()) {
            segmentCoreInfo.setEndTime(spanDecorator.getEndTime());
        }
        segmentCoreInfo.setError(spanDecorator.getIsError() || segmentCoreInfo.isError());
    }
    成功将所有字符串转数值映射
    if (exchanged) {
        long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(segmentCoreInfo.getStartTime());
        segmentCoreInfo.setMinuteTimeBucket(minuteTimeBucket);

        for (int i = 0; i < segmentDecorator.getSpansCount(); i++) {
            SpanDecorator屏蔽版本差异
            SpanDecorator spanDecorator = segmentDecorator.getSpans(i);
            根据span类型调用不同的spanListener处理
            if (spanDecorator.getSpanId() == 0) {
                仅处理第一个span
                notifyFirstListener(spanDecorator);
            }
            根据span类型调用不同的spanListener处理
            if (SpanType.Exit.equals(spanDecorator.getSpanType())) {
                notifyExitListener(spanDecorator);
            } else if (SpanType.Entry.equals(spanDecorator.getSpanType())) {
                notifyEntryListener(spanDecorator);
            } else if (SpanType.Local.equals(spanDecorator.getSpanType())) {
                notifyLocalListener(spanDecorator);
            } else {
                logger.error("span type value was unexpected, span type name: {}", spanDecorator.getSpanType().name());
            }
        }
    }

    return exchanged;
}

源码分析一SegmentSpanListener.build

  • notifyListenerToBuild会调用所有spanlistener的build
 private void notifyListenerToBuild() {
     spanListeners.forEach(SpanListener::build);
 }
  • 调用RecordStreamProcessor进行流式处理,存储至es

@Override public void build() {
    ...... 删除其他代码
    将segment发给SegmentDispatcher处理,SegmentDispatcher
    SegmentDispatcher调用RecordStreamProcessor进行流式处理
    RecordStreamProcessor直接调用es client-api进行存储[不同于MetricsStreamProcessor的L1,L2聚合]
    sourceReceiver.receive(segment);
}

源码分析一MultiScopesSpanListener.build

  • prebuild构建了entrySourceBuilders,exitSourceBuilders和slowDatabaseAccesses数据
  • 基于entrySourceBuilders和exitSourceBuilders生成大量Source对象
  • sourceReceiver根据这些Source对象进行颁发[Dispatcher为oal引擎生成]
  • entrySourceBuilders和exitSourceBuilders生成的source最终交给MetricsStreamProcessor处理
  • preBuild构建了slowDatabaseAccesses对象,
  • sourceReceiver转发给DatabaseStatementDispatcher
  • DatabaseStatementDispatcher构建TopNDatabaseStatement
  • TopNStreamProcessor进行TopNDatabaseStatement慢查询对象处理和存储

@Override public void build() {
    entrySourceBuilders.forEach(entrySourceBuilder -> {
        entrySourceBuilder.setTimeBucket(minuteTimeBucket);
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toAll());
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toService());
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toServiceInstance());
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toEndpoint());
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toServiceRelation());
        动态生成的Dispatcher 交给MetricsStreamProcessor处理
        sourceReceiver.receive(entrySourceBuilder.toServiceInstanceRelation());
        EndpointRelation endpointRelation = entrySourceBuilder.toEndpointRelation();
        
        if (endpointRelation != null) {
            sourceReceiver.receive(endpointRelation);
        }
    });
    动态生成的Dispatcher 交给MetricsStreamProcessor处理
    exitSourceBuilders.forEach(exitSourceBuilder -> {
        if (nonNull(entrySpanDecorator)) {
            exitSourceBuilder.setSourceEndpointId(entrySpanDecorator.getOperationNameId());
        } else {
            exitSourceBuilder.setSourceEndpointId(Const.USER_ENDPOINT_ID);
        }
        exitSourceBuilder.setSourceEndpointName(endpointInventoryCache.get(exitSourceBuilder.getSourceEndpointId()).getName());

        exitSourceBuilder.setTimeBucket(minuteTimeBucket);
        sourceReceiver.receive(exitSourceBuilder.toServiceRelation());
        sourceReceiver.receive(exitSourceBuilder.toServiceInstanceRelation());
        if (RequestType.DATABASE.equals(exitSourceBuilder.getType())) {
            sourceReceiver.receive(exitSourceBuilder.toDatabaseAccess());
        }
    });
    交给TopNStreamProcessor处理topn慢查询
    slowDatabaseAccesses.forEach(sourceReceiver::receive);
}

总结

  • notifyFirstListener和notifyExitListener等notify源码主要用于解析span对象
  • 放置于SpanListener的属性中,将来根据这些解析后初始化的属性进行Stream流式处理和存储
  • SegmentSpanListener主要存储TraceSegment,Record数据到es
  • MultiScopesSpanListener主要基于SourceBuilder构建大量Metrics度量对象存储到es
  • MultiScopesSpanListener同时通过slowDatabaseAccesses调用TopNStreamProcessor存储慢查询到es【默认慢查询阈值200ms】
  • ServiceInstanceMappingSpanListener和ServiceMappingSpanListenery用于更新注册信息[读者可自行阅读]

备注

  • MultiScopesSpanListener的Stream流式作业调用MetricsStreamProcessor
  • 因此Trace的度量存储和告警分析流程同JVMModule[参见第十篇]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值