Arthas在线诊断案例实战整理

会一直持续更新。。。

Arthas 是一款线上监控诊断产品，通过全局视角实时查看应用 load、内存、gc、线程的状态信息，并能在不修改应用代码的情况下，对业务问题进行诊断，包括查看方法调用的出入参、异常，监测方法执行耗时，类加载信息等，大大提升线上问题排查效率。

工欲善其事，必先利其器

arthas idea 插件 - 汪小哥

monitor/watch/trace 相关

monitor/watch/trace 相关

• watch - 方法执行数据观测
• stack - 输出当前方法被调用的调用路径
• trace - 方法内部调用路径，并输出方法路径上的每个节点上耗时
• tt - 方法执行数据的时空隧道，记录下指定方法每次调用的入参和返回信息，并能对这些不同的时间下调用进行观测

查看JDK代理生成的代理实例和调用栈

通过sc命令发现InterfaceCacheConfigMapper，多生成了一个com.sun.proxy.$Proxy106实例。
想看看这个实例是谁生成的？执行路径是咋样的？

import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.xxx.gateway.admin.persistence.model.InterfaceCacheConfigDo;
import org.apache.ibatis.annotations.Mapper;

/**
 * 接口缓存配置的映射
 *
 * @since 2024/5/17
 */
@Mapper
public interface InterfaceCacheConfigMapper extends BaseMapper<InterfaceCacheConfigDo> {
    // BaseMapper
}

[arthas@25193]$ sc com.xxx.gateway.admin.persistence.mapper.InterfaceCacheConfigMapper
com.xxx.gateway.admin.persistence.mapper.InterfaceCacheConfigMapper
com.sun.proxy.$Proxy106

options unsafe true 
options strict false 
options json-format true 

[arthas@21643]$ watch -m 1000 java.lang.reflect.Proxy newProxyInstance '{params[1][0].getName(),returnObj}' 'params[1][0].getName().contains("RightsConsumptMapper")' 
Affect(class count: 116 , method count: 1) cost in 711 ms, listenerId: 1

method=java.lang.reflect.Proxy.newProxyInstance location=AtExit
ts=2024-11-26 18:25:36.963; [cost=0.687417ms] result=["com.xxx.dubbo.rights.sharding.RightsConsumptMapper",{}]

[arthas@22253]$ stack -m 1000 java.lang.reflect.Proxy newProxyInstance 'params[1][0].getName().contains("RightsConsumptMapper")' 
Affect(class count: 116 , method count: 1) cost in 648 ms, listenerId: 1

ts=2024-11-26 18:33:13.180;thread_name=main;id=1;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@18b4aac2
    @java.lang.reflect.Proxy.newProxyInstance()
        at org.apache.ibatis.binding.MapperProxyFactory.newInstance(MapperProxyFactory.java:48)
        at org.apache.ibatis.binding.MapperProxyFactory.newInstance(MapperProxyFactory.java:53)
        at org.apache.ibatis.binding.MapperRegistry.getMapper(MapperRegistry.java:50)
        at org.apache.ibatis.session.Configuration.getMapper(Configuration.java:823)
        at org.mybatis.spring.SqlSessionTemplate.getMapper(SqlSessionTemplate.java:311)
        at org.mybatis.spring.mapper.MapperFactoryBean.getObject(MapperFactoryBean.java:95)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1828)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1265)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:334)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
        at com.xxx.Application.main(Application.java:25)

SkyWalking链路追踪

实践出真知识！！！
若不了解其底层实现原理，是很难想到这些切面的拦截点。

// 【切面的拦截点】生成新的追踪身份traceId + wanda_event开头的线程
stack org.apache.skywalking.apm.agent.core.context.ids.NewDistributedTraceId <init> '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")'

watch org.apache.skywalking.apm.agent.core.context.ids.NewDistributedTraceId <init> '{target, returnObj}' '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")' -x 6

// 【切面的拦截点】获取全局追踪身份traceId + wanda_event开头的线程
stack org.apache.skywalking.apm.agent.core.context.AbstractTracerContext getReadablePrimaryTraceId '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")'

watch org.apache.skywalking.apm.agent.core.context.AbstractTracerContext getPrimaryTraceId '{target, returnObj}' '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")' -x 6

wanda事件线程的traceId是谁新生成的？

这些操作是否合理？

使用stack命令，可以查看生成新的全局traceId的调用栈。 根据调用栈，traceId是由Guava事件总线的订阅者Subscriber.invokeSubscriberMethod触发生成的。

[arthas@7]$ stack org.apache.skywalking.apm.agent.core.context.ids.NewDistributedTraceId <init> '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")'
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 432 ms, listenerId: 5
ts=2024-03-05 11:52:45;thread_name=wanda_event-thread-1;id=f6;is_daemon=false;priority=5;TCCL=org.springframework.boot.loader.LaunchedURLClassLoader@8dfe921
    @org.apache.skywalking.apm.agent.core.context.ids.NewDistributedTraceId.<init>()
        at org.apache.skywalking.apm.agent.core.context.trace.TraceSegment.<init>(TraceSegment.java:74)
        at org.apache.skywalking.apm.agent.core.context.TracingContext.<init>(TracingContext.java:122)
        at org.apache.skywalking.apm.agent.core.context.ContextManagerExtendService.createTraceContext(ContextManagerExtendService.java:91)
        at org.apache.skywalking.apm.agent.core.context.ContextManager.getOrCreate(ContextManager.java:60)
        at org.apache.skywalking.apm.agent.core.context.ContextManager.createLocalSpan(ContextManager.java:123)
        // guava-eventbus-plugin
        // 调用方法拦截器
        at org.apache.skywalking.apm.plugin.guava.eventbus.EventBusSubscriberInterceptor.beforeMethod(EventBusSubscriberInterceptor.java:38)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInterWithOverrideArgs.intercept(InstMethodsInterWithOverrideArgs.java:75)
        // 原生方法
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:-1)
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:145)
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:73)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

其是由apm-guava-eventbus-plugin插件的EventBusSubscriberInstrumentation操作改变字节码。

在wanda事件线程追踪段中，查看在哪些地方获取traceId？

这些操作是否合理？

XxxJsonLayout.addCustomDataToJsonMap(XxxJsonLayout.java:29)方法中有调用TraceContext.traceId()。

[arthas@7]$ stack org.apache.skywalking.apm.agent.core.context.AbstractTracerContext getReadablePrimaryTraceId '@java.lang.Thread@currentThread().getName().startsWith("wanda_event")'
Press Q or Ctrl+C to abort.
Affect(class count: 2 , method count: 1) cost in 423 ms, listenerId: 3
ts=2024-03-04 21:03:59;thread_name=wanda_event-thread-1;id=140;is_daemon=false;priority=5;TCCL=org.springframework.boot.loader.LaunchedURLClassLoader@67fe380b
    @org.apache.skywalking.apm.agent.core.context.TracingContext.getReadablePrimaryTraceId()
        at org.apache.skywalking.apm.agent.core.context.ContextManager.getGlobalTraceId(ContextManager.java:77)
        at org.apache.skywalking.apm.toolkit.activation.trace.TraceIDInterceptor.beforeMethod(TraceIDInterceptor.java:35)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:73)
        at org.apache.skywalking.apm.toolkit.trace.TraceContext.traceId(TraceContext.java:-1)
        // SkyWalking核心链路是上面👆🏻
        // 调用TraceContext.traceId()的地点
        at com.leoao.lpaas.logback.LeoaoJsonLayout.addCustomDataToJsonMap(LeoaoJsonLayout.java:29)
        at ch.qos.logback.contrib.json.classic.JsonLayout.toJsonMap(null:-1)
        at ch.qos.logback.contrib.json.classic.JsonLayout.toJsonMap(null:-1)
        at ch.qos.logback.contrib.json.JsonLayoutBase.doLayout(null:-1)
        at ch.qos.logback.core.encoder.LayoutWrappingEncoder.encode(LayoutWrappingEncoder.java:115)
        at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:230)
        at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
        at ch.qos.logback.classic.Logger.info(Logger.java:583)
        // 输出打印日志
        // log.info("receive event persistUserPositionEvent=[{}]", event);
        at com.lefit.wanda.domain.event.listener.PersistUserPositionEventListener.change(PersistUserPositionEventListener.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod$original$ToNcZpNk(Subscriber.java:88)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod$original$ToNcZpNk$accessor$utMvob4N(Subscriber.java:-1)
        at com.google.common.eventbus.Subscriber$auxiliary$8fYqzzq0.call(null:-1)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInterWithOverrideArgs.intercept(InstMethodsInterWithOverrideArgs.java:85)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:-1)
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:145)
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:73)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

观测CorrelationContext.put(key, value)操作后，关联上下文的状态变化过程

[arthas@1]$ watch org.apache.skywalking.apm.agent.core.context.CorrelationContext put '{params,target,returnObj}'  -n 5  -x 3 

ts=2024-06-10 15:01:22; [cost=0.134165ms] result=@ArrayList[
    @Object[][
        @String[scene-label],
        @String[biz-route],
    ],
    @CorrelationContext[
        data=@ConcurrentHashMap[
            @String[scene-label]:@String[biz-route],
        ],
        AUTO_TAG_KEYS=@ArrayList[
            @String[cyborg-flow],
            @String[scene-label],
            @String[scene-tag],
        ],
    ],
]

ts=2024-06-10 15:01:22; [cost=0.01605ms] result=@ArrayList[
    @Object[][
        @String[cyborg-flow],
        @String[true],
    ],
    @CorrelationContext[
        data=@ConcurrentHashMap[
            @String[scene-label]:@String[biz-route],
            @String[cyborg-flow]:@String[true],
        ],
        AUTO_TAG_KEYS=@ArrayList[
            @String[cyborg-flow],
            @String[scene-label],
            @String[scene-tag],
        ],
    ],
]

ts=2024-06-10 15:01:22; [cost=0.007927ms] result=@ArrayList[
    @Object[][
        @String[scene-tag],
        @String[stress-test],
    ],
    @CorrelationContext[
        data=@ConcurrentHashMap[
            @String[scene-label]:@String[biz-route],
            @String[cyborg-flow]:@String[true],
            @String[scene-tag]:@String[stress-test],
        ],
        AUTO_TAG_KEYS=@ArrayList[
            @String[cyborg-flow],
            @String[scene-label],
            @String[scene-tag],
        ],
    ],
]

观测提供者上下文拦截器ContextFilter.invoke(invoker, invocation)的调用对象

[arthas@7]$ watch org.apache.dubbo.rpc.filter.ContextFilter invoke '{params[1].getAttachments(),returnObj}'  -n 5  -x 3 

ts=2024-06-10 15:16:30; [cost=24.479313ms] result=@ArrayList[
    @ObjectToStringMap[
        @String[traceid]:@String[0a57ddf0732748208240f278a248de88.66.17181765903061237],
        @String[x-request-id]:@String[0fe97869-15d9-452f-9374-228f23e56f43],
        @String[x-forwarded-proto]:@String[http],
        @String[sw8-correlation]:@String[c2NlbmUtbGFiZWw=:Yml6LXJvdXRl,Y3lib3JnLWZsb3c=:dHJ1ZQ==,c2NlbmUtdGFn:c3RyZXNzLXRlc3Q=],
        @String[timeout]:@String[5000],
        @String[generic]:@String[gson],
        @String[x-envoy-attempt-count]:@String[1],
        @String[remote.application]:@String[xxx-reactor-gateway],
        @String[sw8-x]:@String[0- ],
        @String[sw8]:@String[1-MGE1N2RkZjA3MzI3NDgyMDgyNDBmMjc4YTI0OGRlODguNjYuMTcxODE3NjU5MDMwNjEyMzc=-MGE1N2RkZjA3MzI3NDgyMDgyNDBmMjc4YTI0OGRlODguNjYuMTcxODE3NjU5MDMwODEyMzg=-0-bGVmaXQtcmVhY3Rvci1nYXRld2F5fHxzaXQ=-ZjdjNjRjNjcwYjcyNDkxZGFmNGQ5YTIyOTc5ZGZjZjdAMTkyLjE2OC4xMTAuMjUx-bnVsbC5nZXRBZHZlcnRpc2VDb25maWdOZXcoKQ==-c2l0L2xlZml0LWNtcy5zaXQuc3ZjLmNsdXN0ZXIubG9jYWw6MA==],
        @String[x-forwarded-client-cert]:@String[By=spiffe://cluster.local/ns/sit/sa/default;Hash=7e7ef818f1a9cd3156d98010276ff6004b5439ce8548d1b5972066e4138a8e0f;Subject="";URI=spiffe://cluster.local/ns/sit/sa/default],
        @String[id]:@String[605975],
    ],
]

@String[sw8-correlation]:@String[c2NlbmUtbGFiZWw=:Yml6LXJvdXRl,Y3lib3JnLWZsb3c=:dHJ1ZQ==,c2NlbmUtdGFn:c3RyZXNzLXRlc3Q=],

观测RPC服务提供者上下文的附加参数

RpcContext.getServiceContext().getRemoteApplicationName()
RpcContext.getServiceContext().getObjectAttachments()
RpcContext.getServerAttachment().getObjectAttachments()
RpcContextAttachment.getObjectAttachments()

服务网格的边车代理模式下，在服务提供者的访问日志Filter，打印远程消费者的应用名称。

[arthas@7]$ watch org.apache.dubbo.rpc.filter.AccessLogFilter invoke '@org.apache.dubbo.rpc.RpcContext@getServiceContext().getRemoteApplicationName()' -x 2 
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 425 ms, listenerId: 9
method=org.apache.dubbo.rpc.filter.AccessLogFilter.invoke location=AtExit
ts=2025-04-11 16:16:56; [cost=20.650966ms] result=@String[taobao-reactor-gateway]

[arthas@7]$ watch com.dubbo.xxx.user.service.XxxProviderService doCodeLogin '@org.apache.dubbo.rpc.RpcContext@getServerAttachment().getObjectAttachments()' -x 2 

ts=2024-11-19 17:47:50; [cost=112.214342ms] result=@HashMap[
    @String[traceid]:@String[e08607b038634e97b565580b6a4d49ac.68.17320096702815325],
    @String[xxx-tenantid]:@String[1],
    @String[mock_now_time]:@String[1731319316000],
    @String[sw8-correlation]:@String[],
    @String[remote.application]:@String[xxx-reactor-gateway],
    @String[sw8]:@String[1-ZTA4NjA3YjAzODYzNGU5N2I1NjU1ODBiNmE0ZDQ5YWMuNjguMTczMjAwOTY3MDI4MTUzMjU=-ZTA4NjA3YjAzODYzNGU5N2I1NjU1ODBiNmE0ZDQ5YWMuNjguMTczMjAwOTY3MDI4MjUzMjY=-0-bGVmaXQtcmVhY3Rvci1nYXRld2F5fHx1YXQ=-Y2JmMzAxOGY3YjEzNDMyYmJlYzgzZWRmNmVhN2I1MDRAMTkyLjE2OC4xMTEuMTkz-Y29tLmxlZml0LnVzZXIuY2xpZW50LmNvcmUuUGhwRnJvbnRBcHBBcGkuZG9Db2RlTG9naW4oKQ==-dWF0LzE5Mi4xNjguMTExLjE5Mzow],
    @String[xxx-devid]:@String[befca983c695d219],
    @String[xxx-forwarded-for]:@String[172.16.26.173],
    @String[xxx-city]:@String[2],
    @String[id]:@String[126116],
    @String[xxx-appv]:@String[6.11.0],
    @String[xxx-screen]:@String[1080,2028,px],
    @String[cookie]:@String[],
    @String[xxx-user-agent]:@String[MagiVersion:2.13.1/XxxAPP/6.11.0 (Android;12;phone;Google;Pixel 3;hdpi)],
    @String[xxx-network]:@String[WIFI],
    @String[xxx-sdkv]:@String[1.0.0],
    @String[generic]:@String[gson],
    @String[xxx-appb]:@String[467],
    @String[xxx-mtoken]:@String[],
    @String[input]:@String[1940],
    @String[xxx-timestamp]:@String[1732009787324],
    @String[sw8-x]:@String[0- ],
    @String[xxx-appkey]:@String[xxx],
    @String[xxx-mpkey]:@String[none],
    @String[xxx-pv]:@String[1.0],
    @String[test_agent]:@String[true],
]

jvm 相关

https://arthas.aliyun.com/doc/commands.html#jvm-%E7%9B%B8%E5%85%B3

• getstatic - 查看类的静态属性
• ognl - 执行 ognl 表达式
• vmtool - 从 jvm 里查询对象，执行 forceGc

SkyWalking链路追踪

关联上下文的自动跨度标签的键集合未配置未生效

【原因】可能是当时改了application.yml文件中的searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:配置项，OAP和UI未重启。OAP和UI都需要配置和重启，这样才能一起生效。

1.业务应用侧，skywalking-agent.jar使用方

[arthas@1]$ getstatic org.apache.skywalking.apm.agent.core.context.CorrelationContext AUTO_TAG_KEYS -x 3 
field: AUTO_TAG_KEYS
@ArrayList[
    @String[sw8_userId],
    @String[scene.label],
    @String[scene],
]

[arthas@1]$ getstatic org.apache.skywalking.apm.agent.core.conf.Config$Correlation AUTO_TAG_KEYS -x 3 
field: AUTO_TAG_KEYS
@String[sw8_userId,scene.label,scene]

[arthas@1]$ getstatic org.apache.skywalking.apm.agent.core.conf.Config$Correlation ELEMENT_MAX_NUMBER -x 3 
field: ELEMENT_MAX_NUMBER
@Integer[8]

2.SkyWalking OAP/UI服务端

[arthas@1]$ vmtool -x 3 --action getInstances --className org.apache.skywalking.oap.server.core.CoreModuleConfig  --express 'instances[0].searchableTracesTags' 
@String[http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker,sw8_userId,scene.label,scene]

[arthas@1]$ vmtool -x 3 --action getInstances --className org.apache.skywalking.oap.server.core.config.SearchableTracesTagsWatcher  --express 'instances[0].searchableTags' 
@HashSet[
    @String[db.instance],
    @String[mq.topic],
    @String[http.status_code],
    @String[db.type],
    @String[scene.label],
    @String[mq.queue],
    @String[sw8_userId],
    @String[http.method],
    @String[rpc.status_code],
    @String[mq.broker],
    @String[scene],
]

Sentinel 控制台流量治理

查看控制台配置项的值

ognl -x 3 '@com.alibaba.csp.sentinel.dashboard.config.DashboardConfig@getRemoveAppNoMachineMillis()' 


ognl -c 629adce '@com.alibaba.csp.sentinel.dashboard.config.DashboardConfig@getUnhealthyMachineMillis()' 
60000

ognl -c 138caeca '@com.alibaba.csp.sentinel.dashboard.config.DashboardConfig@getAutoRemoveMachineMillis()' 
0
300000

ognl -c 629adce '@com.alibaba.csp.sentinel.dashboard.config.DashboardConfig@getRemoveAppNoMachineMillis()' 
0