一、环境准备
先下载SkyWalking安装包,需要注意的是SkyWalking 版本在10.X以上使用的nacos-client是2.X,如果安装的Nacos版本是1.X就会存在兼容性的问题。由于本人使用的SpringBoot项目是2.7.X版本,安装的Nacos版本只能是1.X版本的,就选择最新的是1.4.8,所以只能选择SkyWalking版本是9.7.0,对应的nacos-client版本是1.4.2。
1、下载安装
wget https://archive.apache.org/dist/skywalking/9.7.0/apache-skywalking-apm-9.7.0.tar.gz
tar -zxvf apache-skywalking-apm-9.7.0.tar.gz
cd apache-skywalking-apm-bin
2、配置集群方式
修改SkyWalking的配置文件 config/application.yml中集群的方式:
cluster:
selector: ${SW_CLUSTER:nacos}
nacos:
serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
hostPort: ${SW_CLUSTER_NACOS_HOST_PORT:10.60.1.63:8848}
namespace: ${SW_CLUSTER_NACOS_NAMESPACE:"public"} # 替换为你的Namespace ID,这里使用默认的命名空间
username: ${SW_CLUSTER_NACOS_USERNAME:"nacos"} # nacos用户名
password: ${SW_CLUSTER_NACOS_PASSWORD:"nacos"} # nacos登录密码
# 高级配置(可选)
clusterName: ${SW_CLUSTER_NACOS_CLUSTER_NAME:"DEFAULT"}
healthCheckInterval: ${SW_CLUSTER_NACOS_HEALTH_CHECK_INTERVAL:5}
3、配置 Elasticsearch 8 存储
关于ES8存储的配置出现了很多问题,搞了几个小时才成功,主要是 安全证书问题,针对该问题,这里会详细描述遇到的问题和解决方案。
首先第一步是使用如下命令将oap-libs中oap-libs/storage-elasticsearch-plugin-9.7.0.jar复制到plugins文件夹下。
# 进入skywalking安装目录下
cd /home/app/apache-skywalking-apm-bin
#创建plugins文件夹
mkdir plugins
#将storage-elasticsearch-plugin-9.7.0.jar拷贝到plugins文件夹下
cp oap-libs/storage-elasticsearch-plugin-9.7.0.jar plugins/
由于Elasticsearch 自动生成的自签名CA证书http_ca.crt
是 PEM 格式证书,但 SkyWalking 9.7.0 默认期望 JKS 或 PKCS12 格式的密钥库。如果不转化就会报如下错误信息:
2025-03-30 07:06:12,544 - org.apache.skywalking.oap.server.starter.OAPServerBootstrap - 64 [main] ERROR [] - Invalid keystore format
org.apache.skywalking.oap.server.library.module.ModuleStartException: Invalid keystore format
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:281) ~[storage-elasticsearch-plugin-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:46) ~[library-module-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:75) ~[library-module-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:52) [server-starter-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:23) [server-starter-9.7.0.jar:9.7.0]
Caused by: java.io.IOException: Invalid keystore format
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:688) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]
所以需要将 PEM 证书转换为 PKCS12 格式(推荐)
# 进入elasticsearch安装包的证书目录
cd /home/app/elasticsearch-8.17.4/config/certs
# 转换证书(无密码版)
openssl pkcs12 -export -nokeys -in http_ca.crt -out http_ca.p12 -passout pass:
# 设置权限
chmod 644 http_ca.p12
接下来执行下面脚本需要验证证书有效性
# 检查PKCS12文件有效性
keytool -list -v -keystore /home/app/elasticsearch-8.17.4/config/certs/http_ca.p12 -storepass ""
验证结果如下:
此时可以配置SkyWalking的application.yml文件的存储模块内容如下:
storage:
selector: ${SW_STORAGE:elasticsearch}
elasticsearch:
nameSpace: ${SW_NAMESPACE:""}
clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:10.60.1.63:9200} # 修改为单节点地址
protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"https"}
trustStorePath: ${SW_STORAGE_ES_SSL_JKS_PATH:"/home/app/elasticsearch-8.17.4/config/certs/http_ca.p12"} # 使用CA证书
trustStorePass: ${SW_STORAGE_ES_SSL_JKS_PASS:""} # 如果证书有密码需填写
user: ${SW_ES_USER:"elastic"}
password: ${SW_ES_PASSWORD:"HAIyi123*"}
indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:1} # 单节点建议设为1
indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0} # 单节点必须设为0
secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:"/home/app/elasticsearch-8.17.4/config/certs/credentials.json"} # 可选密钥文件
由于trustStorePass为空,在启动skywalking时会报如下错误信息:
2025-03-30 07:02:56,422 - org.apache.skywalking.oap.server.starter.OAPServerBootstrap - 64 [main] ERROR [] - Cannot invoke "String.toCharArray()" because "this.trustStorePass" is null
org.apache.skywalking.oap.server.library.module.ModuleStartException: Cannot invoke "String.toCharArray()" because "this.trustStorePass" is null
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:281) ~[storage-elasticsearch-plugin-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:46) ~[library-module-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:75) ~[library-module-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:52) [server-starter-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:23) [server-starter-9.7.0.jar:9.7.0]
Caused by: java.lang.NullPointerException: Cannot invoke "String.toCharArray()" because "this.trustStorePass" is null
at org.apache.skywalking.library.elasticsearch.ElasticSearchBuilder.build(ElasticSearchBuilder.java:167) ~[library-elasticsearch-client-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.connect(ElasticSearchClient.java:152) ~[library-client-9.7.0.jar:9.7.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:268) ~[storage-elasticsearch-plugin-9.7.0.jar:9.7.0]
... 4 more
也可以在执行上面的转换证书是进行加密,如下:
keytool -list -v -keystore /home/app/elasticsearch-8.17.4/config/certs/http_ca.p12 -storepass "HAIyi123*" # 设置证书的密码
然后指定trustStorePass,再次启动skywalking时会报如下错误信息:
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200) ~[?:?]
at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120) ~[?:?]
at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104) ~[?:?]
at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:98) ~[?:?]
at sun.security.validator.Validator.getInstance(Validator.java:181) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:309) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:183) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:255) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:790) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36) ~[netty-tcnative-classes-2.0.61.Final.jar:2.0.61.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48) ~[netty-tcnative-classes-2.0.61.Final.jar:2.0.61.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42) ~[netty-tcnative-classes-2.0.61.Final.jar:2.0.61.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1534) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:96) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1509) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1647) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1493) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1345) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) ~[netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
这表明 Java 安全库无法从您的证书文件中提取可信的 CA 证书链。以下是完整的解决方案:
步骤1:验证证书完整性
# 检查证书内容
openssl x509 -in /home/app/elasticsearch-8.17.4/config/certs/http_ca.crt -noout -text
# 检查证书链(应显示完整的CA链)
openssl crl2pkcs7 -nocrl -certfile /home/app/elasticsearch-8.17.4/config/certs/http_ca.crt | openssl pkcs7 -print_certs -noout
步骤2:重建证书链
如果证书链不完整,手动构建完整链:
# 获取Elasticsearch生成的CA证书
cat /home/app/elasticsearch-8.17.4/config/certs/http_ca.crt > full_chain.crt
# 追加系统CA证书(可选)
cat /etc/ssl/certs/ca-certificates.crt >> full_chain.crt
# 转换为PKCS12格式(必须)
openssl pkcs12 -export -nokeys -in full_chain.crt -out full_chain.p12 -passout pass:
# 设置权限
chmod 644 full_chain.p12
chown skywalking:skywalking full_chain.p12
步骤3:将自签名证书加入Java信任库(推荐)
# 1. 进入证书目录
cd /home/app/elasticsearch-8.17.4/config/certs
# 2. 将CA证书导入Java默认信任库
sudo keytool -importcert \
-alias elasticsearch-ca \
-file http_ca.crt \
-keystore $JAVA_HOME/lib/security/cacerts \
-storepass changeit \
-noprompt
# 3. 修改SkyWalking配置(不再需要指定trustStore)
storage:
elasticsearch:
protocol: "HTTPS"
# 注释掉trustStore相关配置
# trustStorePath: ""
# trustStorePass: ""
user: "elastic"
password: "HAIyi123*"
步骤4:验证Java信任库
keytool -list -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit | grep elasticsearch
应显示:
elasticsearch-ca, Mar 30, 2025, trustedCertEntry
测试HTTPS连接
curl --cacert /home/app/elasticsearch-8.17.4/config/certs/http_ca.crt \
-u elastic:HAIyi123* \
https://10.60.1.63:9200/_cluster/health
应显示:
[root@localhost certs]# curl --cacert /home/app/elasticsearch-8.17.4/config/certs/http_ca.crt -u elastic:HAIyi123* https://10.60.1.63:9200/_cluster/health
{"cluster_name":"my-es-cluster","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":3,"active_shards":3,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"unassigned_primary_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@localhost certs]#
[root@localhost certs]#
二、启动服务
-
启动OAP服务:
cd /home/app/apache-skywalking-apm-bin/bin ./oapService.sh
-
Web UI部署
cd /home/app/apache-skywalking-apm-bin/bin ./webappService.sh
启动后,直接可以在浏览器上输入http://10.60.1.63:8080/打开SkyWalking的页面: