prometheus重启后，遇到level=warn ts=2019-05-13T09:02:15.270Z caller=head.go:454 component=tsdb msg=“unknow

最新推荐文章于 2025-02-25 16:17:43 发布
最新推荐文章于 2025-02-25 16:17:43 发布
阅读量3.3k 收藏
点赞数
CC 4.0 BY-SA版权
分类专栏： 云原生 文章标签： Prometheus 1085错误
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/liuCoding/article/details/108589798
在Prometheus重启后，系统日志显示level=warn，具体为tsdb组件报错，出现'unknown series references'，错误计数达到1085。这个问题涉及到Prometheus的时间序列数据库管理。

prometheus重启后，遇到level=warn ts=2019-05-13T09:02:15.270Z caller=head.go:454 component=tsdb msg="unknown series references" count=1085错误

 

欢迎扫码加群讨论

./generator generate ，snmp_exporter生成 snmp.yml文件
wsyzxss的博客
08-23 2408
[root@xue generator]# pwd /opt/aspire/product/prometheus/snmp_exporter/generator [root@xue generator]# [root@xue generator]# [root@xue generator]# ./generator generate level=info ts=2021-08-23T09:44:37.670Z caller=net_snmp.go:142 msg="Loading...
【云原生】prometheus报错排查
zlb15250644563的博客
03-22 709
百度了下说是"Tombstones header: invalid size" 这个错误信息通常与 Prometheus 的数据块（block）或墓碑文件（tombstone file）的头部大小不正确有关。在 Prometheus 中，墓碑文件是用来标记那些已经从磁盘中删除或清理的数据块的，以避免在重新启动时重新处理这些旧数据。进入到对应节点所配置的存储目录/data，进入01HRVMTHEGRSGZZXV7A239KRNG文件，也没有查出什么问题，磁盘空间也是正常的。
启动prometheus报错
qq_45814815的博客
11-02 4125
启动prometheus报错
Kuboard无法发送邮件和钉钉告警问题解决
Devin's Blog
05-28 3249
Kuboard无法发送邮件和钉钉告警问题解决
{“level“:“warn“,“ts“:“2023-09-11T21:07:50.897791+0800“,“logger“:“etcd-client“,“caller“:“v3@v3.5.8/re
weixin_39608351的博客
09-11 939
运行go获取数据库，报以上错误，重启etcd即可。
prometheus 部署过程中出现的问题
wuzhi222的博客
07-21 6960
1、level=error ts=2021-07-20T08:43:06.984Z caller=main.go:727 err="error loading config from \"prometheus.yml\": couldn't load configuration (--config.file=\"prometheus.yml\"): parsing YAML file prometheus.yml: yaml: line 29: did not find expected key" 出现原
prometheus 日志打印 caller=compact.go:460 level=info component=tsdb msg=“compact blocks” count=3
喝醉酒的小白
12-20 663
因此，如果你的 Prometheus 服务器在处理大量的时间序列数据，或者是在一个资源有限的环境中运行，那么数据块的压缩操作可能会对性能产生影响。是的，Prometheus 的 Time Series Database (TSDB) 组件在执行数据块的压缩操作时会占用系统的 IO（输入/输出）和 CPU 资源。总的来说，此日志表示 Prometheus 的 TSDB 组件正在执行数据块的压缩操作，这是Prometheus用于存储优化和空间管理的常规操作，没有任何问题。这两个过程都会消耗大量的 IO 资源。
-06-17_06:09:24.66584 time="2025-06-17T14:09:24+08:00" level=info msg="Providing metrics at localhost:9121/metrics" ==> /var/log/gitlab/prometheus/current <== 2025-06-17_06:09:26.82251 ts=2025-06-17T06:09:26.815Z caller=manager.go:292 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes config=kubernetes-pods 2025-06-17_06:09:26.82251 ts=2025-06-17T06:09:26.815Z caller=manager.go:292 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes config=kubernetes-cadvisor 2025-06-17_06:09:26.82252 ts=2025-06-17T06:09:26.815Z caller=manager.go:292 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes config=kubernetes-nodes 2025-06-17_06:09:26.82252 ts=2025-06-17T06:09:26.821Z caller=main.go:1441 level=info msg="updated GOGC" old=100 new=75 2025-06-17_06:09:26.82252 ts=2025-06-17T06:09:26.821Z caller=main.go:1452 level=info msg="Completed loading of configuration file" filename=/var/opt/gitlab/prometheus/prometheus.yml totalDuration=7.99528ms db_storage=6.104µs remote_storage=1.711µs web_handler=480ns query_engine=3.189µs scrape=168.192µs scrape_sd=305.836µs notify=26.207µs notify_sd=6.99µs rules=6.254412ms tracing=10.127µs 2025-06-17_06:09:26.82252 ts=2025-06-17T06:09:26.821Z caller=main.go:1182 level=info msg="Server is ready to receive web requests." 2025-06-17_06:09:26.82253 ts=2025-06-17T06:09:26.821Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..." 2025-06-17_06:09:31.84939 ts=2025-06-17T06:09:31.848Z caller=manager.go:186 level=error component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-cadvisor 2025-06-17_06:09:31.84942 ts=2025-06-17T06:09:31.848Z caller=manager.go:186 level=error component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-nodes 2025-06-17_06:09:31.84942 ts=2025-06-17T06:09:31.848Z caller=manager.go:186 level=error component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to read CA cert: unable to read file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-pods ==> /var/log/gitlab/alertmanager/current <== 2025-06-17_06:09:06.82586 ts=2025-06-17T06:09:06.822Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=master, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2025-06-17_06:09:06.82589 ts=2025-06-17T06:09:06.822Z caller=main.go:182 level=info build_context="(go=go1.23.2, platform=linux/amd64, user=GitLab-Omnibus, date=, tags=unknown)" 2025-06-17_06:09:06.95589 ts=2025-06-17T06:09:06.944Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=192.168.58.161 port=9094 2025-06-17_06:09:06.98836 ts=2025-06-17T06:09:06.972Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2025-06-17_06:09:07.06211 ts=2025-06-17T06:09:07.060Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2025-06-17_06:09:07.06213 ts=2025-06-17T06:09:07.060Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2025-06-17_06:09:07.10829 ts=2025-06-17T06:09:07.108Z caller=tls_config.go:313 level=info msg="Listening on" address=127.0.0.1:9093 2025-06-17_06:09:07.10831 ts=2025-06-17T06:09:07.108Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9093 2025-06-17_06:09:08.97340 ts=2025-06-17T06:09:08.973Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000709881s 2025-06-17_06:09:17.00509 ts=2025-06-17T06:09:17.005Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.032381279s ==> /var/log/gitlab/postgres-exporter/current <== 2025-06-17_06:03:31.65751 ts=2025-06-17T06:03:31.654Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9187 2025-06-17_06:03:42.90526 ts=2025-06-17T06:03:42.904Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2025-06-17_06:03:43.05861 ts=2025-06-17T06:03:43.054Z caller=postgres_exporter.go:613 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=14.17.0 2025-06-17_06:09:22.08609 ts=2025-06-17T06:09:22.084Z caller=main.go:86 level=warn msg="Error loading config" err="Error opening config file \"postgres_exporter.yml\": open postgres_exporter.yml: no such file or directory" 2025-06-17_06:09:22.08611 ts=2025-06-17T06:09:22.085Z caller=proc.go:272 msg="Excluded databases" databases=[] 2025-06-17_06:09:22.08611 ts=2025-06-17T06:09:22.085Z caller=main.go:99 level=warn msg="The extended queries.yaml config is DEPRECATED" file=/var/opt/gitlab/postgres-exporter/queries.yaml 2025-06-17_06:09:22.08611 ts=2025-06-17T06:09:22.085Z caller=tls_config.go:274 level=info msg="Listening on" address=127.0.0.1:9187 2025-06-17_06:09:22.08611 ts=2025-06-17T06:09:22.086Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9187 2025-06-17_06:09:42.90780 ts=2025-06-17T06:09:42.906Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2025-06-17_06:09:42.95498 ts=2025-06-17T06:09:42.953Z caller=postgres_exporter.go:613 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=14.17.0 ==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2025-06-17T14:13:00+08:00","uri":""}
最新发布
06-19
首先，用户的问题是：“我想解决GitLab相关组件（Prometheus、Alertmanager、PostgresExporter、GitLabWorkhorse）的日志错误问题请问GitLabPrometheusAlertmanagerPostgresExporterWorkhorse日志错误分析及解决方案...
监控实践：promethus step by step
CuriousAn的博客
08-06 733
“纸上得来终觉浅，绝知此事要躬行” “实践出真知” 都在告诉我们，想要真正的了解，就要充分的接触。为了更好的了解promethus，我们今天就动手安装promethus+node-exporter+grafana体验体验 第一步：环境准备 我们在某云海外局点购买虚拟机一台，配置如下： CPU 2vCPU 内存 4GiB OS CentOS8.2 存储 40GiB系统盘 网络 300Mbit/s公网IP 设置防火墙规则： 方向 协议端口
Loki query报错解决方法
booklijian的专栏
04-18 1376
Loki query 查询出错，如何解决
ETCD坑：挂载旧集群数据
qq_37106501的博客
03-17 1807
ETCD坑：挂载旧集群数据
单节点prometheus
Lori_jishumeng123的博客
11-23 880
单节点搭建： zabbix server搭建过程了解 采集的数据默认在./data中，默认以2h的数据存储为一个block，https://www.ctolib.com/docs/sfile/prometheus-book/ha/prometheus-local-storage.html 告警配置如何生效？确定当前配置的告警配置哪里有问题？ 未生效原因及配置的主要点： rules file中的...
prometheus 异常退出 报错：opening storage failed
weixin_46478763的博客
06-30 7163
prometheus opening storage failed
prometheus错题集
志当存高远
06-14 1242
prometheus错题集问题一、level=error ts=2021-06-14T03:32:31.473Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=file_sd/targets/nodes_centos.yaml err="yaml: line 2: did not find expected '-' indicator" 问题一、leve
【IT运维】go编写程序充当prometheus告警触发端和接收端
FixPng的博客
07-08 649
编写golang程序充当告警触发端，向alertmanager发送告警 编写golang程序充当告警接收端，从alertmanager 接收webhook的告警信息
记一次prometheus出现CrashLoopBackOff故障
m0_59041021的博客
02-25 445
prometheus-k8s-0 Pod 中的 prometheus 容器启动失败，原因是存储文件损坏：“opening storage failed: /prometheus/chunks_head/000014: invalid magic number 0”。发现故障pod的挂载文件bound在本地，查找具体路径。挂载路径在/var/openebs/local/发现pod中有两个容器，查看故障日志。1.查看具体故障容器，查看日志。删除故障文件重构，查找挂载文件。
references an unknown entity
淘技术
07-06 696
注意实体类注解开发，需要在类上面添加注解@Entity
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

JAVA程序哥

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值