etcd报错：failed to send out heartbeat on time

最新推荐文章于 2025-05-26 16:57:33 发布
转载 最新推荐文章于 2025-05-26 16:57:33 发布 · 6.1k 阅读 · 3 ·
CC 4.0 BY-SA版权
原文链接：http://www.cnblogs.com/mldblue/articles/10980955.html
文章标签：

#数据库 #网络 #运维

本文详细解析了Etcd心跳超时错误的原因及解决策略，包括磁盘速度慢、CPU性能不足和网络不稳定等问题，提供了针对不同场景的优化建议。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

报错内容：

2019-06-05 02:09:03.008888 W | rafthttp: health check for peer 8816eaa680e63c73 could not connect: dial tcp 192.168.49.138:2380: connect: connection refused (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-06-05 02:09:03.010827 W | rafthttp: health check for peer 8816eaa680e63c73 could not connect: dial tcp 192.168.49.138:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
2019-06-05 02:09:04.631367 I | rafthttp: peer 8816eaa680e63c73 became active
2019-06-05 02:09:04.631405 I | rafthttp: established a TCP streaming connection with peer 8816eaa680e63c73 (stream MsgApp v2 reader)
2019-06-05 02:09:04.632227 I | rafthttp: established a TCP streaming connection with peer 8816eaa680e63c73 (stream Message reader)
2019-06-05 02:09:04.634697 I | rafthttp: established a TCP streaming connection with peer 8816eaa680e63c73 (stream MsgApp v2 writer)
2019-06-05 02:09:04.635154 I | rafthttp: established a TCP streaming connection with peer 8816eaa680e63c73 (stream Message writer)
2019-06-05 02:09:04.961320 I | etcdserver: updating the cluster version from 3.0 to 3.3
2019-06-05 02:09:04.965052 N | etcdserver/membership: updated the cluster version from 3.0 to 3.3
2019-06-05 02:09:04.965231 I | etcdserver/api: enabled capabilities for version 3.3

2019-06-05 02:20:39.344648 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 237.022208ms, to a3d1fb0d28ed2953)
2019-06-05 02:20:39.344676 W | etcdserver: server is likely overloaded
2019-06-05 02:20:39.344685 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 237.127928ms, to 8816eaa680e63c73)
2019-06-05 02:20:39.344689 W | etcdserver: server is likely overloaded

报错信息主要为：failed to send out heartbeat on time (exceeded the 100ms timeout for 401.80886ms)

心跳检测报错主要与以下因素有关（磁盘速度、cpu性能和网络不稳定问题）：

  • etcd使用了raft算法，leader会定时地给每个follower发送心跳，如果leader连续两个心跳时间没有给follower发送心跳，etcd会打印这个log以给出告警。通常情况下这个issue是disk运行过慢导致的，leader一般会在心跳包里附带一些metadata，leader需要先把这些数据固化到磁盘上，然后才能发送。写磁盘过程可能要与其他应用竞争，或者因为磁盘是一个虚拟的或者是SATA类型的导致运行过慢，此时只有更好更快磁盘硬件才能解决问题。etcd暴露给Prometheus的metrics指标walfsyncduration_seconds就显示了wal日志的平均花费时间，通常这个指标应低于10ms。

  • 第二种原因就是CPU计算能力不足。如果是通过监控系统发现CPU利用率确实很高，就应该把etcd移到更好的机器上，然后通过cgroups保证etcd进程独享某些核的计算能力，或者提高etcd的priority。

  • 第三种原因就可能是网速过慢。如果Prometheus显示是网络服务质量不行，譬如延迟太高或者丢包率过高，那就把etcd移到网络不拥堵的情况下就能解决问题。但是如果etcd是跨机房部署的，长延迟就不可避免了，那就需要根据机房间的RTT调整heartbeat-interval，而参数election-timeout则至少是heartbeat-interval的5倍。

参考
https://blog.youkuaiyun.com/linux_player_c/article/details/79875806

转载于:https://www.cnblogs.com/mldblue/articles/10980955.html

【踩坑日志】微服务集成 Seata启动报错：Failed to get available servers: service.vgroupMapping.xxx configuration item
yunnuo的博客
10-26 1298
该报错是表明 Seata 无法找到 service.vgroupMapping 配置项中指定的事务组（smart-construction-auth-group）。这个配置项是 Seata 用来映射事务组到服务组的，从而确定应该连接到哪个 Seata 服务器集群。 从哪些方面进行排查呢： 1. seata服务的配置文件(老版本是：`registry.conf`，新版本：`application.ym`l） 2. nacos的seata的配置文件（`seata-server.properties` 或者 `s
报错：The client socket has failed to connect to...如何解决？
**My Coding Family**
04-16 988
🏆本文收录于《全栈Bug调优(实战版)》专栏，主要记录项目实战过程中所遇到的Bug或因后果及提供真实有效的解决方案，希望能够助你一臂之力，帮你早日登顶实现财富自由🚀；同时，欢迎大家关注&&收藏&&订阅！持续更新中，up！up！up！！ 备注：部分问题/疑难杂症搜集于互联网。
etcd 日志 timeout
喝醉酒的小白
05-31 1781
具体地，该消息中的数字 “156.ms” 表示发送心跳的时间，而 “100ms” 则是预期的时间限制。预期的心跳发送间隔一般为 100ms，如果某个节点在发送心跳时超过了这个时间限制（如本例中的 156ms），就会出现该错误消息。优化集群负载：如果 etcd 集群的负载过重，可以考虑优化应用程序或减少其他任务的负载，以确保节点能够及时发送心跳信号。检查网络：确保节点之间的网络连接稳定，并且延迟较低。通过以上步骤，可以改善 etcd 服务器发送心跳信号的性能，减少超时错误的发生，提高集群的可靠性和性能。
故障排查：PD 的 leader 切换，某 tikv 的 leader 被驱逐
TiDB 社区干货传送门
06-11 525
作者： 这里介绍不了我 原文来源： https://tidb.net/blog/e28ecf2b ...
etcd日志分析：failed to send out heartbeat on time
喝醉酒的小白
05-26 541
心跳延迟与过载告警读请求耗时严重超标GRPC 通信异常Raft 选举日志[Node A] became candidate -> 发起新一轮投票[Node A] 拒绝来自其他节点的投票请求，因 log index 更高Kubelet 重启失败。
好文推荐 | etcd 问题、调优、监控
朱小厮的博客
05-27 4432
点击上方“朱小厮的博客”，选择“设为星标”后台回复"加群"，加入新技术来源：www.xuyasong.com/?p=1983etcd 原理解析：读《etcd 技术内幕》这...
etcd系列-----集大成者etcdserver 模块
cyq6239075的博客
04-16 2156
1、raftNode 相关 raftNode是充当raft模块与上层模块之间交互的桥梁 term ( uint64类型）： 当前节点己应用Entry记录的最大任期号，term、 index和lead三个宇段的读写都是原子操作。 index ( uint64类型）： 当前节点中己应用Entry记录的最大索引值。 lead ( uint64类型）：记录当前集群中Leader节...
【kubernetes/k8s概念】k8s 坑问题汇总
热门推荐
zhonglinzhang
12-26 3万+
1.Pod始终处于Pending状态 如果Pod保持在Pending的状态，意味着无法被正常的调度到节点上。由于某种系统资源无法满足Pod运行的需求 系统没有足够的资源：已经用尽了集群中所有的CPU或内存资源。需要清理一些不在需要的Pod，调整它们所需的资源量，或者向集群中增加新的节点。 用户指定了hostPort：通过hostPort用户能够将服务暴露到指定的主机端口上，会限......
报错：Heartbeating to master:7182 failed.
weixin_30325971的博客
04-29 3291
报错背景： cloudera-scm-agent 可以启动并且存活，但是jps没有进程。 报错现象： 查看报错日志：/opt/cm-5.15.1/log/cloudera-scm-agent/cloudera-scm-agent.log [29/Apr/2019 14:27:52 +0000] 10528 MainThread throttling_logger INFO ...
ETCD报错「raft: elected leader is outdated」：集群脑裂与选举超时的配置优化
shejizuopin的博客
05-14 804
关键避坑点避免跨机房部署奇数节点在跨机房场景下，建议采用2n+1节点（如5节点跨3机房），避免因单节点故障导致多数派不可用。禁止手动强制覆盖Leader使用可能引发数据回滚，应优先通过清理故障节点。定期演练数据恢复每季度执行一次与演练，确保恢复流程可靠。
ceph报错：Long heartbeat ping times on back interface seen, Long heartbeat ping times on front interfac
liuchao2013的博客
04-23 3368
[root@ceph2 ~]# ceph status cluster: id: 80ce8b29-a744-43f1-9457-c9c7c9df2027 health: HEALTH_WARN Long heartbeat ping times on back interface seen, longest is 7643.739 msec Long heartbeat ping times on front interface ...
CentOS7部署etcd---问题排查
Entity_G的博客
12-23 1818
CentOS7部署etcd—问题排查（学习笔录） 在学习kubernetes，安装etc服务时。出现了一下问题做个小笔记 1、执行etcd时：-bash: ./etcd: 无法执行二进制文件 通过下载其他版本，成功执行 2、使用supervisord服务管理etcd无法成功启动 [root@hdss7-12 etcd]# systemctl restart supervisord [root@hdss7-12 etcd]# supervisorctl update [root@hdss7-12 etcd]#
spark core 2.0 Executor Heartbeat
houzhizhen的专栏
01-10 3184
Executor heartbeat 调度一个任务向driver汇报心跳和部分度量信息 /** * Schedules a task to report heartbeat and partial metrics for active tasks to driver. */ private def startDriverHeartbeater(): Unit = {
Spark RPC 框架源码分析（三）Spark 心跳机制分析
zzzzMing -大数据/数据分析/数据挖掘
01-17 710
一. Spark 心跳概述 前面两节中介绍了 Spark RPC 的基本知识，以及深入剖析了 Spark RPC 中一些源码的实现流程。 具体可以看这里： Spark RPC 框架源码分析（二）运行时序 Spark RPC 框架源码分析（一）简述 这一节我们来看看一个 Spark RPC 中的运用实例 – Spark 的心跳机制。这次主要还是从代码的角度来看。 我们首先要知道 Spark 的...
kafka服务端报错: WARN [SocketServer listenerType=ZK_BROKER, nodeId=1] Unexpected error from
qq_43071699的博客
01-19 837
异常，并因此关闭了与客户端的连接。这个异常通常发生在Kafka尝试读取网络数据包时收到无效的数据大小（例如负数），这在你的日志中表现为。Kafka服务器（Broker）遇到了。
7.Sentinel源码分析—Sentinel是怎么和控制台通信的？
weixin_30782293的博客
09-28 1329
这里会介绍： Sentinel会使用多线程的方式实现一个类Reactor的IO模型 Sentinel会使用心跳检测来观察控制台是否正常 Sentinel源码解析系列： 1.Sentinel源码分析—FlowRuleManager加载规则做了什么？ 2. Sentinel源码分析—Sentinel是如何进行流量统计的？ 3. Sentinel源码分析— QPS流量控制是如何实现的？ 4...
kubernetes学习：7.部署master节点
linux_player_c（系统&开发）
04-11 5424
k8s部署master节点 在之前的章节介绍过，k8s的控制节点扮演者整个调度和管理的角色，所以是非常关键的一部分。k8s的master节点主要包含三个部分： 1. kube-apiserver 提供了统一的资源操作入口; 2. kube-scheduler 是一个资源调度器，它根据特定的调度算法把pod生成到指定的计算节点中; 3. kube-controller-manager 也...
k8s集群部署之环境介绍与etcd数据库集群部署
weixin_30510153的博客
04-19 324
角色 IP 组件 配置 master-1 192.168.10.11 kube-apiserver kube-controller-manager kube-scheduler etcd 2c 2g master-2 192.168.10.12 kube-apiserver kube-controller-manager kub...
K8S学习笔记之ETCD启动失败注意事项
dengxiangbao3167的博客
03-26 4331
最近搭建K8S集群遇到ETCD的报错，报错信息如下，一定要关闭防火墙、iptables和SELINUX，三个都要关闭！！ Mar 26 20:39:24 k8s-m1 etcd[6437]: health check for peer 3de62d4888b330ab could not connect: dial tcp 192.168.26.137:2380: conne...
UART1 and UART2 initialized. ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite rbCreate Success ?? Warning:gizProtocolResendData 624 2 0Warning: timeout, resend data ?? valueLED_OnOff Changedchanged, report data?? r塛arning:gizProtocolResendData 924 637 0Warning: timeout, resend data ?? r塛arning:gizProtocolResendData 1220 932 1Warning: timeout, resend data ?? r塃RR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite valuelight Changedchanged, report data?? q塛arning:gizProtocolResendData 7884 7597 0Warning: timeout, resend data ?? q塛arning:gizProtocolResendData 8180 7892 1Warning: timeout, resend data ?? q塿alueLED_OnOff Changedchanged, report data?? 剾Warning:gizProtocolResendData 9343 9056 0Warning: timeout, resend data ?? 剾valueLED_OnOff Changedchanged, report data?? q媁arning:gizProtocolResendData 9645 9357 0Warning: timeout, resend data ?? q媁arning:gizProtocolResendData 9940 9653 1Warning: timeout, resend data ?? q媣alueLED_OnOff Changedchanged, report data?? ?Warning:gizProtocolResendData 11679 11392 0Warning: timeout, resend data ?? ?Warning:gizProtocolResendData 11975 11688 1Warning: timeout, resend data ?? ?valueLED_OnOff Changedchanged, report data?? t怶arning:gizProtocolResendData 12565 12277 0Warning: timeout, resend data ?? t恦alueLED_OnOff Changedchanged, report data?? 彧ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite Warning:gizProtocolResendData 12867 12578 0Warning: timeout, resend data ?? 彧Warning:gizProtocolResendData 13164 12876 1Warning: timeout, resend data ?? 彧valueLED_OnOff Changedchanged, report data?? w昗arning:gizProtocolResendData 13465 13178 0Warning: timeout, resend data ?? w昗arning:gizProtocolResendData 13761 13473 1Warning: timeout, resend data ?? w晇alueLED_OnOff Changedchanged, report data??  嚖Warning:gizProtocolResendData 17800 17513 0Warning: timeout, resend data ??  嚖Warning:gizProtocolResendData 18096 17809 1Warning: timeout, resend data ??  嚖valueLED_OnOff Changedchanged, report data?? x榃arning:gizProtocolResendData 20698 20411 0Warning: timeout, resend data ?? x榃arning:gizProtocolResendData 20994 20706 1Warning: timeout, resend data ?? x楨RR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite valuelight Changedchanged, report data??  s擶arning:gizProtocolResendData 26749 26461 0Warning: timeout, resend data ??  s擶arning:gizProtocolResendData 27045 26757 1Warning: timeout, resend data ??  s攙aluelight Changedchanged, report data??  u梂arning:gizProtocolResendData 32808 32520 0Warning: timeout, resend data ??  u梂arning:gizProtocolResendData 33104 32816 1Warning: timeout, resend data ??  u桬RR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite valuelight Changedchanged, report data??  s朩arning:gizProtocolResendData 38869 38581 0Warning: timeout, resend data ??  s朩arning:gizProtocolResendData 39165 38877 1Warning: timeout, resend data ??  s杤aluelight Changedchanged, report data?? q昗arning:gizProtocolResendData 44928 44641 0Warning: timeout, resend data ?? q昗arning:gizProtocolResendData 45224 44936 1Warning: timeout, resend data ?? q旹RR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite valuelight Changedchanged, report data?? s榃arning:gizProtocolResendData 50990 50703 0Warning: timeout, resend data ?? s榃arning:gizProtocolResendData 51286 50998 1Warning: timeout, resend data ?? s榲aluelight Changedchanged, report data?? q梂arning:gizProtocolResendData 57049 56762 0Warning: timeout, resend data ?? q梂arning:gizProtocolResendData 57345 57057 1Warning: timeout, resend data ?? q桬RR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite ERR: Failed to rbWrite 串口调试助手接收到的，然后OLEDLOOP每隔一秒加一，而且显示light正确显示，temp和humi是0
最新发布
07-08
"Warning: timeout, resend data"表明设备与云端通信超时，可能原因： - 网络连接不稳定 - 数据处理耗时过长导致响应延迟 - 串口通信故障 **解决方案：** ##### 优化网络通信线程 ```c void gizwits_thread(void *...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值