kubelet 报错： Orphaned pod found - but volume paths are still present on disk

最新推荐文章于 2025-07-06 18:12:08 发布
转载 最新推荐文章于 2025-07-06 18:12:08 发布 · 6.2k 阅读 · 2
文章标签：

#kubelet

本文解决了一个常见的Kubernetes问题，即孤儿Pod在kubelet目录中留下残留的volume路径，导致日志错误。通过检查etc-hosts文件找到Pod名称，手动删除遗留的Pod目录，最终解决了日志持续刷屏的问题。 
kubelet_volumes.go:128] Orphaned pod "86d60ee9-9fae-11e8-8cfc-525400290b20" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
kubelet_volumes.go:128] Orphaned pod "86d60ee9-9fae-11e8-8cfc-525400290b20" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
kubelet_volumes.go:128] Orphaned pod "86d60ee9-9fae-11e8-8cfc-525400290b20" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.

通过id号,进入kubelet的目录,可以发现里面装的是容器的数据,etc-hosts文件中还保留着podname。

# cd /var/lib/kubelet/pods/86d60ee9-9fae-11e8-8cfc-525400290b20

/var/lib/kubelet/pods/86d60ee9-9fae-11e8-8cfc-525400290b20# ls
containers  etc-hosts  plugins  volumes

/var/lib/kubelet/pods/86d60ee9-9fae-11e8-8cfc-525400290b20# cat etc-hosts 
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
172.16.1.180	omc-test-2509590746-mw56s

解决问题

首先通过etc-hosts文件的pod name 发现已经没有相关的实例在运行了，然后按照issue中的提示，删除pod

rm -rf 86d60ee9-9fae-11e8-8cfc-525400290b20

但是这个方法有一定的危险性，还不确认是否有数据丢失的风险，如果可以确认，再执行。或在issue中寻找更好的解决方法。

再去查看日志，就会发现syslog不会再刷类似的日志了。

参考：https://cloud.tencent.com/developer/article/1385911

k8s kubelet 日志报错 There were many similar errors. Turn up verbosity to see them err=“orphaned pod
weixin_40548182的博客
08-21 382
根据日志中输出的 pod id 2a22ee1c-3914-45e7-a916-06f70ab5c8bc。由代码可知，只会删除孤立 pod 的空目录，至于目录下有文件就不会删除该目录。/var/log/syslog 中有很多如下日志存在，每隔 2 秒刷新一次。硬重启机器，使 kubelet 没有机会清理 Pod 目录，即目录变为孤立。删除 vol_data.json 文件。再进入该 pod volume 目录。进入到该 pod 数据目录下。
提示报错：Exception in thread “AWT-EventQueue-0“...如何解决？
**My Coding Family**
07-05 1632
🏆本文收录于 《全栈Bug调优(实战版)》 专栏，该专栏专注于分享我在真实项目开发中遇到的各类疑难Bug及其深层成因，并系统提供高效、可复现的解决思路和实操方案。无论你是刚入行的新手开发者，还是拥有多年项目经验的资深工程师，本专栏都将为你提供一条系统化、高质量的问题排查与优化路径，助力你加速成长，攻克技术壁垒，迈向技术价值最大化与职业发展的更高峰🚀！
kubelet报错：Orphaned pod “$Pod“ found, but volume paths are still present on disk
爱吃鱼的小明的博客
06-15 1047
1.场景： 容器一直处在 ContainerCreating状态 $ kubectl get pod -n liantiao | grep pushserver pushserver-69cc66c679-tx2pg 1/1 Running 0 98d pushserver-76675b6d94-kqh25 0/1 ContainerCreating 0 56m
Orphaned pod found - but volume paths are still present on disk的处理
任我行的博客
12-04 538
一个问题发现集群内节点非正常移除，被调度为not ready查看kubelet日志 通过id号,进入kubelet的目录,可以发现里面装的是容器的数据,etc-hosts文件中还保留着podname 通过搜索相关的信息也有相关的问题: 首先通过etc-hosts文件的pod name 发现已经没有相关的实例在运行了，然后按照issue中的提示，删除pod 但是这个方法有一定的危险性，还不确认是否有数据丢失的风险，如果可以确认，再执行。或在issue中寻找更好的解决方法。再去查看日志，就会发现syslog不会
K8S故障排查指南- but volume paths are still present on disk
qq_24794401的博客
06-29 2732
问题产生在使用 Kubernetes 时，有时会遇到 Pod 状态一直处理 Terminating。Pod 一直没有正常退出，一般情况会使用命令 kubectl delete pods ...
报错：Exception in thread “AWT-EventQueue-0“...如何解决？
**My Coding Family**
07-06 1251
🏆本文收录于 《全栈Bug调优(实战版)》 专栏，该专栏专注于分享我在真实项目开发中遇到的各类疑难Bug及其深层成因，并系统提供高效、可复现的解决思路和实操方案。无论你是刚入行的新手开发者，还是拥有多年项目经验的资深工程师，本专栏都将为你提供一条系统化、高质量的问题排查与优化路径，助力你加速成长，攻克技术壁垒，迈向技术价值最大化与职业发展的更高峰🚀！
@郭川磊 节点message中发现了以下日志 Nov 27 03:50:09 instance-tlx3n3o5-04 kubelet: E1127 03:50:09.178611 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:11 instance-tlx3n3o5-04 kubelet: E1127 03:50:11.179145 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:13 instance-tlx3n3o5-04 kubelet: E1127 03:50:13.179505 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:15 instance-tlx3n3o5-04 kubelet: E1127 03:50:15.178549 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:17 instance-tlx3n3o5-04 kubelet: E1127 03:50:17.179662 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:19 instance-tlx3n3o5-04 kubelet: E1127 03:50:19.178592 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:21 instance-tlx3n3o5-04 kubelet: E1127 03:50:21.179354 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:23 instance-tlx3n3o5-04 kubelet: E1127 03:50:23.178330 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:25 instance-tlx3n3o5-04 kubelet: E1127 03:50:25.178940 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:27 instance-tlx3n3o5-04 kubelet: E1127 03:50:27.179081 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:29 instance-tlx3n3o5-04 kubelet: E1127 03:50:29.179352 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:31 instance-tlx3n3o5-04 kubelet: E1127 03:50:31.178735 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:33 instance-tlx3n3o5-04 kubelet: E1127 03:50:33.179198 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:35 instance-tlx3n3o5-04 kubelet: E1127 03:50:35.179392 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:37 instance-tlx3n3o5-04 kubelet: E1127 03:50:37.178201 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:39 instance-tlx3n3o5-04 kubelet: E1127 03:50:39.178448 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:41 instance-tlx3n3o5-04 kubelet: E1127 03:50:41.178875 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them. Nov 27 03:50:43 instance-tlx3n3o5-04 kubelet: E1127 03:50:43.179405 123978 kubelet_volumes.go:179] orphaned pod "0cf50d7a-69c3-487c-9c66-8f04480f2a30" found, but failed to rmdir() volume at path /cce/kubelet/pods/0cf50d7a-69c3-487c-9c66-8f04480f2a30/volumes/kubernetes.io~empty-dir/engine-data: directory not empty : There were a total of 10 errors similar to this. Turn up verbosity to see them.
最新发布
12-10
根据Kubernetes的工作机制，当一个Pod被删除（或成为孤儿）时，kubelet会尝试清理该Pod使用的卷（Volume）。然而，如果卷目录非空，则删除操作会失败。 根据Kubernetes文档和常见实践，我们可以从以下几个方面来...
K8s 集群排错 volume paths are still present on disk
Gabriel的博客
11-07 454
K8s 集群排错
kubelet残留孤儿pod(Orphaned pod)无法删除
任我行的博客
04-13 2564
记录kubelet残留孤儿pod(Orphaned pod)无法删除的问题分析和解决方法。
K8S-OrphanedPod清理脚本
xyclianying的博客
09-02 1044
清理节点中孤儿pod 原因 公司一直使用的1.13版本k8s，经常会出现pod删除后，目录未清理的情况产生日志报错 孤儿pod越来越多，浪费的磁盘空间无法释放，且报警日志error数过多 解决方案 编写了以下脚本并添加crontab实现自动解决该问题。 #!/bin/bash #worker节点孤儿pod处理 >/tmp/orphanedMount.txt && >/tmp/orphanedPod.txt IFS=$'\n' #设置默认分隔符 echo `date`
rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出
weixin_30793643的博客
11-15 406
1.查看rook-agent(重要)和mysql-wordpress 的日志，如下： MountVolume.SetUp failed for volume "pvc-f002e1fe-469c-11e8-9dca-90b8d0599f2f" : mount command failed, status: Failure, reason: Rook: Error getting RPC clie...
k8s关于Orphaned pod ＜pod_id＞ found，volume paths are still present on disk 的解决方法
linux丶晨星
12-12 1748
问题描述 因k8s节点异常关机导致启动后业务Pod重新部署，关机之前的Pod状态已被删除，今天在查看日志时发现在异常关机之前的集群节点Pod是非正常移除的，一直刷报错信息；如下： 问题排查 查看系统日志/var/log/messages发现一直在刷kubectl服务的以下的报错,从错误信息可以看到，这台节点存在一个孤儿Pod,并且该Pod挂载了数据卷(volume)，阻碍了Kubelet对孤儿Pod正常的回收清理。 [root@sss-010xl-n02 ~]# tail -3 /var/log/messa
kubesphere 问题解决
怨行客
05-12 8221
镜像下载不下来，到对应节点 docker pull 重启节点后，无监控数据，因为你没有正确退出节点，这样退链接 其实就是停止调度，重启kubectl,docker
Kubernetes 遇到的问题
运维老司机
05-22 105
Kubernetes 遇到的问题 k8s 版本 k8s v1.13.5 docker 18.09.2 问题一 kubelet 报错 May 22 10:14:42 test-ops-k8s-03 kubelet: E0522 10:14:42.873650 19795 kubelet_volumes.go:154] Orphaned pod "ed306153-6faf-11e9-9d2...
Kubernetes Kubelet 线程泄漏
小楼一夜听春雨，深巷明朝卖杏花
08-19 947
继续分析历史日志，发现这个pod是 ks-controller-manager 残留的历史文件。livenessProbe/readinessProbe 健康检查最佳实践。查看日志/var/log/messages发现有孤儿pod。巡检node上大量使用线程的pod。巡检node上大量使用线程的pod。20211102 操作记录。后续巡检需要用到的命令。查询对应的pod命令。
kubernetes报错笔记 （二） kubelet
默子昂
06-18 4609
1. 想不起来是什么时候的了 orphaned pod "82646587-78c6-47e4-9509-7d4ff8debbc0" found, but volume subpaths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them. https://www.ziji.work/kubernetes/kubelet-orphane
挂载失败-日志中显示僵尸pod的问题
weixin_34345560的博客
01-23 423
当节点出现pod挂载不上，且查看节点日志(/var/log/message)发现不断有“Orphaned pod”字段的日志打印出来，如下： Jan 21 03:07:08 abc_k8s_worker01 kubelet: E0121 03:07:08.894211 6507 kubelet_volumes.go:140] Orphaned po...
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值