k8sQ&A：no space left on device

最新推荐文章于 2025-06-09 09:13:43 发布

原创最新推荐文章于 2025-06-09 09:13:43 发布 · 置顶 · 6k 阅读

2 ·

CC 4.0 BY-SA版权

大数据和云计算同时被 2 个专栏收录

18 篇文章

订阅专栏

k8s

4 篇文章

订阅专栏

本文针对Kubernetes集群中常见的三个问题进行了解析：kube-node状态显示notready的原因及其解决办法；pod启动失败提示资源不足的问题定位；以及pod调度失败的现象与解决思路。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

问题一：no space left on device

kube-node节点的状态是notready，原因是kubelet启动时报错“no space left on device:”：

218的kube-node节点的状态是notready，分配和申请资源OK，但是服务启动有问题：启动失败

分析：218上kubelt节点启动7s后失败，原因是： no space left on device:
journalctl -u kubelet
:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice/run-29422.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422422.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on device
:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no spaceify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on device
t.go:1339] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on devicey/system.slice/run-29422.scope: no space left on device

ode=exited, status=255/n/a

218上内存和存储资源如下：

[root@hps218 system.slice]# free -m
total used free shared buff/cache available
Mem: 4001 1624 721 222 1655 1477
Swap: 4351 143 4208

[root@hps218 system.slice]# df -h

Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 55M 2.0G 3% /dev/shm
tmpfs 2.0G 220M 1.8G 11% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root 99G 17G 78G 18% /
/dev/mapper/centos-data 296G 3.8G 277G 2% /data
/dev/mapper/centos-home 44G 169M 42G 1% /home
/dev/sda1 477M 147M 301M 33% /boot
tmpfs 2.0G 12K 2.0G 1% /data/ips/app/work/node/kubelet/pods/6204604d-6b7b-11e8-958a-000c295deb47/volumes/kubernetes.io~secret/default-token-6xmh2
tmpfs 2.0G 12K 2.0G 1% /data/ips/app/work/node/kubelet/pods/63725613-6b7b-11e8-958a-000c295deb47/volumes/kubernetes.io~secret/heapster-token-5slgw

218上查看kubelet的日志：都是资源充足：

kubectl describe nodes 10.20.16.218

Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=10.20.16.218
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp: Sat, 09 Jun 2018 08:18:16 +0800
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 11 Jun 2018 11:35:40 +0800 Sat, 09 Jun 2018 08:18:16 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:06:03 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.20.16.218
Hostname: 10.20.16.218
Capacity:
cpu: 2
ephemeral-storage: 309504832Ki
hugepages-2Mi: 0
memory: 4097940Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 285239652699
hugepages-2Mi: 0
memory: 3995540Ki
pods: 110
System Info:
Machine ID: edb038c4975745dbaf2abf3c7fb54d32
System UUID: 564DFD41-7F6F-DC56-E6ED-43B11018119C
Boot ID: 2384972f-22a8-4979-9252-b7cebea31a43
Kernel Version: 4.4.47-1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.12.6
Kubelet Version: v1.10.3-beta.0.10+18f4924fc90ffc
Kube-Proxy Version: v1.10.3-beta.0.10+18f4924fc90ffc
PodCIDR: 172.30.0.0/24
ExternalID: 10.20.16.218
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system docker-health-gjrh6 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system heapster-fzpp5 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
0 (0%) 0 (0%) 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure

Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.

【定位】"/sys/fs/cgroup/memory/system.slice/run-29422.scope""/sys/fs/cgroup/memory/system.slice/目录下run-*.scope"文件过多导致的。

解决方法：

sudo sysctl fs.inotify.max_user_watches=1048576

之后

[ips@hps219 bin]$ ./kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.20.16.218 Ready <none> 2d v1.10.3-beta.0.10+18f4924fc90ffc
10.20.16.219 Ready <none> 1h v1.10.3-beta.0.10+18f4924fc90ffc

问题二：问题描述及现象：主机资源充足，但是pod启动失败，报错资源不够，最终是node资源不够

问题描述：应用app1，未设置分区亲和性，运行中，添加一个分区亲和性：测试域-测试2区(值25主机满足要求)后，13min后，新的pod仍然未被拉起;

环境：http://10.19.10.26:8080

版本：mysql换成oracle的版本

问题详情：

操作步骤：

1、app1应用，新增一个分区亲和性，亲和性选择

3个node节点的cpu资源是充足的，请见截图，10.19.10.25 标签是test2

kubectl get pod/deploy-app1-5fbf695d5d-cmf4m -n ns-business -o yaml

查看pod未被拉起的原因是：

message: '0/3 nodes are available: 2 Insufficient cpu, 2 node(s) didn''t match

node selector.'

重启应用后，报错是：

'0/3 nodes are available: 1 Insufficient cpu, 2 node(s) didn''t match

问题定位：

最终结论：环境问题，node节点中的资源不足。（因为一套K8s环境复用）。

首先要查看下node的资源。

定位过程：

监控中主机监控的cpu使用率跟node节点中的资源不一样。

查看主机cpu、namespace（使用只有10%）的cpu都是ok的；

查看一个node主机上的资源：

kubectl describe node 10.19.10.25

查看一台node主机上的所有pod资源（跨namespace的）：

kubectl get pods -A -o wide |grep 10.19.10.25

删除资源：例如

kubectl delete svc/svc-app0619-canary -n ns-yancheng

kubectl delete deploy/deploy-app0619-canary -n ns-yancheng

删除结束后，

kubectl describe node 10.19.10.25 查看资源已经降低了很多，到50%以下了：

重启服务后，服务启动成功，pod被调度到分区对应的主机上。

问题三：调度失败：查看原因：

原因是：部署环境时，master节点自动会被打污点，避免被调度到该节点上。

问题现象：配置了亲和性，但是未被调度到对应的node节点：

定位过程：

kubectl get pod/deploy-app1-5f795cc4cb-hmvzr -n ns-jiangsu -o yaml有爆粗信息：

message: '0/2 nodes are available: 1 node(s) didn''t match node selector, 1 node(s) had taints that the pod didn''t tolerate.'

kubectl get pod/deploy-app1-5f795cc4cb-hmvzr -n ns-jiangsu -o yaml

tolerations:

- effect: NoExecute

key: node.kubernetes.io/not-ready

operator: Exists

tolerationSeconds: 300

- effect: NoExecute

key: node.kubernetes.io/unreachable

operator: Exists

tolerationSeconds: 300

Node35里有污点信息：

kubectl get node 10.15.49.35 -o yaml

taints:

- effect: NoSchedule

key: node-role.kubernetes.io/master

[foot@h-w89jpoji ~]$ kubectl get nodes --show-labels

NAME STATUS ROLES AGE VERSION LABELS

10.15.49.34 Ready <none> 5d16h v1.15.6-dirty XXX=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,footdomain=test,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.15.49.34,kubernetes.io/os=linux

10.15.49.35 Ready master 5d16h v1.15.6-dirty beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.15.49.35,kubernetes.io/os=linux,node-role.kubernetes.io/master=,tag1=11,testzone=test1

节点亲和性是pod的一种属性，它使 Pod 被吸引到一类特定的节点。这可能出于一种偏好，也可能是硬性要求。

Taint（污点）则相反，它使节点能够排斥一类特定的 Pod。

容忍度（Tolerations）是应用于 Pod 上的，允许（但并不要求）Pod 调度到带有与之匹配的污点的节点上。

污点和容忍度（Toleration）相互配合，可以用来避免 Pod 被分配到不合适的节点上。每个节点上都可以应用一个或多个污点，这表示对于那些不能容忍这些污点的 Pod，是不会被该节点接受的。