深入分析Kubernetes Critical Pod(三)

本文深入分析Kubernetes中CriticalPod在资源不足时如何触发资源抢占机制,及Kubelet如何通过PredicateAdmit流程处理CriticalPod的准入检查。

阅读全文请点击

本文介绍了Kubelet在Predicate Admit准入检查时对CriticalPod的资源抢占的原理,以及Priority Admission Controller对CriticalPod的PriorityClassName特殊处理。

深入分析Kubernetes Critical Pod系列:
深入分析Kubernetes Critical Pod(一)
深入分析Kubernetes Critical Pod(二)
深入分析Kubernetes Critical Pod(三)
深入分析Kubernetes Critical Pod(四)

Kubelet Predicate Admit时对Critical的资源抢占处理

kubelet 在Predicate Admit流程中,会对Pods进行各种Predicate准入检查,包括GeneralPredicates检查本节点是否有足够的cpu,mem,gpu资源。如果GeneralPredicates准入检测失败,对于nonCriticalPod则直接Admit失败,但如果是CriticalPod则会触发kubelet preemption进行资源抢占,按照一定规则杀死一些Pods释放资源,抢占成功,则Admit成功。

流程的源头应该从kubelet初始化的流程开始。

pkg/kubelet/kubelet.go:315

// NewMainKubelet instantiates a new Kubelet object along with all the required internal modules.
// No initialization of Kubelet and its modules should happen here.
func NewMainKubelet(...) (*Kubelet, error) {
    ...
   criticalPodAdmissionHandler := preemption.NewCriticalPodAdmissionHandler(klet.GetActivePods, killPodNow(klet.podWorkers, kubeDeps.Recorder), kubeDeps.Recorder)
    klet.admitHandlers.AddPodAdmitHandler(lifecycle.NewPredicateAdmitHandler(klet.getNodeAnyWay, criticalPodAdmissionHandler, klet.containerManager.UpdatePluginResources))
    // apply functional Option's
    for _, opt := range kubeDeps.Options {
        opt(klet)
    }

    ...
    return klet, nil
}

在NewMainKubelet对kubelet进行初始化时,通过AddPodAdmitHandler注册了criticalPodAdmissionHandler,CriticalPod的Admit的特殊之处就体现在criticalPodAdmissionHandler。

然后,我们进入kubelet的predicateAdmitHandler流程中,看看GeneralPredicates失败后的处理逻辑。

pkg/kubelet/lifecycle/predicate.go:58

func (w *predicateAdmitHandler) Admit(attrs *PodAdmitAttributes) PodAdmitResult {
    ...

    fit, reasons, err := predicates.GeneralPredicates(podWithoutMissingExtendedResources, nil, nodeInfo)
    if err != nil {
        message := fmt.Sprintf("GeneralPredicates failed due to %v, which is unexpected.", err)
        glog.Warningf("Failed to admit pod %v - %s", format.Pod(pod), message)
        return PodAdmitResult{
            Admit:   fit,
            Reason:  "UnexpectedAdmissionError",
            Message: message,
        }
    }
    if !fit {
        fit, reasons, err = w.admissionFailureHandler.HandleAdmissionFailure(pod, reasons)
        if err != nil {
            message := fmt.Sprintf("Unexpected error while attempting to recover from admission failure: %v", err)
            glog.Warningf("Failed to admit pod %v - %s", format.Pod(pod), message)
            return PodAdmitResult{
                Admit:   fit,
                Reason:  "UnexpectedAdmissionError",
                Message: message,
            }
        }
    }
    ...
    return PodAdmitResult{
        Admit: true,
    }
}

转载于:https://www.cnblogs.com/ls123/p/9253839.html

systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since 2025-06-25 10:01:04 CST; 35s ago Docs: https://kubernetes.io/docs/ Main PID: 22211 (kubelet) Tasks: 47 Memory: 122.3M CGroup: /system.slice/kubelet.service └─22211 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod... 6月 25 10:01:28 qr36 kubelet[22211]: E0625 10:01:28.204025 22211 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/kube-proxy-m9mbs" 6月 25 10:01:28 qr36 kubelet[22211]: E0625 10:01:28.204036 22211 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="monitoring/node-exporter-qlpds" 6月 25 10:01:28 qr36 kubelet[22211]: I0625 10:01:28.204045 22211 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node" 6月 25 10:01:38 qr36 kubelet[22211]: I0625 10:01:38.521071 22211 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="memory" 6月 25 10:01:38 qr36 kubelet[22211]: I0625 10:01:38.521113 22211 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="memory" 6月 25 10:01:38 qr36 kubelet[22211]: I0625 10:01:38.521144 22211 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/calico-node-s4p2t kube-system/kube...xporter-qlpds] 6月 25 10:01:38 qr36 kubelet[22211]: E0625 10:01:38.521180 22211 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/calico-node-s4p2t" 6月 25 10:01:38 qr36 kubelet[22211]: E0625 10:01:38.521192 22211 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/kube-proxy-m9mbs" 6月 25 10:01:38 qr36 kubelet[22211]: E0625 10:01:38.521205 22211 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="monitoring/node-exporter-qlpds" 6月 25 10:01:38 qr36 kubelet[22211]: I0625 10:01:38.521215 22211 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node" Hint: Some lines were ellipsized, use -l to show in full. [root@qr36 ~]$>#free -m total used free shared buff/cache available Mem: 64196 27382 336 3251 36477 32916 Swap: 0 0 0
06-26
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符  | 博主筛选后可见
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值