深入分析Kubernetes Critical Pod（二）-优快云博客

深入分析Kubernetes Critical Pod（一）介绍了Scheduler对Critical Pod的处理逻辑，下面我们再看下Kubelet Eviction Manager对Critical Pod的处理逻辑是怎样的，以便我们了解Kubelet Evict Pod时对Critical Pod是否有保护措施，如果有，又是如何保护的。

Kubelet Eviction Manager Admit

kubelet在syncLoop中每个1s会循环调用syncLoopIteration，从config change channel | pleg channel | sync channel | houseKeeping channel | liveness manager's update channel中获取event，然后分别调用对应的event handler进行处理。

configCh: dispatch the pods for the config change to the appropriate handler callback for the event type
plegCh: update the runtime cache; sync pod
syncCh: sync all pods waiting for sync
houseKeepingCh: trigger cleanup of pods
liveness manager's update channel: sync pods that have failed or in which one or more containers have failed liveness checks

特别提一下，houseKeeping channel是每隔houseKeeping（10s）时间就会有event，然后执行HandlePodCleanups，执行以下清理操作：

Stop the workers for no-longer existing pods.（每个pod对应会有一个worker，也就是goruntine）
killing unwanted pods
removes the volumes of pods that should not be running and that have no containers running.
Remove any orphaned mirror pods.
Remove any cgroups in the hierarchy for pods that are no longer running.

pkg/kubelet/kubelet.go:1753

func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
    syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
    select {
    case u, open := <-configCh:
        
        if !open {
            glog.Errorf("Update channel is closed. Exiting the sync loop.")
            return false
        }

        switch u.Op {
        case kubetypes.ADD:
            
            handler.HandlePodAdditions(u.Pods)
        ...
        case kubetypes.RESTORE:
            glog.V(2).Infof("SyncLoop (RESTORE, %q): %q", u.Source, format.Pods(u.Pods))
            // These are pods restored from the checkpoint. Treat them as new
            // pods.
            handler.HandlePodAdditions(u.Pods)
        ...
        }

        if u.Op != kubetypes.RESTORE {
            ...
        }
    case e := <-plegCh:
        ...
    case <-syncCh:
        ...
    case update := <-kl.livenessManager.Updates():
        ...
    case <-housekeepingCh:
        ...
    }
    return true
}

syncLoopIteration中定义了当kubelet配置变更重启后的逻辑：kubelet会对正在running的Pods进行Admission处理，Admission的结果有可能会让该Pod被本节点拒绝。

HandlePodAdditions就是用来处理Kubelet ConficCh中的event的Handler。

// HandlePodAdditions is the callback in SyncHandler for pods being added from a config source.
func (kl *Kubelet) HandlePodAdditions(pods []*v1.Pod) {
    start := kl.clock.Now()
    sort.Sort(sliceutils.PodsByCreationTime(pods))
    for _, pod := range pods {
        ...

        if !kl.podIsTerminated(pod) {
            ...
            // Check if we can admit the pod; if not, reject it.
            if ok, reason, message := kl.canAdmitPod(activePods, pod); !ok {
                kl.rejectPod(pod, reason, message)
                continue
            }
        }
        ...
    }
}

如果该Pod Status不是属于Terminated，就调用canAdmitPod对该Pod进行准入检查。如果准入检查结果表示该Pod被拒绝，那么就会将该Pod Phase设置为Failed。

pkg/kubelet/kubelet.go:1643

func (kl *Kubelet) canAdmitPod(pods []*v1.Pod, pod *v1.Pod) (bool, string, string) {
    // the kubelet will invoke each pod admit handler in sequence
    // if any handler rejects, the pod is rejected.
    // TODO: move out of disk check into a pod admitter
    // TODO: out of resource eviction should have a pod admitter call-out
    attrs := &lifecycle.PodAdmitAttributes{Pod: pod, OtherPods: pods}
    for _, podAdmitHandler := range kl.admitHandlers {
        if result := podAdmitHandler.Admit(attrs); !result.Admit {
            return false, result.Reason, result.Message
        }
    }

    return true, "", ""
}

canAdmitPod就会调用kubelet启动时注册的一系列admitHandlers对该Pod进行准入检查，其中就包括kubelet eviction manager对应的admitHandle。

pkg/kubelet/eviction/eviction_manager.go:123

// Admit rejects a pod if its not safe to admit for node stability.
func (m *managerImpl) Admit(attrs *lifecycle.PodAdmitAttributes) lifecycle.PodAdmitResult {
    m.RLock()
    defer m.RUnlock()
    if len(m.nodeConditions) == 0 {
        return lifecycle.PodAdmitResult{Admit: true}
    }
    
    if utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) && kubelettypes.IsCriticalPod(attrs.Pod) {
        return lifecycle.PodAdmitResult{Admit: true}
    }

    if hasNodeCondition(m.nodeConditions, v1.NodeMemoryPressure) {
        notBestEffort := v1.PodQOSBestEffort != v1qos.GetPodQOS(attrs.Pod)
        if notBestEffort {
            return lifecycle.PodAdmitResult{Admit: true}
        }
    }

        return lifecycle.PodAdmitResult{
        Admit:   false,
        Reason:  reason,
        Message: fmt.Sprintf(message, m.nodeConditions),
    }
}

eviction manager的Admit的逻辑如下：

如果该node的Conditions为空，则Admit成功；
如果enable了ExperimentalCriticalPodAnnotation Feature Gate，并且该Pod是Critical Pod（Pod有Critical的Annotation，或者Pod的优先级不小于SystemCriticalPriority），则Admit成功；
- SystemCriticalPriority的值为2 billion。
如果该node的Condition为Memory Pressure，并且Pod QoS为非best-effort，则Admit成功；
其他情况都表示Admit失败，即不允许该Pod在该node上Running。

Kubelet Eviction Manager SyncLoop

另外，在kubelet eviction manager的syncLoop中，也会对Critical Pod有特殊处理，代码如下。

pkg/kubelet/eviction/eviction_manager.go:226

// synchronize is the main control loop that enforces eviction thresholds.
// Returns the pod that was killed, or nil if no pod was killed.
func (m *managerImpl) synchronize(diskInfoProvider DiskInfoProvider, podFunc ActivePodsFunc) []*v1.Pod {
    ...

    // we kill at most a single pod during each eviction interval
    for i := range activePods {
        pod := activePods[i]
        
        if utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) &&
            kubelettypes.IsCriticalPod(pod) && kubepod.IsStaticPod(pod) {
            continue
        }
        ...
        return []*v1.Pod{pod}
    }
    glog.Infof("eviction manager: unable to evict any pods from the node")
    return nil
}

当触发了kubelet evict pod时，如果该pod满足以下所有条件时，将不会被kubelet eviction manager kill掉。

该Pod Status不是Terminated；
Enable ExperimentalCriticalPodAnnotation Feature Gate；
该Pod是Critical Pod；
该Pod时Static Pod；

总结

经过上面的分析，我们得到以下Kubelet Eviction Manager对Critical Pod处理的关键点：

kubelet重启后，eviction manager的Admit流程中对Critical Pod做如下特殊处理：如果enable了ExperimentalCriticalPodAnnotation Feature Gate，则允许该Critical Pod准入该node，无视该node的Condition。
当触发了kubelet evict pod时，如果该Critical Pod满足以下所有条件时，将不会被kubelet eviction manager kill掉。
- 该Pod Status不是Terminated；
- Enable ExperimentalCriticalPodAnnotation Feature Gate；
- 该Pod是Static Pod；