k8s nodeSelector&affinity

最新推荐文章于 2025-11-24 11:44:14 发布

原创最新推荐文章于 2025-11-24 11:44:14 发布 · 1w 阅读

2 ·

CC 4.0 BY-SA版权

k8s 专栏收录该内容

28 篇文章

订阅专栏

本文介绍了Kubernetes中Pod调度到特定Node的两种方法：nodeSelector与affinity。详细解析了NodeAffinity、PodAffinity及PodAntiAffinity的配置方式，并通过示例展示了如何利用这些特性精确控制Pod的部署位置。

1.分配pod到node的方法
通过node label selector实现约束pod运行到指定节点,有两种方法 nodeSelector 以及affinity
2.nodeSelector 是k8s早起提供的节点选择器实现
1）首先为nodes打对应的label
kubectl label nodes master disktype=ssd
2)创建yaml文件

cat >> nginx-pod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
EOF

创建pod
kubectl create -f nginx-pod.yaml
节点默认的label
kubectl get nodes -o yaml 可以看到默认节点的label,也可以使用这些label进行约束

alpha.kubernetes.io/fluentd-ds-ready: "true"
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
disktype: ssd
kubeadm.alpha.kubernetes.io/role: master
kubernetes.io/hostname: master

3, affinity
根据限制方式有两种affinity：
requiredDuringSchedulingIgnoredDuringExecution （硬限制）
条件必须满足从而调度到一个节点
preferredDuringSchedulingIgnoredDuringExecution （软限制）
条件不满足则调度到不满足条件的节点

IgnoredDuringExecution意味着当节点label改变后，pod将继续运行在这个不满足labelselector的node上面，将来会提供RequiredDuringExecution，让node驱逐不满足affinity条件的pod

根据类型有

NodeAffinity
PodAffinity
PodAntiAffinity

NodeAffinity

下面这个例子将运行pod到kubernetes.io/e2e-az-name值为e2e-az1 或e2e-az2的节点上面

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
  annotations:
    scheduler.alpha.kubernetes.io/affinity: >
      {
        "nodeAffinity": {
          "requiredDuringSchedulingIgnoredDuringExecution": {
            "nodeSelectorTerms": [
              {
                "matchExpressions": [
                  {
                    "key": "kubernetes.io/e2e-az-name",
                    "operator": "In",
                    "values": ["e2e-az1", "e2e-az2"]
                  }
                ]
              }
            ]
          }
        }
      }
    another-annotation-key: another-annotation-value
spec:
  containers:
  - name: with-node-affinity
    image: gcr.io/google_containers/pause:2.0

我们可以看到NodeSelectorTerms可以有多个，之间是或的关系，满足任意一个既满足，，MatchExpressions也可以有多个，他们之间是且的关系必须都满足

preferredDuringSchedulingIgnoredDuringExecution 例子：
apiVersion: v1
kind: Pod
metadata:
  name: with-labels
  annotations:
    scheduler.alpha.kubernetes.io/affinity: >
    {
       "nodeAffinity": {
          "preferredDuringSchedulingIgnoredDuringExecution": [
                {
                    "weight" : 1,
                    "preference": {
                        "matchExpressions": [
                            {
                                "key": "disktype",
                                "operator": "In",
                                "values": ["ssd"]
                            }
                        ]
                    }
                }
            ]
        }
    }
    another-annotation-key: another-annotation-value
spec:
  containers:
  - name: test-affinity
    env:
      - name: MYSQL_ROOT_PASSWORD
        value: pass
    image: mysql

preferredDuringSchedulingIgnoredDuringExecution值为列表，根据权重决定顺序
MatchExpressions 值为列表关系为且，必须都满足

Inter-pod affinity and anti-affinity
Inter-pod affinity 是根据通过已运行在节点上的pod的标签而不是node的标签来决定被调度pod的运行节点，因为pod运行在指定的namespace所以需要自己指定运行pod的namesapce
anti-affinity 和Inter-pod affinity相反

这个例子意思是首先node需要有 failure-domain.beta.kubernetes.io/zone label 不能有kubernetes.io/hostname label 切改节点上需要有一个pod 有一个security=S1的label

apiVersion: v1
kind: Pod
metadata:
  name: with-pod-affinity
  annotations:
    scheduler.alpha.kubernetes.io/affinity: >
        {
          "podAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": [
              {
                "labelSelector": {
                  "matchExpressions": [
                    {
                      "key": "security",
                      "operator": "In",
                      "values": ["S1"]
                    }
                  ]
                },
                "topologyKey": "failure-domain.beta.kubernetes.io/zone"
             }
            ]
           },
          "podAntiAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": [
              {
                "labelSelector": {
                  "matchExpressions": [
                    {
                      "key": "security",
                      "operator": "In",
                      "values": ["S2"]
                    }
                  ]
                },
                "topologyKey": "kubernetes.io/hostname"
             }
            ]
           }
         }
spec:
  containers:
  - name: with-pod-affinity
    image: gcr.io/google_containers/pause:2.0

具体字段的定义以及关系可以参考k8s源码的定义

type Affinity struct {
    NodeAffinity *NodeAffinity
    PodAffinity *PodAffinity
    PodAntiAffinity *PodAntiAffinity
}
type NodeAffinity struct {
    RequiredDuringSchedulingIgnoredDuringExecution *NodeSelector
    PreferredDuringSchedulingIgnoredDuringExecution []PreferredSchedulingTerm
}

type PodAffinity struct {
    RequiredDuringSchedulingIgnoredDuringExecution []PodAffinityTerm
    PreferredDuringSchedulingIgnoredDuringExecution []WeightedPodAffinityTerm
}

type PodAntiAffinity struct {
    RequiredDuringSchedulingIgnoredDuringExecution []PodAffinityTerm
    PreferredDuringSchedulingIgnoredDuringExecution []WeightedPodAffinityTerm
}

type PreferredSchedulingTerm struct {
    Weight int32
    Preference NodeSelectorTerm
}

type NodeSelector struct {
    NodeSelectorTerms []NodeSelectorTerm
}
type NodeSelectorTerm struct {
    MatchExpressions []NodeSelectorRequirement
}
type NodeSelectorRequirement struct {
    Key string
    Operator NodeSelectorOperator
    Values []string
}

const (
    NodeSelectorOpIn           NodeSelectorOperator = "In"
    NodeSelectorOpNotIn        NodeSelectorOperator = "NotIn"
    NodeSelectorOpExists       NodeSelectorOperator = "Exists"
    NodeSelectorOpDoesNotExist NodeSelectorOperator = "DoesNotExist"
    NodeSelectorOpGt           NodeSelectorOperator = "Gt"
    NodeSelectorOpLt           NodeSelectorOperator = "Lt"
)

type PodAffinityTerm struct {
    LabelSelector *metav1.LabelSelector
    Namespaces []string
    TopologyKey string
}

type LabelSelector struct {
    MatchLabels map[string]string 
    MatchExpressions []LabelSelectorRequirement 
}


type LabelSelectorRequirement struct {
    Key string 
    Operator LabelSelectorOperator 
    Values []string 
}

const (
    LabelSelectorOpIn           LabelSelectorOperator = "In"
    LabelSelectorOpNotIn        LabelSelectorOperator = "NotIn"
    LabelSelectorOpExists       LabelSelectorOperator = "Exists"
    LabelSelectorOpDoesNotExist LabelSelectorOperator = "DoesNotExist"
)

type WeightedPodAffinityTerm struct {
    Weight int32
    PodAffinityTerm PodAffinityTerm
}

type PodAffinityTerm struct {
    LabelSelector *metav1.LabelSelector
    Namespaces []string
    TopologyKey string
}