k8s部署metallb

最新推荐文章于 2025-05-04 20:07:05 发布

原创

最新推荐文章于 2025-05-04 20:07:05 发布 · 1.7k 阅读

9 ·

CC 4.0 BY-SA版权

文章标签：

#kubernetes #容器 #云原生

一、介绍

k8s的service资源分为cluster IP类型、nodeport类型、loadbalancer类型、externalname类型

其中最常用的k8s默认的是cluster IP类型，此类型只能在k8s集群和集群之间的服务器之间访问。

而nodeport 类型是K8s将内部服务对外暴露的基础，service资源默认会产生一个IP地址，此IP地址只能在k8s集群内访问，如何让集群外访问service资源，通过对nodeport 暴露，

上面我们提到，将K8s内部的服务通过NodePort方式暴露出去，K8s会在每个Worker节点上都开启对应的NodePort端口。逻辑上看，K8s集群中的所有节点都会暴露这个服务，或者说这个服务是以集群方式暴露的(实际支持这个服务的Pod可能就分布在其中有限几个节点上，但是因为所有节点上都有Kube-Proxy，所以所有节点都知道该如何转发)。既然是集群，就会涉及负载均衡问题，谁负责对这个服务的负载均衡访问？答案是需要引入负载均衡器(Load Balancer)。下图是通过LoadBalancer，将服务对外暴露的概念模型

LoadBalancer底层有赖于NodePort

Kubernetes没有为裸机集群提供LoadBalance，私有云架构的kubernetes集群不支持LoadBalance，在自己的集群没有这个情况所以会出现以下的情况，为解决使用LoadBalance我们引入了metallb。

如果没有部署mtallb，直接 service 资源引用LoadBalance，就出现我安装的service在请求EXTERNAL-IP 时候一直处于pending

kubectl get service
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP          7d20h
my-service   LoadBalancer   10.104.156.116   <pending>     8080:32586/TCP   12m

二、准备工作

如果在 IPVS 模式下使用 kube-proxy，则从 Kubernetes v1.14.2 开始，您必须启用严格 ARP 模式。Kubernetes v1.14.2 以下的版本不支持IPVS调度，所以默认都是使用iptables，这样的话就不需要更改 kube-proxy 的 configmap。如果使用Kubernetes v1.14.2以上的版本，默认还是iptables，如果你不想更改为IPVS调度，也不需要更改。但是如果你想改成IPVS调度，就需要如下的操作。
请注意，如果使用 kube-router 作为服务代理，则不需要此选项，因为它默认启用严格 ARP。
可以通过编辑当前集群中的 kube-proxy 配置来实现此目的：

如何确认自己使用是iptables还是IPVS的调度算法：

kubectl get configmap kube-proxy -n kube-system -oyaml

mode: "" 或者 mode: "iptables" 则是iptables

更改为IPVS调度算法

kubectl edit configmap -n kube-system kube-proxy
并设置：
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"      // 注意，这里修改
ipvs:
  strictARP: true  // 注意，这里也修改，因为通信的底层是通过mac地址进行区分的，这就的意思代表开启

建议：建议在k8s在v1.14.2 以上的建议都启动IPVS，并且都使用metallb的高版本，比如我的环境是v1.21.1

[root@master mnt]# kubectl get node
NAME      STATUS   ROLES                  AGE    VERSION
master    Ready    control-plane,master   511d   v1.21.1
slave01   Ready    <none>                 511d   v1.21.1
slave02   Ready    <none>                 511d   v1.21.1

我使用 iptables模式，metallb低版本，可能会出现问题，最简单的是部署的时候会有如下的提示

DaemonSet、Deployment、StatefulSet 和 ReplicaSet 在 v1.16 中将不再从 extensions/v1beta1、apps/v1beta1 或 apps/v1beta2 提供服务

解决方法是：

将yml配置文件内的api接口修改为 apps/v1 ，导致原因为之间使用的kubernetes 版本是1.14.x版本，1.16.x 版本放弃部分API支持

而且可能会存在 service 资源引用LoadBalance，就出现我安装的service在请求EXTERNAL-IP 时候一直处于pending。

三、部署metallb负载均衡器

1、修改kbe-proxy

kubectl edit configmap -n kube-system kube-proxy
并设置：
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"      // 注意，这里修改
ipvs:
  strictARP: true  // 注意，这里也修改，因为通信的底层是通过mac地址进行区分的，这就的意思代表开启

修改后需要重新启动 kube-proxy 的 pod，可以用过滚动升级也可以直接 delete pod

kubectl rollout restart daemonset kube-proxy -n kube-syste

2、下载metallb的配置

wget https://raw.githubusercontent.com/google/metallb/v0.13.9/config/manifests/metallb-native.yaml

kubectl apply -f metallb-native.yaml


注意：如果你使用的是较新的 MetalLB 版本（特别是从 v0.10.0 版本开始），则需要配置 IPAddressPool 和 L2Advertisement 资源，而不是通过 ConfigMap 来配置地址池。MetalLB 的新版本采用了 CRD（Custom Resource Definitions）来配置。
这里直接使用了低版本的v0.7.3

wget https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml

内容配置

apiVersion: v1
kind: Namespace
metadata:
  labels:
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
  name: metallb-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.1
  name: addresspools.metallb.io
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tDQpNSUlGWlRDQ0EwMmdBd0lCQWdJVU5GRW1XcTM3MVpKdGkrMmlSQzk1WmpBV1MxZ3dEUVlKS29aSWh2Y05BUUVMDQpCUUF3UWpFTE1Ba0dBMVVFQmhNQ1dGZ3hGVEFUQmdOVkJBY01ERVJsWm1GMWJIUWdRMmwwZVRFY01Cb0dBMVVFDQpDZ3dUUkdWbVlYVnNkQ0JEYjIxd1lXNTVJRXgwWkRBZUZ3MHlNakEzTVRrd09UTXlNek5hRncweU1qQTRNVGd3DQpPVE15TXpOYU1FSXhDekFKQmdOVkJBWVRBbGhZTVJVd0V3WURWUVFIREF4RVpXWmhkV3gwSUVOcGRIa3hIREFhDQpCZ05WQkFvTUUwUmxabUYxYkhRZ1EyOXRjR0Z1ZVNCTWRHUXdnZ0lpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElDDQpEd0F3Z2dJS0FvSUNBUUNxVFpxMWZRcC9vYkdlenhES0o3OVB3Ny94azJwellualNzMlkzb1ZYSm5sRmM4YjVlDQpma2ZZQnY2bndscW1keW5PL2phWFBaQmRQSS82aFdOUDBkdVhadEtWU0NCUUpyZzEyOGNXb3F0MGNTN3pLb1VpDQpvcU1tQ0QvRXVBeFFNZjhRZDF2c1gvVllkZ0poVTZBRXJLZEpIaXpFOUJtUkNkTDBGMW1OVW55Rk82UnRtWFZUDQpidkxsTDVYeTc2R0FaQVBLOFB4aVlDa0NtbDdxN0VnTWNiOXlLWldCYmlxQ3VkTXE5TGJLNmdKNzF6YkZnSXV4DQo1L1pXK2JraTB2RlplWk9ZODUxb1psckFUNzJvMDI4NHNTWW9uN0pHZVZkY3NoUnh5R1VpSFpSTzdkaXZVTDVTDQpmM2JmSDFYbWY1ZDQzT0NWTWRuUUV2NWVaOG8zeWVLa3ZrbkZQUGVJMU9BbjdGbDlFRVNNR2dhOGFaSG1URSttDQpsLzlMSmdDYjBnQmtPT0M0WnV4bWh2aERKV1EzWnJCS3pMQlNUZXN0NWlLNVlwcXRWVVk2THRyRW9FelVTK1lsDQpwWndXY2VQWHlHeHM5ZURsR3lNVmQraW15Y3NTU1UvVno2Mmx6MnZCS21NTXBkYldDQWhud0RsRTVqU2dyMjRRDQp0eGNXLys2N3d5KzhuQlI3UXdqVTFITndVRjBzeERWdEwrZ1NHVERnSEVZSlhZelYvT05zMy94TkpoVFNPSkxNDQpoeXNVdyttaGdackdhbUdXcHVIVU1DUitvTWJzMTc1UkcrQjJnUFFHVytPTjJnUTRyOXN2b0ZBNHBBQm8xd1dLDQpRYjRhY3pmeVVscElBOVFoSmFsZEY3S3dPSHVlV3gwRUNrNXg0T2tvVDBvWVp0dzFiR0JjRGtaSmF3SURBUUFCDQpvMU13VVRBZEJnTlZIUTRFRmdRVW90UlNIUm9IWTEyRFZ4R0NCdEhpb1g2ZmVFQXdId1lEVlIwakJCZ3dGb0FVDQpvdFJTSFJvSFkxMkRWeEdDQnRIaW9YNmZlRUF3RHdZRFZSMFRBUUgvQkFVd0F3RUIvekFOQmdrcWhraUc5dzBCDQpBUXNGQUFPQ0FnRUFSbkpsWWRjMTFHd0VxWnh6RDF2R3BDR2pDN2VWTlQ3aVY1d3IybXlybHdPYi9aUWFEa0xYDQpvVStaOVVXT1VlSXJTdzUydDdmQUpvVVAwSm5iYkMveVIrU1lqUGhvUXNiVHduOTc2ZldBWTduM3FMOXhCd1Y0DQphek41OXNjeUp0dlhMeUtOL2N5ak1ReDRLajBIMFg0bWJ6bzVZNUtzWWtYVU0vOEFPdWZMcEd0S1NGVGgrSEFDDQpab1Q5YnZHS25adnNHd0tYZFF0Wnh0akhaUjVqK3U3ZGtQOTJBT051RFNabS8rWVV4b2tBK09JbzdSR3BwSHNXDQo1ZTdNY0FTVXRtb1FORXd6dVFoVkJaRWQ1OGtKYjUrV0VWbGNzanlXNnRTbzErZ25tTWNqR1BsMWgxR2hVbjV4DQpFY0lWRnBIWXM5YWo1NmpBSjk1MVQvZjhMaWxmTlVnanBLQ0c1bnl0SUt3emxhOHNtdGlPdm1UNEpYbXBwSkI2DQo4bmdHRVluVjUrUTYwWFJ2OEhSSGp1VG9CRHVhaERrVDA2R1JGODU1d09FR2V4bkZpMXZYWUxLVllWb1V2MXRKDQo4dVdUR1pwNllDSVJldlBqbzg5ZytWTlJSaVFYUThJd0dybXE5c0RoVTlqTjA0SjdVL1RvRDFpNHE3VnlsRUc5DQorV1VGNkNLaEdBeTJIaEhwVncyTGFoOS9lUzdZMUZ1YURrWmhPZG1laG1BOCtqdHNZamJadnR5Mm1SWlF0UUZzDQpUU1VUUjREbUR2bVVPRVRmeStpRHdzK2RkWXVNTnJGeVVYV2dkMnpBQU4ydVl1UHFGY2pRcFNPODFzVTJTU3R3DQoxVzAyeUtYOGJEYmZFdjBzbUh3UzliQnFlSGo5NEM1Mjg0YXpsdTBmaUdpTm1OUEM4ckJLRmhBPQ0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
        service:
          name: webhook-service
          namespace: metallb-system
          path: /convert
      conversionReviewVersions:
      - v1alpha1
      - v1beta1
  group: metallb.io
  names:
    kind: AddressPool
    listKind: AddressPoolList
    plural: addresspools
    singular: addresspool
  scope: Namespaced
  versions:
  - deprecated: true
    deprecationWarning: metallb.io v1alpha1 AddressPool is deprecated
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: AddressPool is the Schema for the addresspools API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: AddressPoolSpec defines the desired state of AddressPool.
            properties:
              addresses:
                description: A list of IP address ranges over which MetalLB has authority.
                  You can list multiple ranges in a single pool, they will all share
                  the same settings. Each range can be either a CIDR prefix, or an
                  explicit start-end range of IPs.
                items:
                  type: string
                type: array
              autoAssign:
                default: true
                description: AutoAssign flag used to prevent MetallB from automatic
                  allocation for a pool.
                type: boolean
              bgpAdvertisements:
                description: When an IP is allocated from this pool, how should it
                  be translated into BGP announcements?
                items:
                  properties:
                    aggregationLength:
                      default: 32
                      description: The aggregation-length advertisement option lets
                        you “roll up” the /32s into a larger prefix.
                      format: int32
                      minimum: 1
                      type: integer
                    aggregationLengthV6:
                      default: 128
                      description: Optional, defaults to 128 (i.e. no aggregation)
                        if not specified.
                      format: int32
                      type: integer
                    communities:
                      description: BGP communities
                      items:
                        type: string
                      type: array
                    localPref:
                      description: BGP LOCAL_PREF attribute which is used by BGP best
                        path algorithm, Path with higher localpref is preferred over
                        one with lower localpref.
                      format: int32
                      type: integer
                  type: object
                type: array
              protocol:
                description: Protocol can be used to select how the announcement is
                  done.
                enum:
                - layer2
                - bgp
                type: string
            required:
            - addresses
            - protocol
            type: object
          status:
            description: AddressPoolStatus defines the observed state of AddressPool.
            type: object
        required:
        - spec
        type: object
    served: true
    storage: false
    subresources:
      status: {}
  - deprecated: true
    deprecationWarning: metallb.io v1beta1 AddressPool is deprecated, consider using
      IPAddressPool
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: AddressPool represents a pool of IP addresses that can be allocated
          to LoadBalancer services. AddressPool is deprecated and being replaced by
          IPAddressPool.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: AddressPoolSpec defines the desired state of AddressPool.
            properties:
              addresses:
                description: A list of IP address ranges over which MetalLB has authority.
                  You can list multiple ranges in a single pool, they will all share
                  the same settings. Each range can be either a CIDR prefix, or an
                  explicit start-end range of IPs.
                items:
                  type: string
                type: array
              autoAssign:
                default: true
                description: AutoAssign flag used to prevent MetallB from automatic
                  allocation for a pool.
                type: boolean
              bgpAdvertisements:
                description: Drives how an IP allocated from this pool should translated
                  into BGP announcements.
                items:
                  properties:
                    aggregationLength:
                      default: 32
                      description: The aggregation-length advertisement option lets
                        you “roll up” the /32s into a larger prefix.
                      format: int32
                      minimum: 1
                      type: integer
                    aggregationLengthV6:
                      default: 128
                      description: Optional, defaults to 128 (i.e. no aggregation)
                        if not specified.
                      format: int32
                      type: integer
                    communities:
                      description: BGP communities to be associated with the given
                        advertisement.
                      items:
                        type: string
                      type: array
                    localPref:
                      description: BGP LOCAL_PREF attribute which is used by BGP best
                        path algorithm, Path with higher localpref is preferred over
                        one with lower localpref.
                      format: int32
                      type: integer
                  type: object
                type: array
              protocol:
                description: Protocol can be used to select how the announcement is
                  done.
                enum:
                - layer2
                - bgp
                type: string
            required:
            - addresses
            - protocol
            type: object
          status:
            description: AddressPoolStatus defines the observed state of AddressPool.
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.1
  creationTimestamp: null
  name: bfdprofiles.metallb.io
spec:
  group: metallb.io
  names:
    kind: BFDProfile
    listKind: BFDProfileList
    plural: bfdprofiles
    singular: bfdprofile
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - jsonPath: .spec.passiveMode
      name: Passive Mode
      type: boolean
    - jsonPath: .spec.transmitInterval
      name: Transmit Interval
      type: integer
    - jsonPath: .spec.receiveInterval
      name: Receive Interval
      type: integer
    - jsonPath: .spec.detectMultiplier
      name: Multiplier
      type: integer
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: BFDProfile represents the settings of the bfd session that can
          be optionally associated with a BGP session.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BFDProfileSpec defines the desired state of BFDProfile.
            properties:
              detectMultiplier:

最低0.47元/天解锁文章