Calico使用BGP做网络隔离

Calico使用BGP做网络隔离，关闭全互联模式，自定义BGPPeer做网络隔离

本次测试环境节点情况

[root@bgp-node1 ~]# calicoctl get node -o wide NAME          ASN       IPV4              IPV6   bgp-master1   (64512)   172.20.42.80/32          bgp-node1     (64512)   172.20.42.81/32          bgp-node2     (64512)   172.20.42.82/32

关闭全互联模式

calicoctl get bgpconfiguration -o yaml > bgpconfiguration.yaml

将nodeToNodeMeshEnabled改为false

apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata:   name: default spec:   asNumber: 64512   logSeverityScreen: Info   nodeToNodeMeshEnabled: false

应用使其生效

calicoctl apply -f bgpconfiguration.yaml

此时主机路由中已没有跨节点路由，只有本机到本机容器的路由

[root@bgp-master1 bgp]# ip route default via 172.20.43.253 dev enp1s0 proto static metric 100 blackhole 10.233.94.0/24 proto bird 10.233.94.1 dev cali36dac03ac5d scope link 10.233.94.2 dev cali05c862e5940 scope link 10.233.94.3 dev cali0e83553fade scope link 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.20.40.0/22 dev enp1s0 proto kernel scope link src 172.20.42.80 metric 100

配置全局对等体

表示所有的节点都把172.20.42.80当做对等体去连，172.20.42.80将会有到其他所有节点容器的路由，172.20.42.81和172.20.42.82只会有到172.20.42.80节点容器的路由

apiVersion: projectcalico.org/v3 kind: BGPPeer metadata:   name: my-global spec:   peerIP: 172.20.42.80   asNumber: 64512

在172.20.42.80中
对等体连接情况，与81和82都建立了对等连接

root@bgp-master1 bgp]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.20.42.81 | node specific | up | 01:30:03 | Established | | 172.20.42.82 | node specific | up | 01:30:03 | Established | +--------------+---------------+-------+----------+-------------+

路由情况，拥有到172.20.42.81和172.20.42.82所有容器路由

[root@bgp-master1 bgp]# ip route default via 172.20.43.253 dev enp1s0 proto static metric 100 10.233.87.0/24 via 172.20.42.81 dev tunl0 proto bird onlink blackhole 10.233.94.0/24 proto bird 10.233.94.1 dev cali36dac03ac5d scope link 10.233.94.2 dev cali05c862e5940 scope link 10.233.124.0/24 via 172.20.42.82 dev tunl0 proto bird onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.20.40.0/22 dev enp1s0 proto kernel scope link src 172.20.42.80 metric 100

在172.20.42.81中
只与80建立了对等连接

[root@bgp-node1 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 172.20.42.80 | global | up | 01:30:04 | Established | +--------------+-----------+-------+----------+-------------+

路由情况，只有到80也就是全局对等体所在节点容器的路由

[root@bgp-node1 ~]# ip route default via 172.20.43.253 dev enp1s0 proto static metric 100 blackhole 10.233.87.0/24 proto bird 10.233.87.1 dev calie8d9aa79070 scope link 10.233.87.11 dev calia3e6aa9defc scope link 10.233.94.0/24 via 172.20.42.80 dev tunl0 proto bird onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.20.40.0/22 dev enp1s0 proto kernel scope link src 172.20.42.81 metric 100

在172.20.42.82中

只与80建立了对等连接

[root@bgp-node2 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 172.20.42.80 | global | up | 01:30:04 | Established | +--------------+-----------+-------+----------+-------------+

路由情况，路由也只有到80也就是全局对等体所在节点容器的路由

[root@bgp-node2 ~]# ip route default via 172.20.43.253 dev enp1s0 proto static metric 100 10.233.94.0/24 via 172.20.42.80 dev tunl0 proto bird onlink blackhole 10.233.124.0/24 proto bird 10.233.124.1 dev calia64c26be845 scope link 10.233.124.2 dev cali569dfdef2c4 scope link 10.233.124.3 dev cali0f9f35a43e6 scope link 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.20.40.0/22 dev enp1s0 proto kernel scope link src 172.20.42.82 metric 100

其他节点没有共享80节点的路由，是由于BGP的安全机制，需要将80节点设置为路由反射器才行，详见后文

配置节点间的对等协议

和全局对等体的区别在于加了nodeSelector，将匹配符合节点选择器的节点
先给节点打上标签，将bgp-master1（172.20.42.80）和bgp-node1（172.20.42.81）都打上rack=rack-1的标签

kubectl label node bgp-master1 rack=rack-1 kubectl label node bgp-node1 rack=rack-1

将原来的global对等体删除，创建新的BGPPeer

apiVersion: projectcalico.org/v3 kind: BGPPeer metadata:   name: rack-1 spec:   peerIP: 172.20.42.80   asNumber: 64512   nodeSelector: rack=rack-1

应用使其生效

calicoctl apply -f rack-bgppeer.yaml

172.20.42.80只与172.20.42.81建立了对等连接

Calico process is running.[root@bgp-master1 bgp]# calicoctl node status IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.20.42.81 | node specific | up | 01:50:08 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found.

172.20.42.81只与172.20.42.80建立了对等连接

[root@bgp-node1 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.20.42.80 | node specific | up | 01:50:08 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found.

将节点设置为路由反射器

此处将80节点设置为路由反射器

calicoctl get node bgp-master1 -o yaml > bgp-master1.yaml

修改bgp-master1.yaml

spec:   bgp:     ipv4Address: 172.20.42.80/32     ipv4IPIPTunnelAddr: 10.233.94.0     routeReflectorClusterID: 244.0.0.1   #添加此行   orchRefs:   - nodeName: bgp-master1     orchestrator: k8s

应用使其生效

calicoctl apply -f bgp-master1.yaml

因为可以配置多个路由反射器，不再直接使用ip，而是用peerSelector来匹配多个路由反射器

apiVersion: projectcalico.org/v3 kind: BGPPeer metadata:   name: peer-with-route-reflectors spec:   nodeSelector: rack ==  'rack-1'   peerSelector: route-reflector == 'true'

为了让BGPPeer匹配到，因此还需要给80节点添加标签

kubectl label node bgp-master1 route-reflector=true

此时在81节点查看对等体情况，依然还是和80建立了对等体连接

[root@bgp-node1 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.20.42.80 | node specific | up | 06:01:44 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found.

在81节点查看路由情况，此时已经同步了80节点的路由规则，既有到80节点的路由，还有到82节点的路由

[root@bgp-node1 ~]# ip route default via 172.20.43.253 dev enp1s0 proto static metric 100 blackhole 10.233.87.0/24 proto bird 10.233.87.1 dev calie8d9aa79070 scope link 10.233.87.11 dev calia3e6aa9defc scope link 10.233.94.0/24 via 172.20.42.80 dev tunl0 proto bird onlink 10.233.124.0/24 via 172.20.42.82 dev tunl0 proto bird onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.20.40.0/22 dev enp1s0 proto kernel scope link src 172.20.42.81 metric 100

82节点情况同81节点，确保82节点也添加了rack='rack-1'标签

如果需要设置多个路由选择器，则也需要如80节点一样，先添加routeReflectorClusterID: 244.0.0.1，在添加标签route-reflector=true

参考

Configure BGP peering

如何在kubernetes中配置Calico路由反射器 - KubeSphere 开发者社区

calico 二、配置BGP对等体 | 大番茄