【kubernetes/k8s源码分析】 kubernetes csi external-attacher 源码分析

Status and Releases

Latest stable release	Branch	Min CSI Version	Max CSI Version	Container Image	Min K8s Version	Max K8s Version	Recommended K8s Version
external-attacher v1.2.1	release-1.2	v1.0.0	-	quay.io/k8scsi/csi-attacher:v1.2.1	v1.13	-	v1.15
external-attacher v1.1.1	release-1.1	v1.0.0	-	quay.io/k8scsi/csi-attacher:v1.1.1	v1.13	-	v1.14
external-attacher v1.0.1	release-1.0	v1.0.0	-	quay.io/k8scsi/csi-attacher:v1.0.1	v1.13	-	v1.13
external-attacher v0.4.2	release-0.4	v0.3.0	v0.3.0	quay.io/k8scsi/csi-attacher:v0.4.2	v1.10	-	v1.10

 

The following table reflects the head of this branch.

Feature	Status	Default	Description
CSINode*	Beta	On	external-attacher uses the CSINode object to get the driver's node name instead of the Node annotation.
CSIMigration*	Alpha	On	Migrating in-tree volume plugins to CSI.

 

Description

    external-attacher 对 kubernetes api-server watch VolumeAttachment 资源对象，对相应的后端插件 Controller[Publish|Unpublish]Volume 操作，前提是为 controller 设置  PUBLISH_UNPUBLISH_VOLUME 能力

 

     分为三阶段就是 Provision Attach Mount

CreateVolume +------------+ DeleteVolume
 +------------->|  CREATED   +--------------+
 |              +---+----+---+              |
 |       Controller |    | Controller       v
+++         Publish |    | Unpublish       +++
|X|          Volume |    | Volume          | |
+-+             +---v----+---+             +-+
                | NODE_READY |
                +---+----^---+
               Node |    | Node
              Stage |    | Unstage
             Volume |    | Volume
                +---v----+---+
                |  VOL_READY |
                +------------+
               Node |    | Node
            Publish |    | Unpublish
             Volume |    | Volume
                +---v----+---+
                | PUBLISHED  |
                +------------+

The lifecycle of a dynamically provisioned volume, from
creation to destruction, when the Node Plugin advertises the
STAGE_UNSTAGE_VOLUME capability.

 

     监听 VolumeAttachments 和 PersistentVolumes 两个对象，会调用 SyncNewOrUpdatedVolumeAttachment 来同步，调用 csi dirver 的 Attach 函数

     attach-controller 会创建 VolumeAttatchment.Spec.Attacher 指向的是 external-attacher

 

  ① external attacher  组件启动与 CSI-plugin 插件建立 GRPC 连接，发送 GetPluginInfo 获取 driver 名字，调用 GetPluginCapabilities 获取插件具有的能力，调用 ControllerGetCapabilies 获取 contoller 的能力
  ② kube-controller-manager 中的 AD controller 关注 PVC PV Node Pod等 资源

  ③ AD controller 根据 volume 的 spec 获插件，创建 volumeattachment 资源

  ④ external attacher watch 到 volumeattachment 进行 syncPV syncVA 

  ⑤ 最终调用 CSI plugin 的 Attach Detach 处理

 

     attacher 与 plugin，Attach 则 GRPC ControllerPublishVolumeRequest 请求到插件，这个插件是与 provisioner attacher 为同一个pod

apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  creationTimestamp: "2020-05-14T02:28:32Z"
  name: csi-fbcecaed2a122919ca11ef1fa40d9d51f545db042b9a0acf386c0726c9100a5d
  resourceVersion: "995957"
  selfLink: /apis/storage.k8s.io/v1/volumeattachments/csi-fbcecaed2a122919ca11ef1fa40d9d51f545db042b9a0acf386c0726c9100a5d
  uid: a2ab45ec-a90c-4614-8361-76c0144e70e0
spec:
  attacher: hostpath.csi.k8s.io
  nodeName: master-node
  source:
    persistentVolumeName: pvc-149bb57c-4c8f-48bf-8439-09dd505e6aa2
status:
  attached: true

 

启动命令

    csi-attacher --v=5 --csi-address=/var/lib/csi/sockets/pluginproxy/mock.socket --leader-election --leader-election-type=leases

 

1. main函数

     1.1 创建CSIAttachController对象

       watch API server的volumeAttachment对象

       csiHandler实现了handler，定义在pkg/controller/csi_handler.go

ctrl := controller.NewCSIAttachController(
	clientset,
	csiAttacher,
	handler,
	factory.Storage().V1beta1().VolumeAttachments(),
	factory.Core().V1().PersistentVolumes(),
	workqueue.NewItemExponentialFailureRateLimiter(*retryIntervalStart, *retryIntervalMax),
	workqueue.NewItemExponentialFailureRateLimiter(*retryIntervalStart, *retryIntervalMax),
)

  

2. Run函数

    路径： pkg/controller/controller.go，同样的套路

    同步volumeAttachement与pv

// Run starts CSI attacher and listens on channel events
func (ctrl *CSIAttachController) Run(workers int, stopCh <-chan struct{}) {
	defer ctrl.vaQueue.ShutDown()
	defer ctrl.pvQueue.ShutDown()

	klog.Infof("Starting CSI attacher")
	defer klog.Infof("Shutting CSI attacher")

	if !cache.WaitForCacheSync(stopCh, ctrl.vaListerSynced, ctrl.pvListerSynced) {
		klog.Errorf("Cannot sync caches")
		return
	}
	for i := 0; i < workers; i++ {
		go wait.Until(ctrl.syncVA, 0, stopCh)
		go wait.Until(ctrl.syncPV, 0, stopCh)
	}

	<-stopCh
}

 

3 SyncNewOrUpdatedPersistentVolume函数

    路径 pkg/controller/csi_handler.go

    3.1 如果pv已经被删除，则删除队列无需处理

// Sync and remove finalizer on given PV
if pv.DeletionTimestamp == nil {
	// Don't process anything that has no deletion timestamp.
	klog.V(4).Infof("CSIHandler: processing PV %q: no deletion timestamp, ignoring", pv.Name)
	h.pvQueue.Forget(pv.Name)
	return
}

     3.2 更新persistentVolume

clone := pv.DeepCopy()
newFinalizers := []string{}
for _, f := range pv.Finalizers {
	if f == finalizer {
		continue
	}
	newFinalizers = append(newFinalizers, f)
}
if len(newFinalizers) == 0 {
	// Canonize empty finalizers for unit test (so we don't need to
	// distinguish nil and [] there)
	newFinalizers = nil
}
clone.Finalizers = newFinalizers
_, err = h.client.CoreV1().PersistentVolumes().Update(clone)
if err != nil {
	klog.Errorf("Failed to remove finalizer from PV %q: %s", pv.Name, err.Error())
	h.pvQueue.AddRateLimited(pv.Name)
	return
}

     

4. SyncNewOrUpdatedVolumeAttachment

• 如果标记为删除状态则调用syncDetach函数detach操作
•  如果未标记删除状态则调用syncAttach函数attach操作

func (h *csiHandler) SyncNewOrUpdatedVolumeAttachment(va *storage.VolumeAttachment) {
	klog.V(4).Infof("CSIHandler: processing VA %q", va.Name)

	var err error
	if va.DeletionTimestamp == nil {
		err = h.syncAttach(va)
	} else {
		err = h.syncDetach(va)
	}
	if err != nil {
		// Re-queue with exponential backoff
		klog.V(2).Infof("Error processing %q: %s", va.Name, err)
		h.vaQueue.AddRateLimited(va.Name)
		return
	}
	// The operation has finished successfully, reset exponential backoff
	h.vaQueue.Forget(va.Name)
	klog.V(4).Infof("CSIHandler: finished processing %q", va.Name)
}

    4.1 syncAttach函数

     前面巴拉巴拉一大堆，最终函数调用attacher实现了Attacher接口

// We're not interested in `detached` return value, the controller will
// issue Detach to be sure the volume is really detached.
publishInfo, _, err := h.attacher.Attach(ctx, volumeHandle, readOnly, nodeID, volumeCapabilities, attributes, secrets)
if err != nil {
	return va, nil, err
}

    4.1.1 Attach函数GRPC请求

func (a *attacher) Attach(ctx context.Context, volumeID string, readOnly bool, nodeID string, caps *csi.VolumeCapability, context, secrets map[string]string) (metadata map[string]string, detached bool, err error) {
	client := csi.NewControllerClient(a.conn)

	req := csi.ControllerPublishVolumeRequest{
		VolumeId:         volumeID,
		NodeId:           nodeID,
		VolumeCapability: caps,
		Readonly:         readOnly,
		VolumeContext:    context,
		Secrets:          secrets,
	}

	rsp, err := client.ControllerPublishVolume(ctx, &req)
	if err != nil {
		return nil, isFinalError(err), err
	}
	return rsp.PublishContext, false, nil
}

    4.1.2 更新 volumeattachment 状态

     AD controller 会waitForAttach，检查为 true 则成功完成 attach 操作

func markAsAttached(client kubernetes.Interface, va *storage.VolumeAttachment, metadata map[string]string) (*storage.VolumeAttachment, error) {
	klog.V(4).Infof("Marking as attached %q", va.Name)
	clone := va.DeepCopy()
	clone.Status.Attached = true
	clone.Status.AttachmentMetadata = metadata
	clone.Status.AttachError = nil
	patch, err := createMergePatch(va, clone)
	if err != nil {
		return va, err
	}
	newVA, err := client.StorageV1beta1().VolumeAttachments().Patch(va.Name, types.MergePatchType, patch)
	if err != nil {
		return va, err
	}
	klog.V(4).Infof("Marked as attached %q", va.Name)
	return newVA, nil
}

 

总结：

     watch volumeAttachment pv 资源进行同步操作

     最终向插件发送 GRPC ControllerPublishVolumeRequest 请求

 

     部署 deployment.yaml 配置

kind: Deployment
apiVersion: apps/v1
metadata:
  name: csi-attacher
spec:
  replicas: 3
  selector:
    matchLabels:
      external-attacher: mock-driver
  template:
    metadata:
      labels:
        external-attacher: mock-driver
    spec:
      serviceAccount: csi-attacher
      containers:
        - name: csi-attacher
          image: quay.io/k8scsi/csi-attacher:canary
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
          env:
            - name: MY_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/mock.socket
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/

        - name: mock-driver
          image: quay.io/k8scsi/mock-driver:canary
          imagePullPolicy: "IfNotPresent"
          env:
            - name: CSI_ENDPOINT
              value: /var/lib/csi/sockets/pluginproxy/mock.socket
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/

      volumes:
        - name: socket-dir
          emptyDir:

 

参考：

    https://github.com/kubernetes-csi/external-attacher

    https://kubernetes-csi.github.io/docs/external-attacher.html