statefulset pod's volumeattachment cannot delete after node deleted

[wnxn]

No description provided.

[min-zh]

statefulset elasticsearch-logging-data pod's volumeattachment cannot delete after node deleted

        image: dockerhub.qingcloud.com/elasticsearch/elasticsearch-oss:6.7.0
        imagePullPolicy: IfNotPresent
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/bash
              - /post-start-hook.sh
          preStop:
            exec:
              command:
              - /bin/bash
              - /pre-stop-hook.sh

Let's see the pre-stop-hook.sh

#!/bin/bash
exec &> >(tee -a "/var/log/elasticsearch-hooks.log")
NODE_NAME=${HOSTNAME}
echo "Prepare to migrate data of the node ${NODE_NAME}"
echo "Move all data from node ${NODE_NAME}"
curl -s -XPUT -H 'Content-Type: application/json' 'elasticsearch-logging-data:9200/_cluster/settings' -d "{
  \"transient\" :{
      \"cluster.routing.allocation.exclude._name\" : \"${NODE_NAME}\"
  }
}"
echo ""

while true ; do
  echo -e "Wait for node ${NODE_NAME} to become empty"
  SHARDS_ALLOCATION=$(curl -s -XGET 'http://elasticsearch-logging-data:9200/_cat/shards')
  if ! echo "${SHARDS_ALLOCATION}" | grep -E "${NODE_NAME}"; then
    break
  fi
  sleep 1
done

So pre-stop-hook.sh migrate data of the container to be deleted, so it take a long time， several minutes nearly.

Let's take another simple expirements in QKE.

Statefulset Nginx

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  serviceName: nginx
  selector:
    matchLabels:
      app: nginx
      tier: csi-qingcloud
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
        tier: csi-qingcloud
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          preStop:
            exec:
              command:
                - /bin/bash
                - sleep 1000
        volumeMounts:
        - mountPath: /mnt
          name: mypvc

It sleep 1000s, when stopping the pod's container

After delete the node of pod, it accour the same cirtuation with the stateful "ES" pod.
kubeclt describe pod nginx-0

Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Normal   Scheduled           36m                 default-scheduler        Successfully assigned default/nginx-0 to i-huhm15fj
  Warning  FailedAttachVolume  36m                 attachdetach-controller  Multi-Attach error for volume "pvc-26e5f5f7eb2411e9" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedMount         57s (x16 over 34m)  kubelet, i-huhm15fj      Unable to mount volumes for pod "nginx-0_default(665b8139-eb24-11e9-8721-52542205602a)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx-0". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-x5zsm]

The error same with the "ES".

The process maybe like this:

1. Delete the node of the pod
2. Stop the containners of the pod(it take serveral minutes)
3. During 2, the node is deleted by the cloud-platform
4. After stopping the pod, csi tell cloud-platform to detach the disk, but the node has been
   deleted, so it fails.
5. The pod is transfered to another pod, it tell cloud-platform to attach the disk, but the disk
   hasn't been detached, so it fails.
6. Because of 4 and 5, the pod-transformation locks.