FailedMount due to "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"

[kuldipshpe]

Describe the bug
Volume is not able to attach to the pod with below errors and pods are not going to running state.
Events:
Type Reason Age From Message

---

Normal SuccessfulAttachVolume 5m40s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ec70e2c3-750f-4736-9099-6419d601f3fa"
Warning FailedMount 2m59s (x6 over 4m55s) kubelet MountVolume.MountDevice failed for volume "pvc-ec70e2c3-750f-4736-9099-6419d601f3fa" : rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"
Warning FailedMount 43s kubelet MountVolume.MountDevice failed for volume "pvc-ec70e2c3-750f-4736-9099-6419d601f3fa" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

To Reproduce
Noticed this issue when one of VM is restarted which is part of the replica.

Expected behavior
Volume should be able to attach and pod going to Running state.

See attached mayastor dump system logs
mayastor-read_unix_error.tar.gz

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: [e.g. NixOS]
• Kernel version
• MayaStor revision or container image

Additional context
Add any other context about the problem here.

[tiagolobocastro]

Hi, are you using the latest kubectl-plugin? I don't see the k8s logs on the bundle (when loki is not used we take the logs from k8s)

[kuldipshpe]

@tiagolobocastro
inspection-report-20241015_150709.tar.gz
using Kubectl Plugin (kubectl-mayastor) revision c1ec1d7ee8f2 (v2.7.1+0)
see attached k8s and mayastor dump system logs.
mayastor-2024-10-15--20-42-42-UTC-node-reboot-sock-error.tar.gz

[tiagolobocastro]

hmm can't find that volume ec70e2c3-750f-4736-9099-6419d601f3fa in those logs?