Description
What happened?
When the csi interface is invoked to attach a volume, if the volume fails to be attached, the removeMountDir method is invoked to clear the directory where the volume is attached. If the pod is deleted, the csi volume fails to be attached, and a message is displayed indicating that the volume fails to be detached. The cause is that the vol_data.json file does not exist.
The code for mount a volume is as follows:
kubernetes/pkg/volume/csi/csi_mounter.go
Lines 300 to 323 in 88dfcb2
The NewUnmounter method is invoked to detach a volume. The vol_data.json file needs to be read. The code is as follows:
kubernetes/pkg/volume/csi/csi_plugin.go
Lines 556 to 559 in 88dfcb2
The log is as follows:
E0120 10:53:49.870240 4083505 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/31c052e2-63f5-4127-95cb-0923db30dd5a-dump podName:31c052e2-63f5-4127-95cb-0923db30dd5a nodeName:}" failed. No retries permitted until 2025-01-20 10:53:50.370182329 +0000 UTC m=+5.022372153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "dump" (UniqueName: "kubernetes.io/csi/31c052e2-63f5-4127-95cb-0923db30dd5a-dump") pod "ndp-spark-history-0" (UID: "31c052e2-63f5-4127-95cb-0923db30dd5a") : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/sop-csi-driver-reg.sock: connect: resource temporarily unavailable"
E0120 13:16:00.518419 4083505 reconciler_common.go:169] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume "dump" (UniqueName: "kubernetes.io/csi/31c052e2-63f5-4127-95cb-0923db30dd5a-dump") pod "31c052e2-63f5-4127-95cb-0923db30dd5a" (UID: "31c052e2-63f5-4127-95cb-0923db30dd5a") : UnmountVolume.NewUnmounter failed for volume "dump" (UniqueName: "kubernetes.io/csi/31c052e2-63f5-4127-95cb-0923db30dd5a-dump") pod "31c052e2-63f5-4127-95cb-0923db30dd5a" (UID: "31c052e2-63f5-4127-95cb-0923db30dd5a") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/vol_data.json]: open /var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume "dump" (UniqueName: "kubernetes.io/csi/31c052e2-63f5-4127-95cb-0923db30dd5a-dump") pod "31c052e2-63f5-4127-95cb-0923db30dd5a" (UID: "31c052e2-63f5-4127-95cb-0923db30dd5a") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/vol_data.json]: open /var/lib/kubelet/pods/31c052e2-63f5-4127-95cb-0923db30dd5a/volumes/kubernetes.iocsi/dump/vol_data.json: no such file or directory"
What did you expect to happen?
Volumes that fail to be mounted can also be unmounted successfully.
How can we reproduce it (as minimally and precisely as possible)?
Simulate a scenario in which a volume fails to be attached to the CSI and the vol_data.json file is deleted. After the volume fails to be mounted, the volume is unmounted.
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
# paste output here
1.31
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here