velero install failed due to velero-plugin-for-vsphere v1.3.1 is not working in tkgm CSI setup

[jhuisss]

Describe the bug

In my tkgm CSI setup with kubernetes version v1.22.8+vmware.1, I can not install velero server with velero-plugin-for-vsphere image v1.3.1, backup-driver pod and datamgr-for-vsphere-plugin pod are in error status due to:
...
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
...
An error occurred: secrets "velero-vsphere-config-secret" not found
...

To Reproduce

velero install --provider aws --bucket velero --secret-file credentials-velero --snapshot-location-config region=minio --plugins velero/velero-plugin-for-aws:v1.1.0,vsphereveleroplugin/velero-plugin-for-vsphere:v1.3.1 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.187.103.168:9000

Expected behavior

velero server installed successfully. All pods under velero ns are running or completed.

Troubleshooting Information

velero version

Client:
Version: v1.7.0_vmware.1
Git commit: 9e52260568430ecb77ac38a677ce74267a8c2176
Server:
...

velero client config get features

features:

velero-plugin-for-vsphere version: v1.3.1,
and all other plugins in velero deployment: velero-plugin-for-aws:v1.1.0

Kubernetes cluster flavor: tkgm v1.22.8+vmware.1

vSphere CSI driver version: 2.1.0

helm list -A | grep vsphere-csi

vsphere-csi tca-system 1 2022-05-06 08:21:10.673453831 +0000 UTC deployed vsphere-csi-2.1.0 2.1.0

all images in vsphere-csi-controller deployment:
Containers:
csi-attacher:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:a388619e471677fae9120e5169572f9b6f1532cb5cc608ab447fe3e2b43e628f
csi-resizer:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:9e62e1b29b9cb054a8e930a0615627ad1c4fbdf4b0c1aae58bd43489cc997f69
vsphere-csi-controller:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:3eae8a0a84153978f2c85c4c4b387c9ef0380777f5c8d4d368e68083cf36d1ea
liveness-probe:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:f891e2d1871755b0e2f905052fc922fef02bdea3053df8e7d07062265214467c
vsphere-syncer:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:d68a71c0d87febdb1368fbebde07538f5e5438c459b018a7642d4c2de308a2ea
csi-provisioner:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:d939d67d10967b0b1fc4db7784238d5de2efdee1129ef2087534cb653cc6ef69

and all images of vsphere-csi-node daemonset:
Containers:
node-driver-registrar:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:aaee5642312edc75df5ec8dd63d250f215233e3922d37c206b739dd4b0931928
vsphere-csi-node:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:3eae8a0a84153978f2c85c4c4b387c9ef0380777f5c8d4d368e68083cf36d1ea
liveness-probe:
Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:f891e2d1871755b0e2f905052fc922fef02bdea3053df8e7d07062265214467c

kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8+vmware.1", GitCommit:"d797572df69c3951e4e8d495bf7720b594fd1c43", GitTreeState:"clean", BuildDate:"2022-03-21T23:22:25Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8+vmware.1", GitCommit:"d797572df69c3951e4e8d495bf7720b594fd1c43", GitTreeState:"clean", BuildDate:"2022-03-21T23:17:28Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Screenshots

Anything else you would like to add:

I have compared source code of v1.1.1 and v1.3.1:

velero-plugin-for-vsphere/pkg/utils/utils.go

Line 63 in 8c3f94e

func RetrieveVcConfigSecret(params map[string]interface{}, config *rest.Config, logger logrus.FieldLogger) error {

and

velero-plugin-for-vsphere/pkg/utils/utils.go

Line 60 in f5fc6cf

func RetrieveVcConfigSecret(params map[string]interface{}, config *rest.Config, logger logrus.FieldLogger) error {

. In the CSI setup, VC credential is stored as a secret under kube-system namespace, but in v1.3.1 it is not working, it is seeking for the configmap and secret under velero namespace. Seems it is a bug in v1.3.1.

[deepakkinni]

please follow instructions specified in https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/vanilla.md