unable to uninstall calico and get stuck on 'test-install-calicovpp-dev'

[sheercoding]

Environment

• Calico/VPP version: master,bff8eba4cf614b9f3591150ded186e914a0105ab
• Kubernetes version: V1.23.12
• Deployment type: VM
• Network configuration: Calico / host-local IPAM? IPIP / VXLAN enabled? Specific BGP settings?

Issue description
Can't remove calico pods when 'make test-install-calicovpp-dev'.

To Reproduce
Steps to reproduce the behavior:

• 1> successfully set up develop env according to 'https://github.com/projectcalico/vpp-dataplane/blob/master/docs/developper_guide.md'
• 2> change&modify VPP then reload vpp-nodes by 'make load-images'
• 3> update calico-vpp by 'make test-install-calicovpp-dev'

Expected behavior
Remove the old version calico-app and successfully upgrade the new one.

Additional context
Some of the pods stuck on the status of 'Terminating'.
$k get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-apiserver calico-apiserver-69fcc8ccf5-8j9qt 0/1 Terminating 0 7d19h 172.16.104.2 node2
calico-apiserver calico-apiserver-69fcc8ccf5-xv74m 0/1 Terminating 0 7d19h 172.16.166.130 node1 calico-system calico-kube-controllers-555bcb4844-zhn9k 0/1 CrashLoopBackOff 7 (44s ago) 7d19h 172.16.219.67 master calico-system calico-node-gnmkf 1/1 Running 0 7d19h 192.168.0.12 node2 calico-system calico-node-p55z9 1/1 Running 0 7d19h 192.168.0.254 master
calico-system calico-node-tdvw2 1/1 Running 0 7d19h 192.168.0.11 node1
calico-system calico-typha-844764bf54-27ps6 1/1 Running 0 7d19h 192.168.0.11 node1
calico-system calico-typha-844764bf54-d8z6b 1/1 Running 0 7d19h 192.168.0.12 node2
calico-system csi-node-driver-4vrp6 2/2 Running 0 7d19h 172.16.219.68 master
calico-system csi-node-driver-b5lxz 2/2 Running 0 7d19h 172.16.104.1 node2
calico-system csi-node-driver-tt9k2 2/2 Running 0 7d19h 172.16.166.129 node1
calico-vpp-dataplane calico-vpp-node-97trd 2/2 Running 1 (7d19h ago) 7d19h 192.168.0.12 node2
calico-vpp-dataplane calico-vpp-node-s87g5 2/2 Running 0 7d19h 192.168.0.254 master
calico-vpp-dataplane calico-vpp-node-x4mmk 2/2 Running 1 (7d19h ago) 7d19h 192.168.0.11 node1
kube-system coredns-64897985d-b74hd 1/1 Running 0 7d19h 172.16.219.66 master
kube-system coredns-64897985d-wfl6l 1/1 Running 0 7d19h 172.16.219.65 master
kube-system etcd-master 1/1 Running 0 7d19h 192.168.0.254 master
kube-system kube-apiserver-master 1/1 Running 0 7d19h 192.168.0.254 master
kube-system kube-controller-manager-master 1/1 Running 0 7d19h 192.168.0.254 master
kube-system kube-proxy-9svks 1/1 Running 0 7d19h 192.168.0.254 master
kube-system kube-proxy-bpzlp 1/1 Running 0 7d19h 192.168.0.11 node1
kube-system kube-proxy-kbr6g 1/1 Running 0 7d19h 192.168.0.12 node2
kube-system kube-scheduler-master 1/1 Running 0 7d19h 192.168.0.254 master

================
make test-install-calicovpp-dev
make test-install-calico CALICO_INSTALLATION=installation-test-v4
make[1]: Entering directory '/home/douchao/project/vpp-dataplane'
kubectl replace --force -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/tigera-operator.yaml
customresourcedefinition.apiextensions.k8s.io "installations.operator.tigera.io" deleted

error: timed out waiting for the condition
make[1]: *** [Makefile:92: test-install-calico] Error 1
make[1]: Leaving directory '/home/douchao/project/vpp-dataplane'
make: *** [Makefile:99: test-install-calicovpp] Error 2

[hedibouattour]

Hello, sorry for the late reply.
I managed to reproduce this issue. This actually is not a calico-vpp issue as I can reproduce it with calico itself as well. It looks to be some mismatch between tigera operator job and vagrant setup. Operator is not launching the uninstallation of some components for some reason. I am still investigating that. Will keep you updated.