DVM/DVMP reconciling too much while waiting on transfers to complete and Pods to come up.

[djwhatle]

Describe the bug
See video, while waiting for DVM to complete, DVM is reconciling at a very high rate when there's no work to be done. I believe I have traced this back to DVMP reconciling at a crazy high frequency when there is no Rsync Pod in existence (7 reconciles/sec) #1004 (comment)

I think we should slow down the reconciles during waiting periods like this to ~3 seconds or so to try to be good citizens on the clusters we're deployed into.

cc @alaypatel07 @pranavgaikwad

Screen.Recording.2021-03-16.at.7.10.27.PM.mov

[djwhatle]

It looks like we are already using PollReQ. Maybe this is DVMP causing frequent watch events? Or maybe this is the echo reconcile from lack of predicates.

[djwhatle]

Got some more data from Jaeger, it looks like DVMP is kicking DVM to reconcile every ~500ms even after the migration is completed.

[djwhatle]

@alaypatel07 I think I figured out what's going. The DVMP resource, if it still exists with an invalid PodRef, will reconcile non-stop and increments its resource version about 15 times every 2 seconds, meaning all of those watch events get sent up the chain.

Edit: I was getting confused between resourceVersion and generation. Forgot resourceVersion is global for etcd. Regardless, as you can see from the jaeger trace, there are a lot of DVMP updates going around.

Screen.Recording.2021-03-18.at.11.08.53.AM.mov

[djwhatle]

Possibly fixed by #1013