quarry on k8s

[vivian-rook]

https://quarry-test.wmcloud.org/ offers a running, and seemingly working, quarry on k8s. Which is running off of this PR.

I'm not sure if we want to include ansible or not. The original intention of it was to deploy volumes, though it would seem that we're not going to be able to deploy a ReadWriteMany volume that will function across nodes. So the utility of ansible is reduced in this case. Opinions on if it should be removed outright?

[siddharthvp]

I don't have any experience with Ansible or Terraform, so wouldn't be of much help in reviewing this.

[vivian-rook]

I don't have any experience with Ansible or Terraform, so wouldn't be of much help in reviewing this.

No problem, I can deal with those bits fine. Though I could use some help in testing that everything seems to be working. In particular the error in https://phabricator.wikimedia.org/T349032 I propped up the web pod with a livenessProbe, but ideally it wouldn't be restarting a few times an hour.

[vivian-rook]

Increasing the number of workers seemed to remove the restarting issue. Perhaps something required there be multiple threads? That seems odd, but I guess fits with the symptoms.

NAME                         READY   STATUS    RESTARTS   AGE
pod/redis-5bb5646ff8-pnwqf   1/1     Running   0          20d
pod/web-5b9f9bbd5c-8wkbb     1/1     Running   0          22h
pod/web-5b9f9bbd5c-c245k     1/1     Running   0          22h
pod/web-5b9f9bbd5c-mtcsj     1/1     Running   0          22h
pod/web-5b9f9bbd5c-pql7m     1/1     Running   0          22h
pod/web-5b9f9bbd5c-sfs25     1/1     Running   0          22h
pod/web-5b9f9bbd5c-wkcm9     1/1     Running   0          22h
pod/web-5b9f9bbd5c-xlwgz     1/1     Running   0          22h
pod/web-5b9f9bbd5c-xvm8p     1/1     Running   0          22h
pod/worker-8fc758576-62dhh   1/1     Running   0          20d
pod/worker-8fc758576-67f5g   1/1     Running   0          22h