buildkitd.toml mounted in wrong location for rootless builders

[stickshift]

What steps did you take and what happened

Pushing to insecure registries was failing for me whenever I set rootless to true. I spent a lot of time debugging this and eventually found the buildkitd docs said the location of buildkitd.toml is different in rootful and rootless configurations. I created my own deployment.yaml and changed the configmap mount point from /etc/buildkit to /home/user/.config/buildkit and this solved the problem.

The following were the steps I took to reproduce the issue.

Deploy a local registry

docker run -d --restart=always -p "127.0.0.1:5000:5000" --name registry.k8s.local registry:2

Add registry.k8s.local to /etc/hosts

127.0.0.1 registry.k8s.local
::1 registry.k8s.local

Create a rootless builder with insecure registry

cat < /tmp/buildkitd.toml
debug = true

[worker.containerd]
namespace = "k8s.io"

[registry."registry.k8s.local:5000"]
http = true
insecure = true
EOL
kubectl buildkit create buildkit --rootless --config /tmp/buildkitd.toml

Build and push an image

kubectl build -t registry.k8s.local:5000/buildkit1 --push .

The build works but the push fails due to insecure registry.

What did you expect to happen

When I run kubectl buildkit create buildkit --rootless --config /tmp/buildkitd.toml, I expect the configmap to be mounted at /home/user/.config/buildkit and the settings in my config file to take effect.

When I run kubectl build -t registry.k8s.local:5000/buildkit1 --push . I expect the push to succeed.

Environment Details:

• kubectl buildkit version (use kubectl buildkit version)

Client: 0.1.6
Builder: buildkitd github.com/moby/buildkit v0.11.5 252ae63bcf2a9b62777add4838df5a257b86e991

• Kubernetes version (use kubectl version)

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:14:10Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-10-26T15:22:31Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/arm64"}

• Where are you running kubernetes (e.g., bare metal, vSphere Tanzu, Cloud Provider xKS, etc.)

Laptop using Kind

• Container Runtime and version (e.g. containerd sudo ctr version or dockerd docker version on one of your kubernetes worker nodes)

containerd

Vote on this request

• 👍