Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: BPFMAN_IMG & BPFMAN_AGENT_IMG to overwrite image #18

Merged
merged 1 commit into from
Jun 5, 2024

Conversation

Billy99
Copy link
Contributor

@Billy99 Billy99 commented Jun 3, 2024

The bpfman-operator is setup to allow BPFMAN_IMG to overwrite the default bpfman image, and BPFMAN_AGENT_IMG to overwrite the bpfman-agent image. However, the Makefile is leveraging kustomize. kustomize can replace an image string in a yaml when it knows the k8s object layout, but these images are passed via a ConfigMap, which is opaque data. So the current implementation doesn't work. kustomize does have a ConfigMapGenrator, which can replace the contents of a ConfigMap.

The change is to:

  • Change the kustomization.yaml to use a configMapGenerator.
  • Use sed to replace the default images with those passed in (or just the default image if none were passed in). The kustomize edit set image command doesn't work. The sed command is changing the file content, so rename kustomization.yaml to kustomization.yaml.env and piped the changes to kustomization.yaml.
  • Add kustomization.yaml to .gitignore so changes aren't tracked by git.

@Billy99
Copy link
Contributor Author

Billy99 commented Jun 3, 2024

To reproduce, run something like:

BPFMAN_IMG=quay.io/billy99/bpfman-test:pr-1150 make run-on-kind

Then use kubectl describe pod -n bpfman bpfman-daemon-xxxxx to view the container image that was loaded.

Copy link

codecov bot commented Jun 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 26.38%. Comparing base (768a235) to head (aa19fa7).
Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #18   +/-   ##
=======================================
  Coverage   26.38%   26.38%           
=======================================
  Files          75       75           
  Lines        5021     5021           
=======================================
  Hits         1325     1325           
  Misses       3532     3532           
  Partials      164      164           
Flag Coverage Δ
unittests 26.38% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Billy99
Copy link
Contributor Author

Billy99 commented Jun 3, 2024

NOTE: make run-on-kind is currently not working. As a temp workaround, the BPFMAN_AGENT_IMG and BPFMAN_OPERATOR_IMG in Makefile need a -amd64 appended to them. I didn't fix that because I think it needs a little more discussion on the correct fix.

The bpfman-operator is setup to allow BPFMAN_IMG to overwrite the
default bpfman image, and BPFMAN_AGENT_IMG to overwrite the bpfman-agent
image. However, the Makefile is leveraging kustomize. kustomize can
replace an image string in a yaml when it knows the k8s object layout,
but these images are passed via a ConfigMap, which is opaque data. So
the current implementation doesn't work. kustomize does have a
ConfigMapGenrator, which can replace the contents of a ConfigMap.

The change is to:
* Change the kustomization.yaml to use a `configMapGenerator`.
* Use `sed` to replace the default images with those passed in (or just
  the default image if none were passed in). The `kustomize edit set
  image` command doesn't work.  The `sed` command is changing the file
  content, so rename kustomization.yaml to kustomization.yaml.env and
  piped the changes to kustomization.yaml.
* Add kustomization.yaml to .gitignore so changes aren't tracked by git.

Signed-off-by: Billy McFall <[email protected]>
@@ -291,6 +291,10 @@ test: fmt envtest ## Run Unit tests.
.PHONY: test-integration
test-integration: ## Run Integration tests.
go clean -testcache
cd config/bpfman-deployment && \
sed -e 's@bpfman\.image=.*@bpfman.image=$(BPFMAN_IMG)@' \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

u could use yq to replace in place the image as well and avoid the tmp kustomize ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we discussed yq in the past, but decided we didn't want to add another tool. We are already doing something similar in the examples Makefile.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just FYI sed require some magic to work on mac os, probably something like

ifeq (,$(shell which gsed 2>/dev/null))
SED ?= sed
else
SED ?= gsed
endif

then use SED everywhere

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know. I don't think anyone has tried to build/run on a MAC, but should probably be added to the list.

@msherif1234
Copy link
Contributor

NOTE: make run-on-kind is currently not working. As a temp workaround, the BPFMAN_AGENT_IMG and BPFMAN_OPERATOR_IMG in Makefile need a -amd64 appended to them. I didn't fix that because I think it needs a little more discussion on the correct fix.

WDYT about the following to fix -amd64 tag issue ?

.PHONY: load-images-kind
 load-images-kind: ## Load bpfman-agent, and bpfman-operator images into the running local kind devel cluster.
-       kind load docker-image ${BPFMAN_OPERATOR_IMG} ${BPFMAN_AGENT_IMG} --name ${KIND_CLUSTER_NAME}
+       kind load docker-image ${BPFMAN_OPERATOR_IMG}-${MULTIARCH_TARGETS} ${BPFMAN_AGENT_IMG}-${MULTIARCH_TARGETS} --name ${KIND_CLUSTER_NAME}

@Billy99
Copy link
Contributor Author

Billy99 commented Jun 4, 2024

NOTE: make run-on-kind is currently not working. As a temp workaround, the BPFMAN_AGENT_IMG and BPFMAN_OPERATOR_IMG in Makefile need a -amd64 appended to them. I didn't fix that because I think it needs a little more discussion on the correct fix.

WDYT about the following to fix -amd64 tag issue ?

.PHONY: load-images-kind
 load-images-kind: ## Load bpfman-agent, and bpfman-operator images into the running local kind devel cluster.
-       kind load docker-image ${BPFMAN_OPERATOR_IMG} ${BPFMAN_AGENT_IMG} --name ${KIND_CLUSTER_NAME}
+       kind load docker-image ${BPFMAN_OPERATOR_IMG}-${MULTIARCH_TARGETS} ${BPFMAN_AGENT_IMG}-${MULTIARCH_TARGETS} --name ${KIND_CLUSTER_NAME}

That's probably fine as long as that's the only place it is needed.

@msherif1234
Copy link
Contributor

/LGTM

@Billy99 Billy99 merged commit ee50b19 into bpfman:main Jun 5, 2024
7 checks passed
@Billy99 Billy99 deleted the billy99-kustomize-image branch June 5, 2024 20:16
msherif1234 added a commit to msherif1234/bpfman-operator that referenced this pull request Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants