Airgapped install support

[nbykov0]

• example Makefile and generated Dockerfile for packages/system/capi-operator
• script for generating/updating a dockerfile during make update
• script for pushing an image to a custom registry with make image

@kvaps please check if I'm moving in a right direction

[kvaps]

Overall, you can split this PR into three parts:

First part should provide automation for creating structure of the used images for every package by calling:

make update

The second one should provide unified makefile target to build all the images from images/ directory and output <image>.json and <image>.tag files:

make image

so every image can be builded separately:

make image-<image>

It should always output metadata and tag files with same name as image
I can help you with writing this target.

In the third it should substitute builded image metadata and tag into helm chart, so the charts should be modified or override the named template to putting correct image infromaton from these files:

cozystack/packages/system/cilium/templates/_helpers.tpl

Lines 1 to 3 in 02a41e1

	{{- define "cilium.image" -}}
	{{ .Files.Get "images/cilium.tag" | trim }}@{{ index (.Files.Get "images/cilium.json" | fromJson) "containerimage.digest" }}
	{{- end -}}

This is a huge work :)

[kvaps]

This wont work, because we have to replace all the images with ours in the helm chart, the output of helm template . will always return our values. Try parsing upstream helm chart instead

[nbykov0]

I think I'll leave the PR as a place for duscussion, and then make separate ones with actual changes.

[nbykov0]

Next steps:

• a script for pushing all czk images to a provate repo without changing digests (see crane)
• make a global override for registry from czk configmap

[RaSerge]

Hello, @nbykov0!
I'm interested in adopt cozystack for my campus lab air-gapped installation. Yesterday i tested your solution and found that isn't easy to use (for me at least). It looks like i need to change every Makefile for system/core/etc components, right. Maybe you have ideas how to move that to upper level (something in scripts/dir) or to solve this issue more globally?

[nbykov0]

Maybe you have ideas how to move that to upper level (something in scripts/dir) or to solve this issue more globally?

Hi @RaSerge! Yeah, I also dislike how it goes.
Seems to me like there is no good way to solve this on the upper level for every component, as each one of those generate image names in different ways. I'm thinking of parsing every values.yaml with yq or so, and adding this to every Makefile.
If you have any ideas please let me know.

[RaSerge]

My ideas:

1. Use Fluxcd Kustomization to globally add prefix (local registry) to all images - modified images will be like a myregitry.local/ghcr.io/aenix-io etc)
2. Run simple bash script with grep+sed at end of manifests build (target make manifests probably) to globaly replace external registries with local one.

Yesterday i found this helm plugin (it extracts images from charts).
https://github.com/nikhilsbhat/helm-images

3. Use helm-images plugin for extract all images, run crane to copy this images to local registry, modify this plugin with replace functionality or just use bash with grep+sed to replace registries in found images with local one

[gecube]

no need to rewrite image names. It could be achieved simpler by

1. using harbor as local cache images
2. or patching image names directly in the cluster with any kind of policy engine

[RaSerge]

no need to rewrite image names. It could be achieved simpler by

1. using harbor as local cache images

2. or patching image names directly in the cluster with any kind of policy engine

1. Looks good. i already tried variant with mirroring, it works fine, and .. we need to populate caching registry with images.
2. Can you explain this solution, please

[gecube]

Something like this

https://t.me/kubernetes_ru/838830

[nbykov0]

1. Looks good. i already tried variant with mirroring, it works fine, and .. we need to populate caching registry with images.

There is one more way by Talos itself, see
https://www.talos.dev/v1.7/talos-guides/configuration/pull-through-cache/

[nbykov0]

1. using harbor as local cache images
2. or patching image names directly in the cluster with any kind of policy engine

Well, it would work for sure, but to me it seems more like a workaround.

In case of a local cache I can almost feel all the concerns coming from security etc., who will notice images from public registries in private clusters :) And overriding image names on the fly might break image signing and all that.
I'll try to implement image names overides and stuff, but it will be done for every component, and by default all images will be replicated to cozystack repo, and a release will contain only them.

However, a pull-through/local cache is a needed feature for sure.
@RaSerge if it will work for you for now, maybe you could investigate that? I mean, a cache, and a way to upload unmodified images there. In this case you will not be not blocked in any way by this PR, and it would be a separate needed feature.