Draft about tagging strategy #7

davidcassany · 2018-08-08T06:59:51Z

This commit includes a suggestion about how to handle images tags
using the SUSE Container Registry to deliver them. This is just an
initial draft open to discussion.

Rendered version here

This commit includes a suggestion about how to handle images tags using the SUSE Container Registry to deliver them. This is just an initial draft open to discussion.

chentex · 2018-08-08T09:13:53Z

I just see a caveat with that approach, what will happen with the derived images when the base updates. Say this example, you have mariadb:10.2.15-lp150.2.3.2 based on sle12:sp3

REPOSITORY          TAG                     IMAGE ID
caasp/mariadb       10.2                    1ade07d13d13
caasp/mariadb       latest                  1ade07d13d13
caasp/mariadb       10.2.15-lp150.2.3.2     1ade07d13d13
caasp/sles12        sp3                     68861e65035c

And there is a sle12:sp4 out, so this means mariadb:10.2.15-lp150.2.3.2 should be rebuild but the version of mariadb didn't change, should we use a date time stamp as an extra tag?

REPOSITORY          TAG                                     IMAGE ID
caasp/mariadb       10.2                                    1ade07d13d13
caasp/mariadb       latest                                  1ade07d13d13
caasp/mariadb       10.2.15-lp150.2.3.2                     1ade07d13d13
caasp/mariadb       10.2.15-lp150.2.3.2-20180808-1108       83a8fc6948ec
caasp/sles12        sp3                                     68861e65035c
caasp/sles12        sp4                                     81e71ad9c8f6

This is just my point of view. Hope it makes sense.

davidcassany · 2018-08-08T12:54:21Z

And there is a sle12:sp4 out, so this means mariadb:10.2.15-lp150.2.3.2 should be rebuild but the version of mariadb didn't change, should we use a date time stamp as an extra tag?

@chentex, yes your point of view makes sense. I realize I was no clear enough, the release numbers are related to the image, not to the package (in the example I just copied the release of mariadb package as an example, but this is confusing).

So the mariadb:10.2.15-lp150.2.3.2, if rebuild the release part of the tag should be updated, regardless of the release of the mariadb package. Let's use a better numbering examples, the default build ID of images in OBS. Imagine we have mariadb:10.2.15-3.4 the 3.4 part stands for third revision of the mariadb image in OBS and fourth build of the third revision. Thus for each build there is a new release. See the kubic-mariadb image. There you have the file called kubic-mariadb.x86_64-4.0.1-Build18.8.docker.tar, in that case I'd suggest using the build ID 18.8 as the release number.

chentex · 2018-08-08T13:13:38Z

Thanks for the explanation @davidcassany, I understand now.

jordimassaguerpla · 2018-08-08T15:45:25Z

In general I like it but I see that we need a tool to know that there are new updates in the registry and to pull the new images.

If we have such a tool, then this looks good.

Otherwise, what about updating the manifest files but using the digest instead of a "generic tag"?

flavio · 2018-08-10T07:54:15Z

2018/001-containers-tagging-strategy.md

+require a Kubernetes manifest update for each image update. The manifest could
+be set to some stable mariadb version (e.g. 10.2) and still be updated if the
+image requires a rebuild due some security fixes in the SLE base image or even
+in mariadb itself.


I think we have to extend this section explaining that with Kubernetes we can write our manifests to reference the stable tag (eg <registry-domain>/mariadb:10.2) and specify a imagePullPolicy="Always" pull policy.

We can do that - but it will result in having a "random mix" of builds running. Without a manifest change, kubernetes will not roll out a new build of an image. Setting imagePullPolicy=Always just means that, when a Pod happens to restart for some other reason, the latest image will be pulled.

Kubernetes best practice really is - use a image tag that is guaranteed to never change, and update your manifests when you want an updated image.

flavio · 2018-08-10T07:56:39Z

@davidcassany can you update the RFC with the clarification you gave to @chentex ?

flavio · 2018-08-10T08:02:02Z

@jordimassaguerpla wrote:

In general I like it but I see that we need a tool to know that there are new updates in the registry and to pull the new images.

If we have such a tool, then this looks good.

For CaaSP specific containers we might want to write a piece of software that monitors the versions of the containers we actually own. Make sure they are using the latest version, if not force their update.

All the other software stacks handle that in a different way. For example, software installed via Helm charts (like CAP) references specific digests of the images inside of their helm charts. The containers are updated to a more recent version by pushing out a new version of the chart. From a certain POV this shifts the problem to a higher level (how to figure out if there's a new version of the helm chart).

I'm going to ping the CAP folks to shed some light on their current approach.

flavio · 2018-08-10T08:03:24Z

BTW, the CAP guys are also using container image labels to add extra information to the images. AFAIK they are using them to track specific package versions, build origin, git commits,...

This is something we might want to use as well to attach "invisible" (as in not as prominent as tags) details to our images.

flavio · 2018-08-10T08:05:35Z

2018/001-containers-tagging-strategy.md

+## Introduction
+
+As SUSE starts delivering software shipped in containers accessible
+from the [SUSE container registry](https://registry.suse.de). Images within the


That's the internal registry, the final one is going to be registry.suse.com.

Yes, I know, just added it here because this link is browsable from SUSE network. We can remove the link if this is confusing. The registry.suse.com to me leads to a Connection refused error

This is kubic-project - so maybe use registry.opensuse.org?

davidcassany · 2018-08-10T08:10:25Z

@flavio I already did here 9fc6d38 .

I think we have to extend this section explaining that with Kubernetes we can write our manifests to reference the stable tag (eg /mariadb:10.2) and specify a imagePullPolicy="Always" pull policy.

We can do that, I did not consider the usage of imagePullPolicy="Always" as I thought this was not really solving the update in all cases. I understand this would just update the image if the container is restarted, thus we still need something to monitor registry versions to know if container rerun would be required. But yes I'll extend that part to mention this Kubernetes option.

kiall · 2018-08-10T08:07:40Z

2018/001-containers-tagging-strategy.md

+used in Head project in IBS & OBS.
+
+One of the advantages of using this strategy is that it does not necessarily
+require a Kubernetes manifest update for each image update. The manifest could


Without a manifest update, Kubernetes won't restart running containers with the new build of the image - I don't believe we can get away from doing this without hitting the problem you described above.

kiall · 2018-08-10T08:10:00Z

2018/001-containers-tagging-strategy.md

+require a Kubernetes manifest update for each image update. The manifest could
+be set to some stable mariadb version (e.g. 10.2) and still be updated if the
+image requires a rebuild due some security fixes in the SLE base image or even
+in mariadb itself.


We can do that - but it will result in having a "random mix" of builds running. Without a manifest change, kubernetes will not roll out a new build of an image. Setting imagePullPolicy=Always just means that, when a Pod happens to restart for some other reason, the latest image will be pulled.

Kubernetes best practice really is - use a image tag that is guaranteed to never change, and update your manifests when you want an updated image.

kiall · 2018-08-10T08:11:17Z

2018/001-containers-tagging-strategy.md

+in RPMs. This way the update and delivery worklfow is the same as any other
+packages. However within the containers ecosystem this presents some isssues:
+
+* Containers ecosystems based on Kubernetes are not disigned to interact with


typo - designed

kiall · 2018-08-10T08:13:17Z

2018/001-containers-tagging-strategy.md

+
+`kiwi` and `skopeo` tools can build images with multiple tags (since versions
+9.15.3 and 1.30.0 respectively). Thus the main idea is to keep using stable
+tags to facilitate image references in manifests but also tag images in


Thus the main idea is to keep using stable tags to facilitate image references in manifests

Currently, we don't reference "stable" tags - we reference the most specific tag available (i.e. the one that includes the build service's build number). Otherwise, we would have hit the problem you described above :)

Oh I was not aware that there was a manifest rebuild for each image rebuild, I did not see this part of the chain...

It's not baked into the build service, it's currently happening on boot: https://github.com/kubic-project/caasp-container-manifests/blob/master/admin-node-setup.sh#L12-L45

(and, on boot works only because they are currently RPMs and tx-update means a reboot)

kiall · 2018-08-10T08:14:47Z

2018/001-containers-tagging-strategy.md

+caasp/mariadb        10.2                   1ade07d13d13
+caasp/mariadb        latest                 1ade07d13d13
+caasp/mariadb        10.2.15-3.2            1ade07d13d13
+```


This looks perfect to me, where CaaSP manifests would reference the 10.2.15-3.2 tag. The others, at least for CaaSP, would be unused.

Well I was suggesting to use 10.2 for CaaSP instead and delegate to some other piece of sotfware to track the versions in the registry in order to raise a flag if new image is available. The idea was not causing a manifest rebuild for each image rebuild as this will cause tons of manifest updates.

davidcassany · 2018-08-10T08:20:34Z

BTW, the CAP guys are also using container image labels to add extra information to the images. AFAIK they are using them to track specific package versions, build origin, git commits,...

Currently all KIWI container images include a label org.openbuildservice.disturl that include the DISTURL value that can be used to track the specific project and build of an image.

Also, if we have a clear understanding of which environment variables should be included as labels it should be easy from the build service to include them. We prepared KIWI for that not long ago:

OSInside/kiwi#774

MaximilianMeister · 2018-08-10T08:24:52Z

2018/001-containers-tagging-strategy.md

+
+Before SUSE container registry images have been and are being delivered wrapped
+in RPMs. This way the update and delivery worklfow is the same as any other
+packages. However within the containers ecosystem this presents some isssues:


MaximilianMeister · 2018-08-10T08:25:18Z

2018/001-containers-tagging-strategy.md

+checks if those images are available locally and if not pulls them from the
+registry. It means that if an image reference is overwritten with the same
+reference Kubernetes will not pull it again and then images of a running
+cluster start to diverge from the images available into the registry.


s/into/inside

ajaeger · 2018-08-10T12:01:32Z

While this is the kubic-project, the RFC repository is used for both SUSE CaaS Platform and Kubic. Addressing SUSE CaaS Platform only needs is fine here. But indeed, question is whether we have to add any Kubic specifics as well here.

…

On Fri, Aug 10, 2018 at 12:13 PM Fabian Vogt ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In 2018/001-containers-tagging-strategy.md <#7 (comment)>: > @@ -0,0 +1,189 @@ +# Containers Tagging Strategy + +| Field | Value | +|:-------|:-----------| +| Status | Draft | +| Date | 2018-08-8 | + +## Introduction + +As SUSE starts delivering software shipped in containers accessible +from the [SUSE container registry](https://registry.suse.de). Images within the This is kubic-project - so maybe use registry.opensuse.org? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABB40ORio-zQlIhlOI1y0XRaCzJf5Zhks5uPVzigaJpZM4VzYaF> .

-- Andreas Jaeger [email protected] http://andreasjaeger.blogspot.com

flavio · 2018-08-13T07:31:54Z

From my POV the main goal of this RFC is to get us away from doing nasty hacks like the https://github.com/kubic-project/caasp-container-manifests/blob/master/admin-node-setup.sh#L12-L45 described by @kiall.

Instructing Kubernetes to always pull can ensure the latest stable version of an image is pulled whenever a new container is about to run, however does not help with already running ones.

Referencing specific versions of the images will help with freshly started containers and already running ones. However that moves the problem on the generation of the manifests, because they have to be constantly changed to specific the latest version of the images available. How can we solve that without resorting to the current hack?

kiall · 2018-08-13T13:51:47Z

@flavio / @davidcassany - how about we split this out into two documents.

Doc #1 Tagging and so on. Doc #2 Consumption of those tags for our usage.

I think the pieces in here related to #1 are perfect, and implementation wise, are entirely separate. Lets get that out, and work on the consumption from kubernetes separately?

flavio · 2018-09-10T09:14:51Z

@flavio / @davidcassany - how about we split this out into two documents.

Sounds good to me

davidcassany · 2018-10-01T16:07:06Z

@flavio, @kiall just updated this document to reflect the tagging specific parts only.

I have prepared some initial stuff for the second document. Have placed it in this other PR #10

With this commit the scope of the document is reduced and it is only considered how images are finally tagged into the registry. How are they consumed is out of the scope of the document.

Draft about tagging strategy

bf0b026

This commit includes a suggestion about how to handle images tags using the SUSE Container Registry to deliver them. This is just an initial draft open to discussion.

Avoid confusion between package release and build ID

9fc6d38

flavio reviewed Aug 10, 2018

View reviewed changes

kiall reviewed Aug 10, 2018

View reviewed changes

MaximilianMeister reviewed Aug 10, 2018

View reviewed changes

Adding some words to explain imagePullPolicy=Always

9add60e

jordimassaguerpla mentioned this pull request Sep 20, 2018

WIP add images from registry SUSE/caasp-container-manifests#200

Closed

davidcassany force-pushed the tagging_strategy branch from d2bf44c to eff8e84 Compare October 1, 2018 15:48

davidcassany force-pushed the tagging_strategy branch from eff8e84 to d4e846c Compare October 1, 2018 16:32

jordimassaguerpla previously approved these changes Oct 1, 2018

View reviewed changes

Update the document and focus only on how images are tagged

7a5a601

With this commit the scope of the document is reduced and it is only considered how images are finally tagged into the registry. How are they consumed is out of the scope of the document.

davidcassany dismissed jordimassaguerpla’s stale review via 7a5a601 October 4, 2018 13:05

davidcassany force-pushed the tagging_strategy branch from d4e846c to 7a5a601 Compare October 4, 2018 13:05

Update the document with the current status

4c690d0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft about tagging strategy #7

Draft about tagging strategy #7

davidcassany commented Aug 8, 2018

chentex commented Aug 8, 2018

davidcassany commented Aug 8, 2018

chentex commented Aug 8, 2018

jordimassaguerpla commented Aug 8, 2018

flavio Aug 10, 2018

kiall Aug 10, 2018

flavio commented Aug 10, 2018

flavio commented Aug 10, 2018

flavio commented Aug 10, 2018

flavio Aug 10, 2018

davidcassany Aug 10, 2018

Vogtinator Aug 10, 2018

davidcassany commented Aug 10, 2018

kiall Aug 10, 2018

kiall Aug 10, 2018

kiall Aug 10, 2018

kiall Aug 10, 2018

davidcassany Aug 10, 2018

kiall Aug 10, 2018

kiall Aug 10, 2018

davidcassany Aug 10, 2018

davidcassany commented Aug 10, 2018

MaximilianMeister Aug 10, 2018

MaximilianMeister Aug 10, 2018

ajaeger commented Aug 10, 2018 via email

flavio commented Aug 13, 2018

kiall commented Aug 13, 2018

flavio commented Sep 10, 2018

davidcassany commented Oct 1, 2018

Draft about tagging strategy #7

Are you sure you want to change the base?

Draft about tagging strategy #7

Conversation

davidcassany commented Aug 8, 2018

chentex commented Aug 8, 2018

davidcassany commented Aug 8, 2018

chentex commented Aug 8, 2018

jordimassaguerpla commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flavio commented Aug 10, 2018

flavio commented Aug 10, 2018

flavio commented Aug 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidcassany commented Aug 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidcassany commented Aug 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajaeger commented Aug 10, 2018 via email

flavio commented Aug 13, 2018

kiall commented Aug 13, 2018

flavio commented Sep 10, 2018

davidcassany commented Oct 1, 2018