Tart in Gitlab executor pulls image on every run

[konlanx]

Hello,

I am trying to set up a Gitlab Pipeline which builds iOS apps on a MacMini through Tart.
For this I am using the Tart Gitlab Executor, which I installed through brew.

My issue is that the Tart image is being pulled from remote on every run. Since I am using the xcode-image this is around 60GB and therefore inflates the runtime of the pipeline by a lot.

I think this is related to the fact that the MacMini has a total disk space of 240GB of which 11GB are already taken by the actual OS running on the machine. I read in one issue (cirrus-cli#645) that insufficient disk space can lead to such behavior because the cached image is being deleted to make space for the running vm. During the actual execution of the pipeline there is still a lot of space left (>50GB), so I think it could also be that I am doing something wrong with my configuration.

I have tried to use the different options provided with the executor to disable this pruning, but I am unable to resolve this issue.

Tart Executor version: 1.19.0-cbb18ff
Gitlab Runner version: HEAD / built 2024-15-11 with go1.23.3

My Gitlab Runner config looks like this (Please excuse the repeated commands for disabling auto-prune, this is an artifact of me trying to resolve the issue):

concurrent = 2
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "macos-runner"
  url = "https://git.some.host/"
  id = 12297
  token = "some-token"
  token_obtained_at = 2024-08-04T16:00:34Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "custom"
  builds_dir = "/Users/admin/builds"
  cache_dir = "/Users/admin/cache"
  [runners.feature_flags]
    FF_RESOLVE_FULL_TLS_CHAIN = false
  [runners.custom]
    prepare_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    prepare_args = ["prepare", "--auto-prune", "false"]
    run_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    run_args = ["run"]
    cleanup_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    cleanup_args = ["cleanup"]
    environment = ["PATH=/opt/homebrew/bin/gitlab-tart-executor:$PATH", "TART_EXECUTOR_ALWAYS_PULL=false", "TART_NO_AUTO_PRUNE=true"]

In my .gitlab-ci.yml I specify the image like this:

image: ghcr.io/cirruslabs/macos-sonoma-xcode:16

...

I will get an external hard drive to increase the available disk space by 1TB or 2TB to be able to cache multiple images, but I feel the disk space that is currently available should be sufficient to maintain one VM, therefore I am asking here if my configuration is bad.

I was able to improve the situation somewhat by manually calling tart pull on the host system, which then caches the image and makes it available for future runs, but at some point the image is deleted and pulled from scratch. I was unable to determine what leads to the deletion, so for now it is "random".

Thank you in advance for your time.

[edigaryev]

You have to set the TART_EXECUTOR_ALWAYS_PULL in your .gitlab-ci.yml, otherwise the environment variables that come from GitLab Runner configuration will get prepended by a special string and the Tart executor won't be able to detect them.

This could be related to the Tart's automatic pruning functionality.

It could be disabled by passing the --auto-prune=false command-line argument to the prepare stage.

[konlanx]

Thank you very much for this very fast response!

I was not aware of this behaviour.

I updated my .gitlab-ci.yml to look like this:

image: ghcr.io/cirruslabs/macos-sonoma-xcode:16

variables:
  TART_EXECUTOR_ALWAYS_PULL: "false"

...

[edigaryev]

I was not aware of this behaviour.

Whoops, I was wrong about this one.

I've updated my original answer to reflect the more likely cause for the issue you're experiencing.

Sorry for the confusion!

[konlanx]

No need to be sorry! Thanks for supporting me here.

I am currently using both the TART_EXECUTOR_ALWAYS_PULL: "false" and the TART_NO_AUTO_PRUNE: "true" in the .gitlab-ci.yml, as well as the prepare stage args (see also the configuration in my initial post): prepare_args = ["prepare", "--auto-prune", "false"].

But I still see it pulling the images in each pipeline step:
Gitlab logs:

Running with gitlab-runner development version (HEAD)
  on macos-runner foobar, system ID: barfoo
Preparing the "custom" executor
Using Custom executor...
2024/11/18 15:14:02 Cloning and configuring a new VM...

(I have been waiting for the download since then 😄)

I was under the assumption it would download the image once and then keep reusing it. All Gitlab-pipelines use the same tart image, so it does not need to rotate it. Kind of how Docker does it.

The host harddrive looks like this after two runs in Gitlab, where the second one is pulling right now:

Sorry that it's in german, I hope the relevant information is clear.

Am I misunderstanding the concept here? Is there a problem in my configuration?

[edigaryev]

That's weird. Could you please try running the tart list on host where your GitLab Runner is started:

1. Before you run the CI job
   • to ensure that the image you're going to use is present in the list
2. After the CI job had started and shown Cloning and configuring a new VM... message
   • to figure out whether the image is still there