Retrieving the version for large repositories is slow

[balrok]

When following the recommended way to set the version in a gradle project:

project.version = scmVersion.version

it can take a considerable amount of time to compute it for larger projects.

How to reproduce (real project)

• Clone a bigger project: git clone https://github.com/spring-projects/spring-framework.git (28k commits, 10k files) (note: this is actually not that big.)
• Patch it to run scmVersion.version:

diff --git a/build.gradle b/build.gradle
index e8c3550a7c..6da12e0876 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,4 +1,5 @@
 plugins {
+      id 'pl.allegro.tech.build.axion-release' version '1.17.0'
 	id 'io.freefair.aspectj' version '8.3' apply false
 	// kotlinVersion is managed in gradle.properties
 	id 'org.jetbrains.kotlin.plugin.serialization' version "${kotlinVersion}" apply false
@@ -11,6 +12,10 @@ plugins {
 	id 'me.champeau.mrjar' version '0.1.1'
 }
 
+def start = System.currentTimeMillis();
+scmVersion.version
+println("Took " + ((System.currentTimeMillis() - start) / 1000) + "s")
+
 ext {
 	moduleProjects = subprojects.findAll { it.name.startsWith("spring-") }
 	javaProjects = subprojects.findAll { !it.name.startsWith("framework-") }

• spring-framework uses a version-scheme for its tags which are not like it is configured in the plugin. Since they do not seem to have an influence on the speed, simply delete them git tag -d $(git tag) or configure the plugin, that it works
• run ./gradlew cV and watch out for the "Took Xs"

Measurements:

• directly after cloning:
  • 0.5s - 2.0s
• adding 10k files: for i in $(git ls-files|grep -v "/$"); do touch "$i.release-plugin-test"; done
  • 1.5s - 3.5s
• adding 30k commits: for i in $(seq 1 30000); do git commit --allow-empty -m"$i"; done
  • 1.4s - 3.4s

How to reproduce (real project)

Alternatively, one can create an empty git project with only build.gradle:

plugins {
    id 'pl.allegro.tech.build.axion-release' version '1.17.0'
}

def start = System.currentTimeMillis();
scmVersion.version
println("Took " + ((System.currentTimeMillis() - start) / 1000) + "s")

• Depending on the test scenario one can create many commits or many files:
• for i in $(seq 1 30000); do git commit --allow-empty -m"$i"; done
• for i in $(seq 1 100000); do touch "$i.tmp"; done

Measurements:

• 30k empty commits:
  • initially 2s, later 0.15s
• 100k empty uncommitted files in root directory
  • 0.8s - 1.2s

Why is it important?

The current version is calculated with each execution of gradle. This means, that it also runs for each test-execution from the ide.
Also my examples here don't paint the full picture: for a different project it takes 20s on my machine. Due to gradles build-directories and maybe some other random things, I have 600k files.

Proposed solutions

I think most of the time is spent in computing things, which are not always important:

• isClean ScmPosition: added isClean #543
  • here it is maybe possible to add an override-flag, like the overrideBranch-flag (I will prepare a PR for it)
  • alternatively make the isClean-property to be lazily evaluated, so its only computed if it is needed
• retrieving the tags takes some time because it iterates the whole log: https://github.com/allegro/axion-release-plugin/blob/main/src/main/java/pl/allegro/tech/build/axion/release/infrastructure/git/GitRepository.java#L389
  • unclear if all the data is always needed
  • maybe lazy evaluation would also help here, or maybe it is enough to partial-compute the data, only for the relevant tags
  • here https://github.com/allegro/axion-release-plugin/blob/main/src/main/java/pl/allegro/tech/build/axion/release/domain/VersionResolver.java#L85 it is called twice. Maybe there is an optimization possible

[bgalek]

hi! sure, feel free to make a PR with those improvements.

[balrok]

Just found there is also a related issue: #182 (just referencing it in case somebody stumbles over the other)

For the performance-problem with getting the tags I figured out a different optimization:
if the tags are stored inside .git/packed-refs, they are resolved super fast with jgit. If this file doesn't exist yet, it can be created with git gc --aggressive --prune (warning: not sure if it can do any harm, so one might backup the .git-folder before).

Together with the overridenIsClean flag I get the scmVersion.version-calculation down to 0.064s from initially 18s.