#325 #314 Enable multi-module project lib dependency incrementing.

[john-tipper]

Fixes #325 and #314: enables modules dependent on other submodules to have their version incremented, even if there are no code changes in that module.

There were two issues with the plugin when it handled multi-module projects:

• modules that depended on other modules did not have their version incremented if there were no code changes within that module. There is no mechanism for force incrementing the version of a module.
• assuming that there were a mechanism for force incrementing a module's version given that its dependent modules' versions have changed, the release tag is placed on HEAD, not on the last relevant commit for that module (the commit closest to HEAD that occurred with changes within that module's directory).

This PR fixes these issues in a non-breaking manner.

The PR adds 2 additional top-level tasks, createRelaseDependents and releaseDependents, which are analogous to createRelease and release respectively. For the purposes of this PR description, releaseDependents will be used, but createReleaseDependents operates in a similar manner. The purpose of releaseDependents is to create a release for a module that has changed and also for all modules that rely on that module, and for this change to then be transitively cascaded down the dependency tree.

The current release mechanism is idempotent and this is a very nice feature and must be retained. Calling releaseDependents repeatedly should not cause multiple releases to be generated.

The releaseDependents task operates in an identical fashion to Gradle's Java plugin, which adds buildNeeded and buildDependents tasks, as described here. Within each project, the release task is added as a dependency of releaseDependents, and that project's releaseDependents task is added as a dependency of all downstream projects. Calling releaseDependents will thus cause release to be called in all downstream dependent projects.

In order to actually cause the release to be created, it is necessary for the release task to take account of whether upstream releases were created. The plugin adds a task called configureReleaseTasksTask that is set as a dependency of releaseDependents and to run prior to release. Calling releaseDependents will thus cause the following execution order: configureReleaseTasksTask, release, releaseDependents. This task checks upstream tasks of the same name and determines if they released a new release by looking at an ext property on that task. If any upstream task caused a release then the task will configure the release task to create a version.

We don't want a user to be able to mess up the idempotency of the release process through poor config, so there is a property within VersionProperties that determines whether a version should be force incremented (it is not parsed from config). That property is set based on a flag in the ReleaseTask that is set by the configureReleaseTasksTask that runs before it.

If a project has changes then the normal release process will be followed. If there are no changes then a release is created if the force increment flag is set. In this case, the new release tag is written to the last commit that relates to that project, which is not necessarily HEAD. This ensures that the version resolution for these projects is correct as they are set up to ignore changes in directories that are not related to the project in question.

Here is an example of one of the MultiModuleProjectDependencyIntegrationTest unit tests showing where release tags are placed using this new process:

commit 8b8866eded61b593442403a058d91816893d043b (HEAD -> refs/heads/master, tag: refs/tags/project1-2.0.1)
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:34 2020 +0000
  
      commit submodule project1
  
  commit 87ff9829a1b123dcec37f97998556a42d9e72b9e (tag: refs/tags/project3-4.0.1, tag: refs/tags/project3-4.0.0)
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:34 2020 +0000
  
      commit submodule project3
  
  commit 6cd13c3a838d80c87761d6afe27f8f4566f1d372 (tag: refs/tags/project2-3.0.1, tag: refs/tags/project2-3.0.0)
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:33 2020 +0000
  
      commit submodule project2
  
  commit 540259aea51bea5b95f483ad67d0cc05be44f4da (tag: refs/tags/project1-2.0.0)
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:33 2020 +0000
  
      commit submodule project1
  
  commit 1d89f8ba50b5a6f0e37947ea2594272f8edb40fe (tag: refs/tags/release-1.0.0)
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:33 2020 +0000
  
      initial commit of top level project
  
  commit 949dffd4afcafa63ef280670ab26e617aa2ffdbf
  Author: John Tipper <[email protected]>
  Date:   Tue Jan 28 22:57:33 2020 +0000
  
      initial commit

[coveralls]

Coverage decreased (-1.6%) to 56.089% when pulling c98199d on john-tipper:#325-support-force-increment-release into 96563f4 on allegro:master.

[john-tipper]

Coveralls seems to be complaining about lack of tests in DummyRepository, which is a class that does nothing and doesn't seem sensible to test. Please let me know what you think, happy to edit as required.

[john-tipper]

I have found that when there are large numbers of submodules (50+) and Gradle builds in parallel, the creation of the release tags causes contention on the Git repo and some of these then randomly fail without an error message. The release tag is then rolled back and the push command then fails because the object ID of the commits no longer exists. The exception that is caused looks like:

Caused by: java.io.IOException: Source ref refs/tags/example-0.1.3 doesn't resolve to any object.
--
160 | at org.eclipse.jgit.transport.RemoteRefUpdate.<init>(RemoteRefUpdate.java:298)
161 | at org.eclipse.jgit.transport.RemoteRefUpdate.<init>(RemoteRefUpdate.java:199)
162 | at org.eclipse.jgit.transport.Transport.findRemoteRefUpdatesFor(Transport.java:652)
163 | at org.eclipse.jgit.transport.Transport.findRemoteRefUpdatesFor(Transport.java:1469)
164 | at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:167)
165 | ... 106 more

The solution is for the release tasks to be run non-concurrently and this is achieved by having all release tasks declare a common directory as a task output. Gradle ensures that only one release task executes at any one time. Whilst it would theoretically be possible to use mustRunAfter to achieve this non-concurrent behaviour, in practice with a large project with potentially many interrelated dependencies I think it would be very difficult to do so in a manner that would guarantee no deadlocks and thus this was the easiest solution.

[TabraizChel]

Was this ever merged in ?

[bgalek]

@TabraizChel nope, as you see this branch is a bit stale and with conflicts

[TabraizChel]

@bgalek yup granted this branch is stale but was another PR opened up at some point to add any similar kind of feature.
I have the same issue mentioned in #325 and if it's not been resolved since then in a different PR I'll update this PR

[gwweaver]

Is there any other way to accomplish a similar functionality to this using the Axion plugin? @john-tipper or @bgalek Or perhaps even changing where Axion detects changes for the release?

[shashken]

I am not sure I understood everything here perfectly, I will share my use case & proposal and see if it fits,
We are moving away to use composite build, as a result when a dependency project gets updated, the main project stays on the same version, this created all sorts of problems for us so we reverted, I opened a new PR #647 that is actually continuing the work I did in #422 to make monorepos work (closed #332)
They idea is to look for latest tag in both project dir and dependency dirs and to check for diff in commits in the deps dir as well
This allows a more direct (composite build) dependency mechanism in monorepos and can help with some use cases