Maven goal to fail build

[bbaudry]

There should be parameters fo fail build

• on a given threshold (cf. #22)
• if there are pseduo-tested methods (Implement a maven mojo to fail the build if there are pseudo-tested and/or partially tested methods #66)

[danzone]

Hi @bbaudry, this feature is addressed by @nicolabertazzo within issue #22. I guess this issue can be closed.

[monperrus]

For the record, from duplicate #142

This would be similar what jacoco is doing to break the build based on coverage info, see https://www.jacoco.org/jacoco/trunk/doc/check-mojo.html
And its usage in Spoon: https://github.com/INRIA/spoon/pull/3641/files

[monperrus]

How does this prioritize for you @oscarlvp?

[oscarlvp]

@monperrus this is a feature that maybe should not be added to Descartes as a PIT plugin. It might better be implemented outside, for example in PitMP or a dedicated Maven plugin. At the moment I see no way to make the build fail from the code of the PIT plugin that works well for all build systems.
Of course, we can always throw an exception when we find a pseudo-tested method. But, the error the user will see might be misleading as the build will fail almost in the same way as if there was a bug inside Descartes.
On the other hand, implementing this feature is a one-liner in any CI server using a tool like jq. It is just a matter of inspecting the JSON from the report.
Anyways, it would be really nice if you have a suggestion on how to achieve this keeping Descartes agnostic to the build system.

[monperrus]

Hi Oscar,

I understand your point, it's not the responsibility of Descartes, it's the responsibility of the mutation testing driver PIT or PitMP.

It seems that PIT supports this:

• Break build on low test strength hcoles/pitest#128
• https://stackoverflow.com/questions/54888240/possible-to-fail-maven-build-when-any-pit-mutation-fails-rather-than-mutation-c

What's your opinion on having this working with Descartes in conjunction?

[oscarlvp]

This is the simplest option. However, it works with the number of mutants, and not with the method classification. You may not have any pseudo-tested methods but still have live mutants in partially-tested methods. But if that suites your goal, then it is the simplest solution.
This issue was opened with the idea of having something similar but using the method classification instead of the raw number of mutants.
See that this feature is actually implemented in PitMP but we need to update it.

[monperrus]

You may not have any pseudo-tested methods but still have live mutants in partially-tested methods. But if that suites your goal, then it is the simplest solution.

Let's try this first in Spoon an see how it works and scales. Then we'll refine. Would you write a wrap-up on what configuration to write to do this (Descartes line threshold + Pit break the build)? Thanks!