Improve your test quality with Mutation testing

Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests.

How can we measure test quality ?

Use our intuition ?

Not really a good indicator : too much subjective.

Use code coverage ?

Code coverage is a quantitative metric not a qualitative one.

Let's take a simple example :

Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :

It could seem fine at the first look with a 100% lines covered according to my code coverage tool :

But when you take a closer look the tests the assertions are really shitty :

Mutation testing to our rescue

What is it ?

Mutation testing is based on two hypotheses :
The first is the competent programmer hypothesis. This hypothesis states that most software faults introduced by experienced programmers are due to small syntactic errors.
The second hypothesis is called the coupling effect. The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.” - wikipedia

The concept behind :

Test our tests by introducing MUTANTS (fault) into our production code during the test execution :

To check that the test is failing
If the test pass, there is an issue

3 steps

Generate mutants
Launch tests
Check result / Generate report

What is a mutant ?

A mutant is created by altering the production in various ways :

For conditions :
- Change
- Reverse conditions
- Constant
For math operations :
- Change increment with decrement (from x++ to x--)
- Change binary operations
Remove function calls
Rename constants
...

How to generate mutants ?

To use mutation testing we can use tools like pitest (for java) but others are available like stryker.

Start with pitest

Simply add a dependency in your pom to the pitest-maven plugin :

<plugin> 
	<groupId>org.pitest</groupId>
	<artifactId>pitest-maven</artifactId>
	<version>1.5.0</version>
</plugin>

More info here : https://pitest.org/quickstart/maven/

You will then have access to the pitest functionality from your maven :

Mutators

PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.

Available mutators

By default it's offering a lot of mutators : https://pitest.org/quickstart/mutators/

Mutation report

When you run the pitest report it will generate an html report (by default) :

How to read it ?

For each mutation PIT will report one of the following outcomes :

KILLED : exactly what we want our tests have failed so detected the mutant
SURVIVED : our tests have not detected the mutants so we have to improve our assertions
NO COVERAGE : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created
TIMED OUT : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop
NON-VIABLE : mutation that could not be loaded by the JVM as the bytecode was in some way invalid
MEMORY ERROR : might occur as a result of a mutation that increases the amount of memory used by the system
RUN ERROR : means something went wrong when trying to test the mutation

Real life feedback

Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.

You can do it from you pom.xml :

<plugin>    
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.5.0</version>
    <configuration>
        <targetClasses>
            <param>sample.controller*</param>
            <param>com.bil.commons.sample.service*</param>
        </targetClasses>
        <targetTests>
            <param>com.bil.commons.sample*</param>
        </targetTests>
    </configuration>
</plugin>

Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)

Integrate Pitest in your Integration pipelines

You can integrate it easily in your integration pipelines with SonarQube or SonarCloud

Add an XML Output (still in your pom)

<configuration>
    <outputFormats>
        <outputFormat>XML</outputFormat>
    </outputFormats>
    ...

Install the "mutation analysis" plugin on Sonar : https://www.sonarplugins.com/mutationanalysis
Add a stage in your pipeline
- For jenkins for example : stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }