Improve your test quality with Mutation testing

Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests.

How can we measure test quality ?

Use our intuition ?

Not really a good indicator : too much subjective.

Use code coverage ?

Code coverage is a quantitative metric not a qualitative one.

Let's take a simple example :

Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :

It could seem fine at the first look with a 100% lines covered according to my code coverage tool :

But when you take a closer look the tests the assertions are really shitty :

Mutation testing to our rescue

What is it ?

Mutation testing is based on two hypotheses :

The first is the competent programmer hypothesis. This hypothesis states that most software faults introduced by experienced programmers are due to small syntactic errors.

The second hypothesis is called the coupling effect. The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.” - wikipedia

The concept behind :

Test our tests by introducing MUTANTS (fault) into our production code during the test execution :

• To check that the test is failing

• If the test pass, there is an issue

3 steps

1. Generate mutants

2. Launch tests

3. Check result / Generate report

What is a mutant ?

A mutant is created by altering the production in various ways :

• For conditions :

  • Change

  • Reverse conditions

  • Constant

• For math operations :

  • Change increment with decrement (from x++ to x--)

  • Change binary operations

• Remove function calls

• Rename constants

• ...

How to generate mutants ?

To use mutation testing we can use tools like pitest (for java) but others are available like stryker.

Start with pitest

Simply add a dependency in your pom to the pitest-maven plugin :

<plugin> 
	<groupId>org.pitest</groupId>
	<artifactId>pitest-maven</artifactId>
	<version>1.5.0</version>
</plugin>

More info here : https://pitest.org/quickstart/maven/

You will then have access to the pitest functionality from your maven :

Mutators

PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.

Available mutators

By default it's offering a lot of mutators : https://pitest.org/quickstart/mutators/

Example of pitest mutator

Mutation report

When you run the pitest report it will generate an html report (by default) :

Report summary

Report details

How to read it ?

For each mutation PIT will report one of the following outcomes :

• KILLED : exactly what we want our tests have failed so detected the mutant

• SURVIVED : our tests have not detected the mutants so we have to improve our assertions

• NO COVERAGE : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created

• TIMED OUT : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop

• NON-VIABLE : mutation that could not be loaded by the JVM as the bytecode was in some way invalid

• MEMORY ERROR : might occur as a result of a mutation that increases the amount of memory used by the system

• RUN ERROR : means something went wrong when trying to test the mutation

Real life feedback

• Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.

You can do it from you pom.xml :

<plugin>    
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.5.0</version>
    <configuration>
        <targetClasses>
            <param>sample.controller*</param>
            <param>com.bil.commons.sample.service*</param>
        </targetClasses>
        <targetTests>
            <param>com.bil.commons.sample*</param>
        </targetTests>
    </configuration>
</plugin>

• Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)

Integrate Pitest in your Integration pipelines

You can integrate it easily in your integration pipelines with SonarQube or SonarCloud

• Add an XML Output (still in your pom)

<configuration>
    <outputFormats>
        <outputFormat>XML</outputFormat>
    </outputFormats>
    ...

• Install the "mutation analysis" plugin on Sonar : https://www.sonarplugins.com/mutationanalysis

• Add a stage in your pipeline

  • For jenkins for example : stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }

You will now have your pitest report integrated in your Sonar analysis report :

Pros & cons

Pros	Cons
Help identify missing important tests	Very expensive : Take a lot of time to run
It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed	Do not run it on controllers
Quantify quality of our tests and so of our system	Requires brainpower to sort ‘junk’ mutations from useful ones.
Really useful for core domain model tests

From my Point of View Mutation Testing is really a great tool that can help you identify problems and drive your code reviews.