> For the complete documentation index, see [llms.txt](https://yoan-thirion.gitbook.io/knowledge-base/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/testing/mutation-testing.md).

# Improve your test quality with Mutation testing

## How can we measure test quality ?

### Use our intuition ?

Not really a good indicator : too much subjective.

<div align="center"><img src="/files/-MAkjfMZzxftYOR99rj6" alt=""></div>

### Use code coverage ?

Code coverage is a quantitative metric not a qualitative one.&#x20;

![](/files/-MAkyDg4V7c9Ah0Bhn5o)

Let's take a simple example :

Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :

![](/files/-MAkyYqAHwN6vXUKn3qH)

It could seem fine at the first look with a 100% lines covered according to my code coverage tool :

![](/files/-MAkz7wPLbMlIvGadpgK)

But when you take a closer look the tests the assertions are really shitty :

![](/files/-MAkzHe3ksBUrxHmVwSA)

## Mutation testing to our rescue

### What is it ?

> Mutation testing is based on two hypotheses :&#x20;
>
> The first is the **competent programmer hypothesis**. This hypothesis states that most software faults introduced by experienced programmers are due to **small syntactic errors**.&#x20;
>
> The second hypothesis is called the coupling effect. The coupling effect asserts that **simple faults can cascade or couple to form other emergent faults.**” - wikipedia

#### The concept behind :

{% hint style="info" %}
Test our tests by introducing MUTANTS (fault) into our production code during the test execution :

* To check that the test is failing
* If the test pass, there is an issue
  {% endhint %}

#### 3 steps

1. Generate mutants
2. Launch tests
3. Check result / Generate report

### What is a mutant ?

A mutant is created by altering the production in various ways :

![](/files/-MAl1hF3x69NJ1W38c7z)

* For conditions :
  * Change
  * Reverse conditions
  * Constant
* For math operations :
  * Change increment with decrement (from x++ to x--)
  * Change binary operations
* Remove function calls
* Rename constants
* ...

## How to generate mutants ?

To use mutation testing we can use tools like [pitest](https://pitest.org/) (for java) but others are available like [stryker](https://stryker-mutator.io/).

![](/files/-MAl33UbYAR9nGJI53Jm)

### Start with pitest

Simply add a dependency in your pom to the pitest-maven plugin :

```markup
<plugin> 
	<groupId>org.pitest</groupId>
	<artifactId>pitest-maven</artifactId>
	<version>1.5.0</version>
</plugin>
```

More info here : <https://pitest.org/quickstart/maven/>

You will then have access to the pitest functionality from your maven :

![](/files/-MAl3WfP5HYTaHQb0la-)

### Mutators

PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.

![](/files/-MAl437qtABnA762aTIr)

### Available mutators

By default it's offering a lot of mutators : <https://pitest.org/quickstart/mutators/>

![Example of pitest mutator](/files/-MAl4IlXzKh2i88k93cn)

### Mutation report

When you run the pitest report it will generate an html report (by default) :

![Report summary](/files/-MAl4stCzAQokptj9zl1)

![Report details](/files/-MAl4ziFD-YGKhUajrOS)

#### How to read it ?

For each mutation PIT will report one of the following outcomes :

* **KILLED** : exactly what we want our tests have failed so detected the mutant
* **SURVIVED** : our tests have not detected the mutants so we have to improve our assertions
* **NO COVERAGE** : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created&#x20;
* **TIMED OUT** : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop
* **NON-VIABLE** : mutation that could not be loaded by the JVM as the bytecode was in some way invalid
* **MEMORY ERROR** : might occur as a result of a mutation that increases the amount of memory used by the system
* **RUN ERROR** : means something went wrong when trying to test the mutation

## Real life feedback

* Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.

You can do it from you pom.xml :

```markup
<plugin>    
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.5.0</version>
    <configuration>
        <targetClasses>
            <param>sample.controller*</param>
            <param>com.bil.commons.sample.service*</param>
        </targetClasses>
        <targetTests>
            <param>com.bil.commons.sample*</param>
        </targetTests>
    </configuration>
</plugin>
```

* Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)

### Integrate Pitest in your Integration pipelines

You can integrate it easily in your integration pipelines with SonarQube or [SonarCloud](https://sonarcloud.io/)

* Add an XML Output (still in your pom)

```markup
<configuration>
    <outputFormats>
        <outputFormat>XML</outputFormat>
    </outputFormats>
    ...
```

* Install the "mutation analysis" plugin on Sonar : <https://www.sonarplugins.com/mutationanalysis>
* Add a stage in your pipeline&#x20;
  * For jenkins for example : `stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }`

You will now have your pitest report integrated in your Sonar analysis report :&#x20;

![](/files/-MAl8ya4ymq5QbsA82xy)

## Pros & cons

| Pros                                                                                                       | Cons                                                           |
| ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Help identify missing important tests                                                                      | Very expensive : Take a lot of time to run                     |
| It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed | Do not run it on controllers                                   |
| Quantify quality of our tests and so of our system                                                         | Requires brainpower to sort ‘junk’ mutations from useful ones. |
| Really useful for core domain model tests                                                                  |                                                                |

From my Point of View Mutation Testing is really a great tool that can help you identify problems and drive your code reviews.