# Improve your test quality with Mutation testing
## How can we measure test quality ?
### Use our intuition ?
Not really a good indicator : too much subjective.
### Use code coverage ?
Code coverage is a quantitative metric not a qualitative one.
Let's take a simple example :
Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :
It could seem fine at the first look with a 100% lines covered according to my code coverage tool :
But when you take a closer look the tests the assertions are really shitty :
## Mutation testing to our rescue
### What is it ?
> Mutation testing is based on two hypotheses :
>
> The first is the **competent programmer hypothesis**. This hypothesis states that most software faults introduced by experienced programmers are due to **small syntactic errors**.
>
> The second hypothesis is called the coupling effect. The coupling effect asserts that **simple faults can cascade or couple to form other emergent faults.**” - wikipedia
#### The concept behind :
{% hint style="info" %}
Test our tests by introducing MUTANTS (fault) into our production code during the test execution :
* To check that the test is failing
* If the test pass, there is an issue
{% endhint %}
#### 3 steps
1. Generate mutants
2. Launch tests
3. Check result / Generate report
### What is a mutant ?
A mutant is created by altering the production in various ways :
* For conditions :
* Change
* Reverse conditions
* Constant
* For math operations :
* Change increment with decrement (from x++ to x--)
* Change binary operations
* Remove function calls
* Rename constants
* ...
## How to generate mutants ?
To use mutation testing we can use tools like [pitest](https://pitest.org/) (for java) but others are available like [stryker](https://stryker-mutator.io/).
### Start with pitest
Simply add a dependency in your pom to the pitest-maven plugin :
```markup
org.pitest
pitest-maven
1.5.0
```
More info here :
You will then have access to the pitest functionality from your maven :
### Mutators
PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.
### Available mutators
By default it's offering a lot of mutators :
### Mutation report
When you run the pitest report it will generate an html report (by default) :
#### How to read it ?
For each mutation PIT will report one of the following outcomes :
* **KILLED** : exactly what we want our tests have failed so detected the mutant
* **SURVIVED** : our tests have not detected the mutants so we have to improve our assertions
* **NO COVERAGE** : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created
* **TIMED OUT** : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop
* **NON-VIABLE** : mutation that could not be loaded by the JVM as the bytecode was in some way invalid
* **MEMORY ERROR** : might occur as a result of a mutation that increases the amount of memory used by the system
* **RUN ERROR** : means something went wrong when trying to test the mutation
## Real life feedback
* Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.
You can do it from you pom.xml :
```markup
org.pitest
pitest-maven
1.5.0
sample.controller*
com.bil.commons.sample.service*
com.bil.commons.sample*
```
* Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)
### Integrate Pitest in your Integration pipelines
You can integrate it easily in your integration pipelines with SonarQube or [SonarCloud](https://sonarcloud.io/)
* Add an XML Output (still in your pom)
```markup
XML
...
```
* Install the "mutation analysis" plugin on Sonar :
* Add a stage in your pipeline
* For jenkins for example : `stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }`
You will now have your pitest report integrated in your Sonar analysis report :
## Pros & cons
| Pros | Cons |
| ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Help identify missing important tests | Very expensive : Take a lot of time to run |
| It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed | Do not run it on controllers |
| Quantify quality of our tests and so of our system | Requires brainpower to sort ‘junk’ mutations from useful ones. |
| Really useful for core domain model tests | |
From my Point of View Mutation Testing is really a great tool that can help you identify problems and drive your code reviews.
