# Improve your test quality with Mutation testing
## How can we measure test quality ?
### Use our intuition ?
Not really a good indicator : too much subjective.
### Use code coverage ?
Code coverage is a quantitative metric not a qualitative one.
Let's take a simple example :
Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :
It could seem fine at the first look with a 100% lines covered according to my code coverage tool :
But when you take a closer look the tests the assertions are really shitty :
## Mutation testing to our rescue
### What is it ?
> Mutation testing is based on two hypotheses :
>
> The first is the **competent programmer hypothesis**. This hypothesis states that most software faults introduced by experienced programmers are due to **small syntactic errors**.
>
> The second hypothesis is called the coupling effect. The coupling effect asserts that **simple faults can cascade or couple to form other emergent faults.**” - wikipedia
#### The concept behind :
{% hint style="info" %}
Test our tests by introducing MUTANTS (fault) into our production code during the test execution :
* To check that the test is failing
* If the test pass, there is an issue
{% endhint %}
#### 3 steps
1. Generate mutants
2. Launch tests
3. Check result / Generate report
### What is a mutant ?
A mutant is created by altering the production in various ways :
* For conditions :
* Change
* Reverse conditions
* Constant
* For math operations :
* Change increment with decrement (from x++ to x--)
* Change binary operations
* Remove function calls
* Rename constants
* ...
## How to generate mutants ?
To use mutation testing we can use tools like [pitest](https://pitest.org/) (for java) but others are available like [stryker](https://stryker-mutator.io/).
### Start with pitest
Simply add a dependency in your pom to the pitest-maven plugin :
```markup
org.pitest
pitest-maven
1.5.0
```
More info here :
You will then have access to the pitest functionality from your maven :
### Mutators
PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.
### Available mutators
By default it's offering a lot of mutators :
### Mutation report
When you run the pitest report it will generate an html report (by default) :
#### How to read it ?
For each mutation PIT will report one of the following outcomes :
* **KILLED** : exactly what we want our tests have failed so detected the mutant
* **SURVIVED** : our tests have not detected the mutants so we have to improve our assertions
* **NO COVERAGE** : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created
* **TIMED OUT** : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop
* **NON-VIABLE** : mutation that could not be loaded by the JVM as the bytecode was in some way invalid
* **MEMORY ERROR** : might occur as a result of a mutation that increases the amount of memory used by the system
* **RUN ERROR** : means something went wrong when trying to test the mutation
## Real life feedback
* Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.
You can do it from you pom.xml :
```markup
org.pitest
pitest-maven
1.5.0
sample.controller*
com.bil.commons.sample.service*
com.bil.commons.sample*
```
* Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)
### Integrate Pitest in your Integration pipelines
You can integrate it easily in your integration pipelines with SonarQube or [SonarCloud](https://sonarcloud.io/)
* Add an XML Output (still in your pom)
```markup
XML
...
```
* Install the "mutation analysis" plugin on Sonar :
* Add a stage in your pipeline
* For jenkins for example : `stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }`
You will now have your pitest report integrated in your Sonar analysis report :
## Pros & cons
| Pros | Cons |
| ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Help identify missing important tests | Very expensive : Take a lot of time to run |
| It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed | Do not run it on controllers |
| Quantify quality of our tests and so of our system | Requires brainpower to sort ‘junk’ mutations from useful ones. |
| Really useful for core domain model tests | |
From my Point of View Mutation Testing is really a great tool that can help you identify problems and drive your code reviews.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/testing/mutation-testing.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.