Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests.
Last updated
Not really a good indicator : too much subjective.
Code coverage is a quantitative metric not a qualitative one.
Let's take a simple example :
Imagine I have written a Calculator.add method that simply add an integer x and an integer y. I have written some tests on it :
It could seem fine at the first look with a 100% lines covered according to my code coverage tool :
But when you take a closer look the tests the assertions are really shitty :
Mutation testing is based on two hypotheses :
The first is the competent programmer hypothesis. This hypothesis states that most software faults introduced by experienced programmers are due to small syntactic errors.
The second hypothesis is called the coupling effect. The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.” - wikipedia
Test our tests by introducing MUTANTS (fault) into our production code during the test execution :
To check that the test is failing
If the test pass, there is an issue
Generate mutants
Launch tests
Check result / Generate report
A mutant is created by altering the production in various ways :
For conditions :
Change
Reverse conditions
Constant
For math operations :
Change increment with decrement (from x++ to x--)
Change binary operations
Remove function calls
Rename constants
...
To use mutation testing we can use tools like pitest (for java) but others are available like stryker.
Simply add a dependency in your pom to the pitest-maven plugin :
More info here : https://pitest.org/quickstart/maven/
You will then have access to the pitest functionality from your maven :
PIT applies a set of mutation operators (mutators) to our byte code generated by compiling our code.
By default it's offering a lot of mutators : https://pitest.org/quickstart/mutators/
When you run the pitest report it will generate an html report (by default) :
For each mutation PIT will report one of the following outcomes :
KILLED : exactly what we want our tests have failed so detected the mutant
SURVIVED : our tests have not detected the mutants so we have to improve our assertions
NO COVERAGE : same as SURVIVED except there were no tests that exercised the line of code where the mutation was created
TIMED OUT : a mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop
NON-VIABLE : mutation that could not be loaded by the JVM as the bytecode was in some way invalid
MEMORY ERROR : might occur as a result of a mutation that increases the amount of memory used by the system
RUN ERROR : means something went wrong when trying to test the mutation
Configure pitest to skip pojos : exclude every auto generated code (lombok in my case), otherwise ti creates a lot of noise.
You can do it from you pom.xml :
Avoid the run of integration tests otherwise it could be really slow on spring boot micro-services (more than 10 minutes)
You can integrate it easily in your integration pipelines with SonarQube or SonarCloud
Add an XML Output (still in your pom)
Install the "mutation analysis" plugin on Sonar : https://www.sonarplugins.com/mutationanalysis
Add a stage in your pipeline
For jenkins for example : stage('Mutation Test') { mvn "org.pitest:pitest-maven:mutationCoverage" }
You will now have your pitest report integrated in your Sonar analysis report :
Pros
Cons
Help identify missing important tests
Very expensive : Take a lot of time to run
It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed
Do not run it on controllers
Quantify quality of our tests and so of our system
Requires brainpower to sort ‘junk’ mutations from useful ones.
Really useful for core domain model tests
From my Point of View Mutation Testing is really a great tool that can help you identify problems and drive your code reviews.
