Bytes, Java, Mutation Testing with PIT

What?

PIT is a mutation tester for Java. It makes small changes to your code, called mutants, and reruns your tests. A mutant that makes a test fail is killed. A mutant no test notices survives.

Why?

Coverage says a test ran a line. Mutation testing says whether it would notice the line being wrong.
A test with no assertions gets full coverage and catches nothing. Generated tests are often like that.
A survivor points at the exact line your tests run without checking.

How?

Setup

build.gradle.kts

plugins {
    java
    id("info.solidsoft.pitest") version "1.19.0"
}
 
java {
    toolchain { languageVersion = JavaLanguageVersion.of(25) }
}
 
dependencies {
    testImplementation(platform("org.junit:junit-bom:6.1.3"))
    testImplementation("org.junit.jupiter:junit-jupiter")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}
 
tasks.test { useJUnitPlatform() }
 
pitest {
    pitestVersion = "1.30.1"
    junit5PluginVersion = "1.2.3"
    addJUnitPlatformLauncher = false
    targetClasses = listOf("com.example.add.*")
}

JUnit 6 needs addJUnitPlatformLauncher = false. Without it, the plugin forces a JUnit 5 launcher and dependency resolution fails.

The code

src/main/java/com/example/add/Add.java

public interface Add {
  static long add(long left, long right) {
    return left + right;
  }
}

A test that checks nothing

src/test/java/com/example/add/AddTest.java

@Test
void itWorks() {
  assertDoesNotThrow(() -> Add.add(2, 2));
}

./gradlew pitest

>> Line Coverage (for mutated classes only): 1/1 (100%)
>> Generated 2 mutations Killed 0 (0%)
SURVIVED  Replaced long addition with subtraction
SURVIVED  replaced long return with 0

The test passes, coverage is 100%, and it would still pass if add subtracted or returned 0.

A test that checks the answer

src/test/java/com/example/add/AddTest.java

@Test
void itWorks() {
  var result = Add.add(2, 2);
  assertEquals(4, result);
}

>> Line Coverage (for mutated classes only): 1/1 (100%)
>> Generated 2 mutations Killed 2 (100%)
KILLED    Replaced long addition with subtraction
KILLED    replaced long return with 0

Coverage is the same, and every mutant is caught.

Reading the report

open build/reports/pitest/index.html

Line coverage and mutation coverage sit side by side for each class. The gap between them is the code you run without checking.
Each line is colored by its mutants. Green means killed, red means survived, and the mutator that made each change is listed beside it.
Kill the survivors that would hurt. Some don’t matter, such as a removed close() nobody tests for a leak. Record why you’re leaving those.

What 100% doesn’t mean

2 × 2 is also 4. PIT’s default mutators only ever turn + into -. The swap to * is one of the AOR (arithmetic operator replacement) mutators. Since PIT 1.7.5 they live in a separate plugin, org.pitest:pitest-rv-plugin. With it, that mutant survives add(2, 2). Choosing inputs is still your job: add(2, 3) kills it.
The score only covers the mutants PIT generated. It isn’t proof of correctness.

For the same example under mutastic, see Bytes, Java, Mutation Testing with mutastic.

For mutation testing across a whole build, see for Builders.

Sources

PIT: basic concepts: mutants, killed and survived, mutation coverage.
PIT: mutators: what each mutator changes, and which are on by default. The page still lists AOR under ALL, but PIT 1.30.1 reports “Mutator or group AOR is unknown” without the RV plugin.
PIT 1.7.5 release notes: “#993 Move rv mutators to external plugin”.
pitest-rv-plugin: the research mutators, AOR among them.
gradle-pitest-plugin: the info.solidsoft.pitest Gradle plugin and its pitest { … } options.
pitest-junit5-plugin: runs JUnit Platform tests, JUnit 6 included, under PIT.