Carma Documentation

Mutation Testing is a known method of white-box testing. The basic principle is to perform a single modification of the code under test (called Mutation) in a controlled manner and validate whether existing test cases for the code fail or pass. There are two possible outcomes:

The above conditions can be interpreted as follows. If at least one test fails the mutation was revealed by the test harness. This is considered to be a good thing as it shows that the code location where the mutation was performed is covered to a certain degree by a test case which is sensitive to errors of the original code. That means that an error in this code location is less likely. The more mutations are performed and detected by the test harness the more stable is the code. Given the case that a mutation is performed and the test harness does not react with at least one failing test case the modified code location is obviously not very well-tested as errors can survive without test cases showing that.

The conclusion therefore is: The more mutations survive the worse is the quality of your test harness. Accordingly the more mutations are revealed by the test harness the better is the quality of your test harness.

Combined with the fact that the quality of the test cases has a direct relation to the quality of the code base (e.g. your product) delivers a very solid and objective metric for code quality.

Prototypical Mutation Testing tools can be found in the public and prove the feasability of mutation testing. However no mutation testing software has been publicly available which is easy to use and mature to seriously consider it as a tool for software developers in a productive environment.

The compelling advantage of mutation testing is that it is a stable and non-subjective metric in terms of expressiveness and interpretation. Compared to code coverage metrics for example mutation testing results not just tell you wether code has been executed during test exeuction. It tells you whether the test is sensitive to errors in a certain location or not. Consider how easily code coverage tools and their users can be tricked by a developer who creates one big integration test case without any assertions. A code coverage tool would show you that most of the code has been executed which would be interpreted as "well-tested". As the existence of assertions is not considered at all during code coverage evalation the expressiveness is obviously very weak.

In times of outsourcing and large development teams mutation testing is a big benefit for a mature quality verification of code deliveries on a technical level. It reliably identifies lacking or missing test cases and either indicates need for further actions (like for example rework of certain code deliveries) or confirms a high quality of the code base and its test cases.

While Mutation Testing can be done manually in theory it's a process which practiacally must be done in an fully automated manner. The reason is that - based on the principle of mutation testing - it requires large amounts of processing power to execute the test harness. Therefore in small projects (a few hundert lines) can easily be verified by using Mutation Testing. However larger projects require much more processing resources as larger projects have a much larger code base and therefore many more mutations must be created and injected and furthermore the number of test cases most often correlates with the size of the project which means more testcases must be executed as well. Without further action the time required for executing one complete run grows non-linearily.

For attaching Carma to an Ant build Carma provides a few targets and task definitions which must be added in a central location of the target builds ant scripts. The following snippet can be used as a template:

<taskdef name="carma" classpath=".." classname="com.retroduction.carma.ant.AntCarmaTest" />
        
<target name="carma">
	<carma classesDir=".." testClassesDir=".." dependencyClassPathUrls=".."/>
</target>

<taskdef name="carmareport" classpath=".." classname="com.retroduction.carma.ant.AntCarmaReport" />

<target name="carmareport">
	<carmareport sourceDir=".." outputDirectory=".." />
</target>

The first task definition and target is required for processing the code with Carma via the build system. The second task definition and target is required for creating HTML reports via the build system. One goal serves the activation of a Carma run, the second goal allows activation of the report generation mechanism of Carma.

The values for the attributes ("..") above must be replaced with proper values depending on your environment

Carma processes the code base by typing the following statement on the command line:

ant carma

Carma reports can be generated by calling:

ant carmareport

For attaching Carma to an Maven 2 build Carma provides a goals which must be used in the pom.xml of the target build. The following snippet can be used as a template:

<build>
	<plugins>
		<plugin>
			<groupId>com.retroduction.carma</groupId>
			<artifactId>MavenPlugin</artifactId>
			<configuration>
				<mutationConfigFile>./mutationConfig.xml</mutationConfigFile>
			</configuration>
			<executions>
				<execution>
					<id>mutationtest</id>
					<goals>
						<goal>mtest</goal>
					</goals>
				</execution>
				<execution>
					<id>mutationreport</id>
					<goals>
						<goal>mreport</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

As you can see the required declaration in the pom.xml is pretty simple. the property mutationConfigFile points to a separate configuration file. This file defines the required attributes depending on your build environment and must be adapted accordingly.

Carma processes the code base by typing the following statement on the command line:

mvn mutationtest

Carma reports can be generated by calling:

mvn mutationreport