We are going to use the following stuff for the Carma run:

For now just download the above items from each download location and place them somewhere on your harddisk.

These are the things to setup your environment.

Apache Ant version 1.7.0 compiled on December 13 2006

ant test

The are two aspects regarding configuration of Carma. The first one covers the configuration Carma itself (e.g. how to associate classes and test classes, which classes should be completely excluded etc.).

The second aspect is the integration of Carma into the build process which in this guide will be Apache Ant as Commons Codec uses Ant as build tool.

resolver.name = resolver.classmatch

resolver.includePattern = (.*)
resolver.excludePattern =

# ----------------------------------------------------
# settings for class match resolver
# ----------------------------------------------------
resolver.classmatch.testNameSuffix = Test

project.testclassesdir =/home/carmauser/commons-codec-1.3/target/tests
project.classesdir =/home/carmauser/commons-codec-1.3/target/classes

    	
<taskdef name="carma" 
	classname="com.retroduction.carma.ant.AntCarmaTest" />

<target name="carma">
	<carma classesDir="target/classes" testClassesDir="target/tests"
		dependencypath="target/tests" />
</target>

<taskdef name="carmareport" 
	classname="com.retroduction.carma.ant.AntCarmaReport" />

<target name="carmareport">
	<carmareport sourceDir="./src/java/" reportFile="report.xml" outputDirectory="./html/" />
</target>
	

ant compile test

Carma can be run by calling

ant carma

in the root folder of the Commons Codec project. You should see some intermediate output on the console and after half a minute or so you should get a final statement on the results.

    	
[carma] # ---------------------------------------------------------------------------
[carma] # TEST RESULTS SUMMARY 
[carma] #   Total time                : 27.66 sec.
[carma] #   Classes/Tests             : 24/12
[carma] #   Tests Per Class           : 0.50
[carma] #   Mutants/Class             : 29.17
[carma] #   Mutants/Survivors         : 700/188
[carma] #   MutationCoverageRatio     : 73.14 %
[carma] # ---------------------------------------------------------------------------
    
	

You've successfully performed a Carma run on the Commons Codec project and should have a report.xml file in the Commons Codec's root folder. The data in this file is XML and if you like - just have a look on its content. To get some nifty HTML report run

ant carmareport

. Then beside the console output you should find several HTML files on the configured output location (./html/). Just click on the overview.html file and you see a general overview of the Carma results.

The first part of the overview contains the project overview where all classes are listed including a summary of all mutations for each class.

As you can see version 1.3 of the Commons Codec project is notably well tested. However there are a few classes which show signs of weak or missing test cases: DoubleMetaphone, SoundExUtils and RF1522Codec could have better test cases.

The overview contains a histogram called Project Coverage Ration.

This chart visualized the distribution across the number of classes (vertical axis) with a certain coverage. To the right you see a big red bar above the 1 mark. That means that 19 classes have full mutation coverage. More to the left the bars are notably smaller which means a few classes have less than 100% mutation coverage.

Another view - the class browser shows an overview on package level.

It's ment for browsing your project quickly.

Look at the problematic classes in detail. Click on the DoubleMetaphone class in the classes table. After some details about how many mutations have been created and how many mutations have been detected by the tests of the DoubleMetphoneTestCase class you see a long list which explains, which mutations have been detected and which mutations have not been detected by the test case. The mutant column describes the name of the mutation, Line No. tells you where the mutation had been applied, Killed tells you whether the mutation had been successfully detected by the DoubleMetaphone test case and in case it was detected the name of the test case that reveiled the mutation. Obviously you see only the DoubleMetaphoneTestCase class because the class we are currently investigating was covered by this test class due to the configuration we chose earlier.

Scrolling more to the bottom of the class report you find the source code of the DoubleMetaphone class - highlighted with green and red bars. Green bars represent mutations which have been detected, red bars signal mutations which have not been detected.

For example in line 136 there is one of the source code lines mentioned multiple times.

That means that multiple mutations had been injected in this line and due to the different colors the results were mixed. Two mutations were killed by the test case, two other mutations have survived (the red ones). However the mutation process goes from left to right for equal mutation operators. Using this information we can deduct, that the 2nd occurence of line 136 was changing the first plus to minus (see the tooltip of that line) and this mutation survived the test case (the tests were still green). That shoudl enable you to start improving the test class for the DoubleMetaphone class and add a test case which should cover such a change.

If you have any questions, comments or improvements feel free to contact us (contact details are on the website