I’ve decided to plug in Cobertura in (some) of my projects to have an idea on the unit test/code coverage going on. I use Gradle, so I started looking at the Cobertura Gradle plugin. It turns out it’s pretty good — and offers a lot of the functionality that I needed. However, I came across a (weird) issue so I’m going to investigate it and present the findings in this post.

First of all, I’m planning on using Cobertura as part of the build and have it fail the build if the coverage is falling under a certain threshold. This means that if we start with a build that’s acceptable (in terms of code coverage), any changes made in the future need to have accompanying tests, decent enough to keep that threshold within acceptable limits. It turns out this is easily done with the plugin by setting coverageCheckHaltOnFailure coupled with coverageCheckTotalBranchRate and coverageCheckTotalLineRate. To start with I’ve set up the threshold quite low and give it a spin…

Turns out my project failed right away as it had occasionally line coverage falling under 20%!!! WTF?

I set off to look into it — turns out there’s a few packages where my line coverage is nearly 0! A closer look reveals there’s a lot of simple Java beans / POJOs in there and also some exceptions. I’m pretty sure that no one cares about testing their getters and setters and their exceptions constructors, right?

OK, so to remove those from the coverage analysis, Cobertura has coverageIgnoreTrivial — set that to true and off we went. Things are getting better, but I had to couple it with coverageExcludes — simply set this to exclude all my beans and exceptions like this:

coverageExcludes = ['.*com.netflix.twitter.api.ta.beans.*', '.*Exception']

Now, things are getting even better but annoyingly, I see in some of my classes reports of lines not covered by tests — and these are bean-like classes, with very little functionality, which, nevertheless I know it’s tested by unit tests. So what gives?

It appears that a lot of these little annoyances come from (stupid) things like calls to Log4J on one hand and on the other toString() and the likes. There might be classes out there where it’s important to test your toString‘s but this isn’t one of them. So how do I exclude them?

Well, excluding the calls to Log4J is easy: coverageIgnores allows to exclude calls to certain classes — so just include all the Log4J classes and we’re done. The toString() though proved challenging — which is how this blog post started.

To exemplify better, I’ve created a Github project — you can see it here: https://github.com/liviutudor/CoberturaIgnoreAnnotation. You will notice there are a few branches, one for each section of this blog post.

The Problem

I don’t want any of my toString() to be included in code coverage. In this project, the string representations are only used for logging (for debugging reasons) and as such I don’t care about them. For this I’ve created this project, which has only 2 sources — both of them Java beans:

• SimpleBean — offers just 2 properties, a String and an int, together with constructor, getters/setters and a toString() method. The toString() implementation uses Apache Commons Lang3’s ToStringBuilder to build the string via reflection (and basically dump all the bean’s properties in a string).
• ComplexBean — offers still 2 properties, but one of them is a complex object — a Map. It offers getters and setters, constructor and a toString() — however, the toString implementation has a tiny bit of logic in it: it doesn’t use ToStringBuilder as before, as we’re not interested in the contents of the Map here, instead we just want to know if the map has elements or not.

Both of these classes also implement some “business logic” — both have a someLibraryMethod, which we then go ahead and unit test.

This to me is a perfect example of testing only the code that is needed — no need to write unit tests around getter/setter/constructors/toString, right?

Well, turns out setting the threshold for a project like this to 90 (it’s a small project and we are testing everything that matters after all), fails our project during the cobertura stage! To see this in action, have a look at the original branch — running gradle on it gives the following:

lgml-ltudor3QC:CoberturaIgnoreAnnotation > ./gradlew clean build
:clean
:compileJava
:processResources UP-TO-DATE
:classes
:jar
:assemble
:instrument
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
:copyCoberturaDatafile
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
[INFO] Cobertura: Loaded information on 2 classes.
[INFO] Cobertura: Saved information on 2 classes.
:coberturaReport UP-TO-DATE
:coberturaCheck UP-TO-DATE
:generateCoberturaReport
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
Report time: 59ms
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
Report time: 44ms
:performCoverageCheck
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
project failed coverage check. Branch coverage rate of 0.0% is below 90.0%
project failed coverage check. Line coverage rate of 66.6% is below 90.0%
:performCoverageCheck FAILED
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Execution failed for task ':performCoverageCheck'.
> performCoverageCheck: Tests failed to meet minimum coverage levels.
 
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
 
BUILD FAILED
 
Total time: 6.467 secs
lgml-ltudor3QC:CoberturaIgnoreAnnotation >

In other words, the code coverage is not up to scratch. Let’s have a look at what else can we do. Looking at the Cobertura, report, it appears because we have used coverageIgnores with ToStringBuilder, our SimpleBean line coverage is now at 100%. The ComplexBean though still only has 50% coverage and 0% branch coverage.

All due to toString() ! So let’s see what can we do.

Using @Generated Annotation

(For this, we are going to use the branch https://github.com/liviutudor/CoberturaIgnoreAnnotation/tree/using_generated_annotation)

Well, looking through the Cobertura docco’s it seems there’s a nicety out there in the shape of coverageIgnoreMethodAnnotations. This allows us to exclude from coverage analysis all methods annotated with a certain given annotation. The first annotation which comes to mind to use for this is @Generated — as per JavaDoc:

The Generated annotation is used to mark source code that has been generated. It can also be used to differentiate user written code from generated code in a single file.

Ok, so arguably the toString() code is boilerplate code and as such we can make an argument for using this. So we go ahead and annotate the code as follows:

    @Override @Generated("LiviuTudor")
    public String toString() {
        return "ComplexBean: [int=" + propertyInt + ", has data=" + (propertyMap != null && !propertyMap.isEmpty()) + "]";
    }

And then in your build.gradle add this line to the cobertura configuration:

    coverageIgnoreMethodAnnotations = ['javax.annotation.Generated']

Now let’s run the gradle build again and … still failure! In fact if you look at the results, none of the coverage figures changed! And looking at ComplexBean coverage, we can see that toString() is still being flagged as not covered:

Which means that our @Generated annotation and the cobertura configuration had no effect 🙁

(And by the way, you can double check yourself, but the annotation fully-qualified name is correct and so is the spelling of all the properties etc., so we can safely exclude misconfiguration.)

Using @Override Annotation

OK, so I thought, perhaps Cobertura has an issue with classes from javax.* package — let’s try something else here. A quick glance here shows us that both toString() methods are annotated with @Override and in fact the only @Override‘s methods are the toString() ones. So… can we then exclude all the @Override‘s?

(Side note here: all Cobertura examples around this show you how to define your own @interface and use it for exclusion. That is all fine, but in a project where you have a few libraries written by yourself as well and you want a uniform solution, where you can apply the same Cobertura config to all of these that’s not so easy to implement as you want to use the same interface everywhere. Hence my trying to use an interface already defined which also doesn’t introduce extra dependencies — so @Override in this case is as good as any others — it’s in the java.lang package so comes with the JDK.)

So we add this line in our build.gradle (you can see this in this branch of the code https://github.com/liviutudor/CoberturaIgnoreAnnotation/tree/using_override_annotation):

coverageIgnoreMethodAnnotations = ['javax.lang.Override']

And then run our build again…and surprise (really?! :D) the Cobertura report shows exactly the same coverage, nothing has changed … again! 🙁

Using Own @interface

OK, so at this point let’s look at the recommended approach from Cobertura, where we define our own @interface and use it. Look in this branch for the code: https://github.com/liviutudor/CoberturaIgnoreAnnotation/tree/using_own_interface

So we go ahead and define the AvoidCoverage @interface — which is just a placeholder:

public @interface AvoidCoverage {
}

We then annotate our toString() method in ComplexBean:

@Override @AvoidCoverage
public String toString() {
    return "ComplexBean: [int=" + propertyInt + ", has data=" + (propertyMap != null && !propertyMap.isEmpty()) + "]";
}

And then add it to your build.gradle:

    coverageIgnoreMethodAnnotations = ['AvoidCoverage']

Now let’s run our build… woohoo!!! Our build coverage is now at 100% and the Cobertura check passes, so the build completes successfully:

lgml-ltudor3QC:CoberturaIgnoreAnnotation > ./gradlew clean build
:clean
:compileJava
:processResources UP-TO-DATE
:classes
:jar
:assemble
:instrument
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
:copyCoberturaDatafile
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
[INFO] Cobertura: Loaded information on 3 classes.
[INFO] Cobertura: Saved information on 3 classes.
:coberturaReport UP-TO-DATE
:coberturaCheck UP-TO-DATE
:generateCoberturaReport
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
Report time: 62ms
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
Report time: 43ms
:performCoverageCheck
Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
:check
:build
 
BUILD SUCCESSFUL
 
Total time: 6.752 secs
lgml-ltudor3QC:CoberturaIgnoreAnnotation >

Looking at the coverage report for ComplexBean we can see that our toString() is not included now in the coverage:

So that proves a few things:

• our configuration was ok — if there was any doubt in your mind about whether all the config I’m applying is right or not, I hope those doubts are now gone as simply changing the annotation name all works now;
• it seems to be the case that Cobertura does not allow certain annotations to be used in the configuration for this case.

So let’s do one more test to check one last thing: does Cobertura force you to define the @interface in your own project or does it allow for other annotations (pulled in from libraries / dependencies of your project) as long as they’re not in java.* or javax.* package?

 Using Annotations from Other Libraries

As I said as a last test, let’s use an annotation not in java.* or javax.* — pulled in from another library which is a dependency of our project. For this, we are going to use this branch to test: https://github.com/liviutudor/CoberturaIgnoreAnnotation/tree/using_annotation_from_dependencies

I had to search around for an @interface which doesn’t have (that many) side effects so we can use it here. I have settled in the end for using Apache Commons Digester — simply because it seems to be a rather small library and the (brief) look through the docco suggests that using this interface shouldn’t have a side effect. I decided to use the @SetRoot annotation — I have no idea what it does, but based on the name alone, this Cobertura issue has been the root of a lot of evil in my projects, so @SetRoot seemed appropriate 🙂 However, for the purpose of this exercise, feel free to choose any library and annotation you wish really.

We go ahead and annotate our ComplexBean again:

    @Override
    @SetRoot
    public String toString() {
        return "ComplexBean: [int=" + propertyInt + ", has data=" + (propertyMap != null && !propertyMap.isEmpty()) + "]";
    }

And then add the annotation to our Cobertura configuration in build.gradle:

    coverageIgnoreMethodAnnotations = ['org.apache.commons.digester3.annotations.rules.SetRoot']

Now let’s run our build again and what do you know? It works again! We’re actually getting the same results as in the case of using our own defined @interface annotation!

Conclusions

So, I think we can establish the following:

•  Cobertura DOES support the coverageIgnoreMethodAnnotations — but it has a few (silly) restrictions, as it does NOT allow for java.* and javax.* annotations.
• That is sort of silly, bearing in mind the likes of javax.annotation.Generated for instance was introduced exactly to mark generated code — and we don’t want to test generated code, we normally employ code generation tools because they can do a better job than us at this.
• It does allow you to use any other @interface annotation though it seems.
• So ideally, if you want to use this uniformly across all of your projects, have a simple jar/library where you define this annotation and then carry this across in all of your projects; annotate such methods as toString() (and probably hashCode() and others) using this interface and tell Cobertura to ignore methods with this annotation. This has the downside of introducing a dependency just for the sake of Cobertura, which I dislike personally.
• It would be nice to either provide a generic annotation in Cobertura itself (though that does mean pulling all the Cobertura dependencies at compile time) or (my preferred approach) to allow us to use annotations from the standard JDK. As I said, after all, some of them where introduced only to tag the code for cases like this.

I will go ahead and signal this scenario to Cobertura — not sure at this stage if there is already an issue open for this or not but I will probably come back to this post later on with updates.