Wednesday, November 18, 2009

Code Coverage with Cobertura.

Overview
How to use it for black box testing.
Problems I faced while using it in my project

Overview
Recently I came across the tool named cobertura when I was chatting with one of my friend. It is used to find out the percentage code covered by test cases. So I decided to have a look at this and to find out if I can use that in my project.

Cobertura is a free java based tool used to find out the code coverage in java based application. It shows you the percentage of line covered and percentage of branch covered for each class. It also shows you the line covered in source code by marking it red, if you have source code available.

It's the amazing tool for both black box and white box testing. From the report you can find out how much code has covered and can write down further test cases if you think the code coverage is not enough and if you have code you can exactly come to know which functionalities or which part of functionality requires further test cases. And based on the report you can come to know whether product/project has tested properly and take the decision whether to ship the product or not? So let us see how it’s being done by cobertura?

Cobertura generates the another class files (instrumented class files) from your original class files which help in finding out the code coverrage. You can simply replace this with your original class files or you can set the classpath such that java runs instrumented class files instead of normal class files. It also generates cobertura.ser file along with instrumented classes. Which saves the information about the line and branch covered per classes and used to generate the report.

How to use it for black box testing.
Now let's see one by one how to generate the report.
The coubertra already provides the example for how to use cobertura. But the example is with Junit and ANT so people who want to use it for black box testing and doesn't have much experience with ANT and Junit may find it bit complicated.
In the below example we will create a simple java file and then will find out the code coverage after executing some test cases.

1. Download the coberura depends on your OS and JDK version from the following location. The latest version of the cobertura requires JDK 1.5
http://sourceforge.net/projects/cobertura/files/cobertura/

2. Extract the file at your desired location.
3. Set the location of cobertura.jar file in your classpath.
4. Create three folders named Src, Instrumented, and Reports inside d:\ (or any other directory, just changed the path accordingly in following command )
5. Now Create a simple java file as below in d:\src folder
import java.io.DataInputStream;
public class CodeCoverage {
public static void main(String args[]) throws Exception {
DataInputStream in = new DataInputStream(System.in);
System.out.print("Enter data: ");
int number = Integer.parseInt(in.readLine());;
if ( number > 10 ){
System.out.println(number+" It's greater then 10");
}
}
}

6. Compile the file and generate the class file.
7. Now move to cobertura folder (folder where you have extracted the downloaded file) and to generate the instrumented classes give following command on command prompt.
cobertura-instrument.bat --datafile d:\instrumented\cobertura.ser --destination d:\instrumented d:\src

After executing this command instrumented class file would be generated in d:\instrumented folder. And cobertura.ser file will also be generated in the same directory.

8. Now let's check the report before running the file. It should show 0% line coverage and 0% branch coverage as we haven't execute any of the test case. To generate the report run the following command on your command prompt.

cobertura-report.bat --datafile d:\instrumented\cobertura.ser --destination d:\reports d:\src






9. Now to see the report go to the d:\Reports and open the index.html file in any browser.
10. Now lets try to cover some code by executing the file. Move to instrumented folder and then run the file using following command on command prompt and give 9 as input when it asks for.

Java CodeCoverage

11. Now again generates the report same way as we generated before.




Problems I faced while using it in my project
Once I take the overview and find out how to use it for the black box testing. I decided to find out it for my current project.

If you have jar file or war file for the webapplication, it's quite easy as Cobertura directly generates corresponding instrumented jar or war files and you just need to replace it with original one. However, in our project we were using all class files scattered in different folders. So rather than checking whole application I decided to find out code coverage for part of the application first.

When tried to generate the instrumented class files it gave me following error.

Exception in thread "main" java.lang.UnsupportedClassVersionError: net/sourceforge/cobertura/instrument/Main


I got this error because I was using cobertura 1.9.3, which requires JDK 1.5 however; my project is older and uses JDK 1.3. Therefore, I then downloaded the older version of cobertura, cobertura 1.8 and problem get resolved. It successfully generated the instrumented classes and cobertura.ser file. I also checked out the report to make sure there isn't any error. I was happy to see it showed all the classed with 0% coverage.

Therefore, I finally replace the files inside my project directory with instrumented classes. I also copied the copbertura.jar file on project directory and set its path in classpath. However, I was not sure about where to put cobertura.ser file. Therefore, I decided to restart the webserver without putting it anywhere, because anyway it’s going to be generated. However, the weserver did not start, it hung. And when i check the logs I find out the following error.

Exception in thread "main" java.lang.NoClassDefFoundError: net/sourceforge/cobertura/coveragedata/HasBeenInstrumented

According to error webserver was not able to find the class file require to run instrumented classes. I was confused, as I have set the classpath proper.

However, after having a cup of tea, I finally figure out the root cause. Cobertura.jar file wasn't having the executable permission (Ahh, this happens when you are habituated using windows system). I change the permission and restart the web server with fingers crossed. And web server started successfully, I checked the reports and it was showing the coverage successfully. Finally I was revealed and happy.

7 comments:

ProBlogger said...

Hi Gaurang

Thanks a lot for this awesome post that helped me get started on Code Coverage.

However , I have across a problem.
There is one particular jar I want to instrument and find the coverage. However, that jar is invoked by the web application internally. I cannot invoke the jar manually .

I have instrumented the jar and placed it on the webserver. Now, what do I need to do on the webserver, so as to get the Cobertura write in to the ser file when I run the web application ?

I have added the cobertura.jar to the classpath on the webserver which is windows.

Can you please help me out ?

thanks
Siva

SudhirC said...

Thanks for the article..
I dont see the commands in Step 9..

Scorpion said...

Hi Gaurang,

I see that as long as my instrumented class file (in instrumented folder) is having filename same as class name I'm unable to execute the instrumented class. Once I rename the file to match other than the class name, am able to run the class and input the data. But when I generate the report nothing is recorded and stats show 0% coverage. I something is missing which am unable to figure out, plesae help me out at the earliest. Below is the error am seeing when class name inside the class file is matching with its filename.

D:\CoberturaWorkPlace\instrumented>Java CC
Exception in thread "main" java.lang.NoClassDefFoundError: net/sourceforge/cober
tura/coveragedata/HasBeenInstrumented
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(Unknown Source)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$000(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
Caused by: java.lang.ClassNotFoundException: net.sourceforge.cobertura.coveraged
ata.HasBeenInstrumented
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 12 more
Could not find the main class: CC. Program will exit.

nikita said...

Hi.. I was able to do all that is written till step 9 for our project (with some probs here and there)... but when I execute a test manually on that machine (for eg. our project has a feature to add an account from a UI we have provided, so from the UI I manually added the account)..and then I generated the report..even then the coverage reported was 0%... I do not know why did it not consider my manual test ... please help :(

Anonymous said...

Can you please provide the command at point 10). I'm stuck at it. Please it would eb a great help!!!

Unknown said...

command is already mentioned, it is:
Java CodeCoverage

suraj said...

Hi Gaurang,

i got the same issue "Exception in thread "main" java.lang.NoClassDefFoundError: net/sourceforge/cobertura/coveragedata/HasBeenInstrumented" though i have cobertura-1.9.3.jar in my project/classpath.
do i need to add cobertura.jar as well to the project? i thought to add this cobertura.jar file to the project but this time , the cobertura report did not get generated. could you please help here.

Post a Comment