Meaningless numbers and what unit test are not

Introduction
The field of automated testing is a fascinating one with many facets to it. There are perhaps few other areas in the software development discipline (with the exception of software development methodologies, if I may) that teem with so many fallacies and myths.

In this series of blog posts I am going to share my experience of what works when it comes to building automated testing solutions for complex software products based on what I saw work when testing an advanced IDE for model-driven development, online retail banking solutions, and the largest social network in the NL — Hyves.

Meaningless numbers and what unit test are not
Remember those days in the beginning of the last decade when it suddenly became fashionable to create automated tests? The days when the agile manifesto [1] was drawn up, when software engineers started talking of test-driven development (TDD), and Kent Beck’s JUnit became ubiquitous. It was in those days that the whole development organization of the company I worked at received an email that read something like this:

Dear all,

I just took a look at how we are doing with our automated test coverage and see that we have gone up to 54 automated unit tests. Well done! However this is not enough, we should have more. Please keep up the good work.

Kind Regards,
firstName lastName

Out of curiosity, one colleague decided to take a look at the last unit tests contributed by the email’s author. What he found looked similar to this piece of code:

import junit.framework.TestCase;

import com….InterfaceA
import com….ClassBImplementingInterfaceA

public class ModelCheckerTest extends TestCase {

 public void testAParticularChecker() throws Exception {
     ClassBImplementingInterfaceA checker = new ClassBImplementingInterfaceA();
     TestCase.assertTrue(checker instanceof InterfaceA);
 }

}

I still wonder how another hundred of comparable unit-tests could ever have benefited the real users of the product…

Of course, successful software engineers with real-world experience had by then long been building useful automated unit tests (aka component tests) [2], certainly long before JUnit was conceived of and the term TDD coined [3]. Nevertheless, I hope this story gives a taste of excesses brought forth by the hype around automated testing back then.

Sometime after, my team was adding support for deploying applications (generated by our IDE) to a popular enterprise application server. The documentation of the server boasted the following passage:

[The application server’s] final release passes our internal testsuite, the testsuite for the [the application server’s] web service stack and the >2200 web service tests that come with Sun's Compatibility Test Suite (CTS).

Wow, with more than two thousand automated tests and two testsuites into the bargain (most of which were unit tests), this must be a server you can safely deploy your application to and rest assured it would deploy without errors and then run like a train. Well, the brutal facts of reality were a bit more bitter — some standards conforming applications would simply not deploy to this server or would work incorrectly [4].

What could be wrong here? Nothing—only the perception that thousands of automated unit tests can give you any guarantee that the software product you want to use or purchase is usable for you.

Unit tests do have their uses and can bring a lot of value to your software development organization; it’s just that they are more a means of making it easier for your development team to reliably and cost-effectively evolve and maintain its software product. They are simply not intended to directly benefit ultimate users of your product.

What’s next
In the next post I’ll take a look at what unit tests actually are and what value they can bring. Stay tuned.

References
[1] The agile manifesto
[2] Unit testing
[3] The Mythical Man-Month: Essays on Software Engineering by Frederick P. Brooks (Jan 1975)
[4] Are Vendors Becoming More in Charge of Java Enterprise Edition... or is Sun losing control over Java EE?