Emma Testing, 100% Good or Bad?

100% on a test! Sweet, that means my test (download)was awesome!

Unfortunately that isn't always the case when it comes to testing programs. Our newly introduced friend Emma (who is not a hot import model sadly) is a useful tool to check test coverage in programs. I've grown accustomed to thinking 100% is considered the perfect mark. No changes needed. Can't be any better. But Emma changed that.

Emma: The Flirt

100% is Emma's way of flattering you. It's a semi-victory message. On one hand, our test cases have reached every single line of code. But the other hand, just because I was able to reach every single line, it doesn't necessarily mean I exploited every single crack in the code.

When I was attempting to achieve 100%, my main mindset was not to exploit all possible errors. I just wanted to quickly run a test that would incorporate all the missing lines. My tests were moderate at best and were not as complete as it could possibly be. In the readings about equivalence partitions, it mentioned that valid tests fields include those that are valid and invalid. I've tested possibly the more obvious ones in a Stack. Pushing and popping, popping and empty stack, and retrieving the top of an empty & nonempty stack.

A Little Voice In My Head

Okay, so it's not exactly a voice in my head. But, one invalid test never crossed my mind until Dr. Johnson brought it up in class on Monday. And that's the boundaries of a stack/list (the upper end). I tested the negative (zero) end by popping an empty stack. But I never tested the maximum end.

I tried looking into the max size of a list, but it appears dynamic in terms of size. The list grows regardless if the programmer sets the initial size. It is likely there is some unknown limit that could result in an error. Maybe popping 1,000,000 objects could do it. I wouldn't know. I didn't test it.

I would say that improving line coverage improves the system slightly. It can verify methods are working properly by testing output. But it does not reveal everything. The fact of the matter is a bug can still exist if Emma reports 100%. 100% coverage does not necessarily mean it's a good test. But good tests should have 100% coverage plus robustness (and speed). A good tests looks beyond the surface and reads between the lines so to speak.