JUnit Lab

Emma reported 100% coverage for my stack project (download). Good. Awesome. Done?

Nope. As the readings pointed out, there could still be a bug. To prove this, we will purposely insert a bug into the Stack project and show this can happen even with 100% coverage.

Program with a bug? What kind of 100% is this!?

Emma cannot read between the lines. It cannot determine what is a good test. It only reveals what lines have been used in the test.

As hinted in class, if the Stack's list were a static size, then an error like IndexOutofBounds (or something similar) should happen if the number of objects pushed on exceeds the maximum length (limit) of the stack. Emma would never be able to report such a case because it doesn't check the boundaries. It only checks the test cases given by the programmer and if the line was ever executed in its respective class.

Unfortunately, I could not figure out how to create a list object with a static size. The list would dynamically extend itself and make room for more objects even when I set an initial size for the object. The only way I know how to resolve this is to change the list to a simple array. However, I avoided that since the Stack's entire project manipulated that very list. To change that would mean to change practically everything.

So instead, I implemented a test that would push on a very large number of objects onto the stack. The point was to exceed whatever the Stack limit is and throw an error. Since I didn't know this limit's exact amount, I started with base 10 and increased it exponentially to determine the limit. My test would throw an OutOfMemoryError after 10,000,000 pushes.

How Do We Know if the Test is Good?

So there are still bugs to discover even after achieving 100% coverage. It's scary because how does one know when they are truly finished and left with a good test? I have no clue. Is testing ever really absolute 100% perfect. I doubt it. It's probably better to let other hackers try to crash your system if you have that luxury. I didn't, but I do have other blogs to check out since we're all working on the same assignment. Some things are just more apparent to others. But the more scenarios and equivalence classes tested, the more robust the test and program will be.

Emma Testing, 100% Good or Bad?

100% on a test! Sweet, that means my test (download)was awesome!

Unfortunately that isn't always the case when it comes to testing programs. Our newly introduced friend Emma (who is not a hot import model sadly) is a useful tool to check test coverage in programs. I've grown accustomed to thinking 100% is considered the perfect mark. No changes needed. Can't be any better. But Emma changed that.

Emma: The Flirt

100% is Emma's way of flattering you. It's a semi-victory message. On one hand, our test cases have reached every single line of code. But the other hand, just because I was able to reach every single line, it doesn't necessarily mean I exploited every single crack in the code.

When I was attempting to achieve 100%, my main mindset was not to exploit all possible errors. I just wanted to quickly run a test that would incorporate all the missing lines. My tests were moderate at best and were not as complete as it could possibly be. In the readings about equivalence partitions, it mentioned that valid tests fields include those that are valid and invalid. I've tested possibly the more obvious ones in a Stack. Pushing and popping, popping and empty stack, and retrieving the top of an empty & nonempty stack.

A Little Voice In My Head

Okay, so it's not exactly a voice in my head. But, one invalid test never crossed my mind until Dr. Johnson brought it up in class on Monday. And that's the boundaries of a stack/list (the upper end). I tested the negative (zero) end by popping an empty stack. But I never tested the maximum end.

I tried looking into the max size of a list, but it appears dynamic in terms of size. The list grows regardless if the programmer sets the initial size. It is likely there is some unknown limit that could result in an error. Maybe popping 1,000,000 objects could do it. I wouldn't know. I didn't test it.

I would say that improving line coverage improves the system slightly. It can verify methods are working properly by testing output. But it does not reveal everything. The fact of the matter is a bug can still exist if Emma reports 100%. 100% coverage does not necessarily mean it's a good test. But good tests should have 100% coverage plus robustness (and speed). A good tests looks beyond the surface and reads between the lines so to speak.

Ant Lab

00. Purpose:

After being introduced to Ant and the other tools available that help check for potential bugs and non-standardized formatting, we apply these new tools toward our previous assignment, CodeRuler, and see how we measure up to the standards we have set in this course. We are to run CheckStyle, FindBugs, and PMD.

I will be focusing on messages mainly on our written code, MyRuler.java, and not too much on IBM’s supplied classes or methods. Their standards are different from the ones we set so it is unfair to judge theirs.

01. Nervous Going In

I have to admit I am a bit nervous going into this assignment still. Unlike the Stack assignment, my code is about to analyzed by a machine and I am anticipating a lot of lines of “use incorrect (xxx)” or “missing that”. I am the farthest thing from the greatest hacker in the world. Hopefully I learn from the mistakes and bad habits and all of these tools (Checkstyle, PMD, FindBugs, etc) will present less findings in the future. Until then, I’ll get my ass handed to me by the machine.

02. Checkstyle Findings

Checkstyle found many of the errors my reviewer, Philip, found in my CodeRuler program. There were about ten errors in all, several of the same type. Below is a list of what checkstyle returned (the bold are those not indicated in my review):

[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:3:8: Unused import - java.util.Iterator.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:5: Using the '.*' form of import should be avoided -
er.*.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:11: Type Javadoc comment is missing an @author tag.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:23:3: Missing a Javadoc comment.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:58:3: Missing a Javadoc comment.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:102: 'else' construct must use '{}'s.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:176: 'if' construct must use '{}'s.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:182: 'if' construct must use '{}'s.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:191: 'if' construct must use '{}'s.
[checkstyle] C:\eclipse\workspace\Game\src\MyRuler.java:193: 'else' construct must use '{}'s.

One of the new errors Checkstyle found was the curly braces {} not being formatted properly or worst yet, non-existent. Another error is the missing javadocs for IBM’s supplied methods. These reported messages points out all the areas we were too lazy to edit and overlooked because those methods were given by IBM. (Note: I should run and hide in shame. )

02a. CheckStyle - If IBM were an ICS 413 student

Holy smokes, IBM’s other classes turned up a lot of messages though. Thank god we aren’t responsible for fixing those… are we? If they were in our class, FAIL! Haha. Well... not likely, but coding standards are very different.

03. PMD Findings…. YIKES!

[echo] PMD found 36 problem(s).

That’s what stared at me when I ran PMD. Luckily I am only responsible for ten of those. A couple interesting points brought up here. It recommended for if-else statements, we should try to present our passing cases first rather than failing cases. It was called ConfusingTernary:

In an "if" expression with an "else" clause, avoid negation in the test. For example, rephrase: if (x != y) diff(); else same(); as: if (x == y) same(); else diff(); Most "if (x != y)" cases without an "else" are often return cases, so consistent use of this rule makes the code easier to read. Also, this resolves trivial ordering problems, such as "does the error case go first?" or "does the common case go first?".

03a. PMD - If IBM were an ICS 413 student

26 problems were not mine. That's all I'm going to say.

04. FindBugs…. Impressive

DB_DUPLICATE_BRANCHES: Method uses the same code for two branches

I knew FindBugs looks for potential code that could cause code to crash. So I was quite shocked to see this. This “bug” seems to focus more on enhancing code rather than potential problems. Anyways, it was useful and I was able to reduce the if statements so there was no duplicate code for the two branches and instead used simply one.

05. Conclusions: Automated vs. Manual Quality Assurance

I think Checkstyle has its limits in some aspects. Philip made some good points where he pointed out areas of redundancy in comments. It was something Checkstyle couldn’t identify because it does not interpret the content of the comments, at least I think so. It only checks if comments exist and if they are in proper format. So I definitely must thank Philip for his efforts for going above and beyond just looking at syntax. That is not saying I am ungrateful for tools such as CheckStyle, PMD, and FindBugs.

FindBugs and PMD are useful for finding really technical issues that perhaps beginner/intermediate programmers are likely to mess up on or not notice. Issues like using the correct modifiers (i.e. final, static) and object types. Concepts that I do not have full grasp of yet, I can rely on FindBugs and PMD to point out for now. I intend to learn more about these as they arise.

Also, I think I have gotten used to the fact that I am bound to code with errors. I am just more concentrated at correcting what Ant presents to me.

Stack

00. Purpose

To become familiar with the tools Ant, JUnit, PMD, FindBugs, and CheckStyle and apply it towards the Stack project found on the ICS 413 Schedule page. Inside the Stack project are errors that were purposely left unfixed and our job is to use the tools to find and eliminate them.

File Download

My completed distribution zip file can be downloaded here: stack-johnczly-6.0.925

01. Expectations: Trying to Say No to Fear

In regards to the "no fear" attitude, I wasn't exactly on board from the beginning of this assignment. I was actually a bit derailed and a bit concerned working with the Ant XML files. I thought we would need to create those ourselves, which struck some fear into me. I can understand most of the lines in the XML files but if I had to write one myself right now, I have a fair amount (or a lot... depending on your definition) of doubts and fear. But once I read the assignment, a lot of that went away and I got going at a decent pace.

02. Fixing Stack using Ant, JUnit, Checkstyle, FindBugs, and PMD

Majority of the problems were quick fixes (e.g. setting final to variables, fixing curly brackets, and javadocs, etc). There were two brought up by PMD that caused me to scratch my head. The (1) empty catch block and (2) the LooseCoupling: avoid using implementation types and use interface instead.

First was the empty catch block. I must admit I would have done the same thing Ronn had done. Never having any program in production at a professional level, I would have put a System.out.println() for two reasons.

(1) For the sake of putting something inside the empty statement, making the message go away.
(2) Because there is a fail message, it would make sense to have a pass message as well. By the way, it was awkward to test FOR a failure rather to test for no errors.

So I learned something new. Instead of purposely trying to catch an exception and throwing fail cases, we can use the new annotation in JUnit 4. I won’t attempt to summarize IBM’s explanation to avoid screwing it up. So quoted directly from IBM’s page,

In JUnit 4, you can now write the code that throws the exception and use an annotation to declare that the exception is expected:

@Test(expected=ArithmeticException.class) 
  public void divideByZero() {
    int n = 2 / 0;
}

If the exception isn't thrown (or a different exception is thrown), the test will fail.

The second problem I ran into was labeled LooseCoupling. I was initially confused by LooseCoupling. I did not understand what they meant by using the interface instead of implementation types such as ArrayList. It took me awhile. I googled different things and finally came across a post about Interface Programming. It recommended using List (interface) instead of ArrayList, making it

List<Object>

Now the example provided in PMD makes so much more sense. It's one of those times where I look back and go, "D'OH!" In my opinion, they should have put the correct example and a incorrect example rather than just putting it in comments. Maybe I am just tired. Although I fixed the problem, I am unsure on why this is a better case. Hopefully someone can answer that question for me. I am guessing it is because it's more general, and the users can interchange if needed between LinkedList and ArrayList to utilize the different characteristics.

NOTE: It's weird however after eliminating the last problem found by pmd, the echo message didn't set a value for pmd.failure.count so the message I received when running ant -f pmd.build.xml resulted in

[echo] PMD found ${pmd.failure.count} problem(s).

I assume that is the same meaning as PMD found 0 problem(s).


03. Conclusion: Veni, Vidi, Vici

I was able to complete all the tasks and fixed all the errors present in the stack project. I would not have spotted a lot of the problems without the help of the these tools. Ant and all the other tools are tremendously useful. I feel much more comfortable using the tools and learning how to interpret the messages.

At the end of this assignment, most of the fear has gone away. I guess once I got going, it just didn't seem so scary.

Reminder to self:
(1) The plugins folder that FindBugs looks for when invoked does not exist. Do I need one?

CodeRuler Review

Purpose:

Last week, groups were formed to write code for the IBM plugin game CodeRuler. Now that the fun is over, the task this week is to analyze the code formatting and see if it follows the coding standards of Elements of Java Style and our ICS supplemental standards. I was assigned to review Raqueno-Reeves's CodeRuler project. Their source code can be found at the following link: Raqueno-Reeves CodeRuler.

JavaDocs:

The JavaDocs are well written. The intro or description to the class was detailed and useful. It covered everything from purpose of class, strategy, and citation. Each method began with a good concise sentence that summarizes the function of the method.

My only complaint is when describing some of the parameters, perhaps there could be a more detailed description. Some parameter description seem redundant (e.g. xPos - x position). Perhaps a more useful description would be "the current x-coordinate of peasant/knight/etc". I just noticed some of the javadocs did not provide a description on the return value of certain methods. Considering the amount of detail in other areas, I think it was just a minor error that was probably overlooked.

Structure & Logic:

Everything looked good. At first, I was sort of wondering on the method movePeasants. It is quite short which made me wonder if it deserved its own method. But looking at it again, there is a lot going on in the setDir that is worth documenting as a separate entity. Creating the method movePeasants also keeps commands looking parallel in orderSubjects. It would have been weird to have seen....

makeUnits();
for(...){
  move...
}
moveKnights();

Formatting:

File	Lines	Violation	Comments
MyRuler.java	52,187,214,*	ICS-SE-Eclipse-2	Code exceeds 100 characters
MyRuler.java	94,231,236,*	EJS-9	Variable names represented by single char

I like the detail. I personally don't know how many one-line comments is too much. But I like the amount of information given. I thought the comments were helpful and the formatting is readable. Although I didn't look too much in detail, I think the code can be manageable.

Conclusion:

Well first on their strategy. Dang! This thing captures land pretty fast! Its initial knights don't normally last very long. But the amount of land claimed makes production of backup knights overwhelm the opponent. Back to the formatting, I am still fairly new to the standards introduced this semester. I may be misjudging at certain things. Considering it's only been a week since these concepts have been introduced, I thought Robin and Ron did a really good job formatting their code.

CodeRuler

Introduction:

CodeRuler is IBM's fantasy ruler simulation game which runs as a plugin to Eclipse ID. The goal is to come up and implement a strategy to not only stay alive but dominate the competition. As the player, we write the Java code for our ruler and try to beat out the rest.

In addition to writing the Java code to our ruler, this is also our first time working with another classmate as we continue to familiarize ourselves with Eclipse and hopefully have some fun.

Partners:

Yasuyuki Kaneshige
John Ly

Download Link:

CodeRuler (Kaneshige-Ly)

Evaluation Runs:

	yasuyukikaneshige -johnczly vs Gang Up Ruler	yasuyukikaneshige -johnczly vs Migrate Ruler	yasuyukikaneshige -johnczly vs Split Up Ruler
Test 1	634 – 36	477 – 5	444 – 48
Test 2	536 – 119	656 – 0	571 – 77
Test 3	549 -54	565 – 10	556 – 108

My strategy was simple. I had my knights attack the closest enemies. As simple as that was, it worked for well for 1-on-1 battles against the sample rulers. My peasants running around the border was an okay strategy but definitely not the best. However, it did claim land at an above average pace.

Of the three, the hardest to face was Split Up Ruler because it would divide up their knights. However, it is still predictable in the way that they still target peasants first. With my peasants running around the border of the map, it was fairly easy to defeat the knights and keep my castle.

Lessons Learned:

The one on one battles and one versus two battles were simple after a couple of runs. After observing one or two rounds I noticed their first target is the peasants. So adjusting my strategy accordingly, I won. However, four ruler (or more) battles, however, are difficult.

I wanted to implement something similar to this code my partner Yasu found. Peasants would find any unowned land and claim it. The knights also had a smart system of attacking. If attack level is low, they retreat and target peasants to replenish their strength. It was amazing to see but incredibly hard to implement. I just couldn't quite get my version to work in time so I had to revert back to my original strategy.

Coding-wise, I often found myself trying to clean up my code and trying to comment things properly for my partner to hopefully understand it. I usually do not have readibility as my primary focus but I felt it should be in this case. I guess this is because most of the code up to this point have been one-and-done. I would turn it in for an assignment and never look at it again. I think I need to get used to commenting and documenting so external developers can understand as well as myself if I ever need to look back. My audience has grown and so should my programming style.

Working with a partner is good and bad. There's always the pressure to do well because you don't want to see your partner suffer the consequences. But there's also lot of good that comes with the territory. Yasu helped come up with ideas that I probably would have never thought of. It's like the saying goes, "two heads are better than one."

Although my code didn't come out the way I wanted, our code still faired decent against the three other rulers.