Testing Patterns

Now that we’ve explored the how-tos and whys, let’s take a look at how to determine what to test and how to know if it’s tested well enough. Remember, the ultimate goal of testing is to reach complete code coverage. Your tests should test every line of code at least once. However, you still need to consider the possible scenarios to ensure that you’ve written enough code! In this section, we’ll look at some common scenarios that could be tested.

User Input

For programs that take input from the user, the input that the user provides is a perfect source of errors. You can never trust the user! In these cases, you’ll want to test

• valid input
• invalid input
  • which may include input that is out of range, the wrong type (like inputting letters instead of numbers), or unusable data

Your program should work correctly with valid input. That’s a given. For invalid input, however, you need to decide if it makes sense for your program to handle the invalid input. For example, if your program has a menu, it makes sense to be able to recover if the user enters an invalid menu choice. Just let the user try again. On the other hand, if your program is supposed to perform some mathematical computation and the user enters characters, you have to decide if it makes sense to let the user try again or just let the program fail. What you don’t want to do is just push the error down the line; if the user gives you invalid input, your program should immediately handle it and get corrected input or it should fail LOUDLY. A bug based on an exception is much easier to find than a bug where your output is just wrong for some reason.

Classes

When testing a class, you should have at least one test per public method. As discussed earlier, since you can’t directly test private methods, their correctness has to be assessed indirectly. The tests you need to run depend on the method, but the considerations are similar to testing users’ input. Here are some general things to consider.

Constructors

The purpose of a constructor is to initialize the instance variables and get the object ready to use. You should test that the instance variables are set correctly and any other set up has been properly done. This often involves using the getters (accessors).

Getters and Setters

Getters and setters, also known as accessors and mutators, are how other classes access and manipulate the values of the instance variables in a properly encapsulated class. The getters are often used as part of testing the constructors, so you don’t necessarily need specific tests for the getters. It is not wrong, however, to include such tests. Setters should be tested to ensure they properly update the variables, especially if the setter validates the new value. For example, if you have a method like setAge that verifies that the new age is positive, you’ll want to ensure that the age is only changed if the new value is valid. Setters are usually tested with the help of getters.

toString

This is another method that can be used to test other methods, depending on what’s used to create the String representation. Since the String representation of an object usually involves at least some of the instance variables, it can be a quick way to verify that changes have taken effect. For example, if you have a List class that represents a list and toString includes a print out of all of the elements, it can be a quick way to verify that the elements in the list are correct and in the correct order. It is worth testing toString on its own so that if its own test passes but another test using toString doesn’t, you can rule out the bug being in toString.

You generally want to verify that toString is correct on a newly created object and after the object has been modified in such a way that the String representation would have changed.

Equality and Comparisons

While classes you write should always have constructors, getters and/or setters, and a toString, your class may or may not have methods such as equals, compareTo, or compare. If your class does have these methods, you need to test them thoroughly! These are good examples of methods where you need to consider the scenarios in which these methods will be used. Those scenarios will help guide what tests you need to write.

Consider the method equals. There are two possible outcomes of this method: the other object is equal to this object or the other object is not equal to this object. Each of these scenarios should be tested separately (i.e. in different test methods). Don’t stop there, though! If your equals method determines equality based on different instance variables, you should test that objects are or are not equal with varying combinations of equality amongst the instance variables. For example, suppose you have a Dog class that represents a dog and one Dog object is equal to another if and only if both Dogs have the same name and age. This means that a dog named Fido who is 7 years old would not be equal to a dog named Fido who is 3 years old. For an equals method like this, you would have the following tests:

• equal objects where the names and ages are exactly the same
• unequal objects where the names are different and the ages are different
• unequal objects where the names are the same, but the ages are different
• unequal objects where the ages are the same, but the names are different

Similar considerations are required for methods like compareTo. A compareTo method should return the relative ordering between this object and the parameter object. It returns something negative if this object is less than the parameter, something positive if this object is greater than the parameter, and 0 if this object and the parameter are equal. Note that if equals considers objects equal, compareTo should also consider the objects equal.

Loops

While you generally won’t just test a loop, these cases can help you determine how to test a method or program that contains a loop. The most common bugs with loops occur with respect to the stopping condition; off by one errors are quite common. These occur when boundary cases are not handled correctly. For example, if you have a loop that is supposed to run while the value of i is less than 10, 10 is the boundary condition because that’s the value that’s on the border of looping and not looping. You’d want to check that 9 loops but 10 does not. If 10 does loop but 11 doesn’t, that would be an off by one error. The basic values to consider when testing a loop are as follows.

• the initial condition
  • does the loop start on or with the correct condition
  • if it needs to, does the code do the right thing if the loop never runs?
• end condition
  • does the loop stop correctly? Check the boundary cases and off-by one cases.
• side effects of the loop
  • did the loop accomplish what it was supposed to do?

Lists

At some point in this class, you will write your own list. If you’re not there yet, feel free to come back to this section when you are ready to start writing tests for your lists.

A list is a way to organize, store, and access data. To test your list, you must first determine how the user will be able to interact with the list. Does the user have random access to the elements? Can the user add or remove from the list? Can the user update elements? Be sure that you understand the mental model of your list before you being writing tests.

Like testing a class, you will want to write tests for each public method. The tricky part about testing list methods is that you need to consider the state of the list and how that changes the behavior of the method. Here are some general guidelines.

First, you should consider the state of the list. There are three basic states: the list is empty, the list has one or two items, and the list has three or more items. A list with three elements is the smallest list that has a first, middle, and last element. For testing purposes, we consider three elements to be a sufficiently full list unless you need to test a structure that can become “full” (like an array list). If your list can become full, you need to test that it behaves correctly when it’s full.

Second, you should consider the location of the element. Again, there are three basic locations: the front, the middle, and the end. We consider all middle elements equally in the middle; that is, a middle element in a three element list should behave the same as a middle element in a ten element list. Both middle elements have an element on either side of it, so theoretically if your method works for one middle element, it ought to work for all middle elements.

Let’s look at a few examples. For these examples, we will assume we have an unordered, linear list with random access, e.g. a list. We will not assume any particular implementation. Lists will be represented using square brackets. An empty list looks like this: [ ]. A list containing a single element, A, looks like this: [ A ]. A list containing the elements A and B in that order looks like this: [ A, B ].

Furthermore, when we discuss possible list scenarios in this class, we will use the following format:

Starting list 🡪 Resulting list

Thus, the scenario that we start with a list containing the element B and end up with a list containing B and C (in that order) would be represented as:

[ B] 🡪 [ B, C ].

Add Method

When you add to a list, you expect that the element you add ends up in the list. Let’s assume that this add method adds the new element to the end of the list. Now we need to consider the possible scenarios. In how many ways could we add to a list?

Considering the possible states of the list, we could add to an empty list. We could add to a list that has one element. We could add to a list that has two elements. We could add to a list that has three elements. A list with four or more elements should behave the same as a list with three elements.

Next, consider the location of the element. The element could be in the first, middle, or last position. Remember that in this method, the element is always added to the end of the list.

How does all of this translate to test cases? We’ve demonstrated that we really only have four possible scenarios: adding to an empty list, adding to a list with one element, adding to a list with two elements, and adding to a list with three elements. Adding to an empty list covers two scenarios: the list is empty and the newly added element is the first element. Adding to a one element list also covers two scenarios: the list has one element and the newly added element is the last element. The other scenarios are covered by adding to a list with two elements and adding to a list with three elements. For this method, it is impossible to add to the list and end up with the new element in the middle, so we don’t need to worry about that scenario.

Now, we’ve always assumed that we are adding unique elements. In a general list and in this particular method, duplicate items shouldn’t cause a problem. However, it is good to keep in mind whether it matters if duplicate items exist and if they could cause problems. If it does matter, then adding a test that tests duplicated items is prudent!

Get Method

There are a number of different ways to get elements out of a list, depending on whether the list restricts access (like a queue), if the elements have a unique index (like an array), or if the elements are even expected to be in a particular order (like an unordered set). For the purposes of this exercise, let’s define the behavior of get as follows: get takes one parameter, the index of the element to retrieve.

Let’s run through the scenarios. We could try to get an element from an empty list, a list with one element, a list with two elements, or a list with three elements. We could try to get the first element, a middle element, the last element, or an invalid index. Furthermore, an invalid index could be invalid because it is negative or greater than the number of elements in the list. Since we’re getting by using an index, duplicate elements shouldn’t affect anything.

For this method, we could structure the tests to try to get all of the following indexes on each sized list:

• a negative index
• index 0
• a middle index (if one exists)
• the last index
• an index too large

The scenarios tested would be:

• Empty list
  • a negative index
  • an index too large
• One element list
  • A negative index
  • Index 0, the first (and last) element
  • Index 1, an index too large
• Two element list
  • A negative index
  • Index 0, the first element
  • Index 1, the last element
  • Index 2, an index too large
• Three element list
  • A negative index
  • Index 0, the first element
  • Index 1, a middle element
  • Index 2, the last element
  • Index 3, an index too large

It is assumed that a list with four or more elements will behave like a list with three elements, so there’s no need to test a list larger than three elements. This assumption is based on the assumption that the list was implemented using a reasonable and efficient implementation and should hold up for linear lists. That is, for the purposes of testing, lists with three, four, five, or more elements are all considered equivalent lists.

At this point, you may be concerned with how to create the most efficient tests so that you only test each scenario exactly once. In short, don’t worry. While you don’t want to run every single test multiple times, some redundancy and overlap is fine and could expose tricky bugs. For example, depending on how the list is implemented, getting the last element could be done differently for a one element vs. a three element list. Again, you want to ensure you’ve tested the possible scenarios and have tested every line of code that you wrote.

Remove Method

Just like get, there are several different ways to remove from a list. For the purposes of this exercise, let’s define the behavior of remove as follows: remove takes one parameter, the element to remove. If the element is found, that element is returned. If the element is not found, then an exception is thrown. If there are duplicate elements, the element that comes first in the list should be removed.

Similarly to how we tested get, there could be some redundancy in the tests as you’d test removing elements in the same positions in different sized lists.