Testing your code — test independence, avoiding test clash, and learning from it

This is a short story about a test I couldn’t get to pass and what it taught me about 3 things:

1. Running and stepping in to your tests
2. Writing your tests for independence
3. Ensuring consistency of intent across your code

The example happens to be in Ruby, but the lessons are applicable elsewhere.

The problem

It was the last test in an exercise writing custom relationship methods and I couldn’t get it to pass.

The method was supposed to find and associate an Artist with a Song, and it was when I tried it out myself manually, but the tests weren’t happy — the wrong artist object was being returned.

expected: #<Artist:0x007ffc482d0018 @name="King of Pop", @songs=[]>
     got: #<Artist:0x007ffc490ff378 @name="King of Pop", @songs=[]>

Hmm — we appear to have two Kings of Pop, and the method is returning the wrong one! What gives?

1. Running and stepping in to your tests

Running the tests told me what was wrong with the output, but not what was going on in the background.

Tests create and manipulate objects and variables which are lost in the ether as soon as the test is over, leaving only the final result — but rather than guessing what is going on with them, get stuck in and find out!

Once armed with the output of a failing test, use something that allows you to debug in the middle of the relevant method (e.g. in Ruby with byebug or pry), to check the value of objects, variables and collections as the test sees them, and execute code in that exact environment to check what works.

By doing so I was able to learn more details which pointed me to the cause of the problem — an earlier group of tests which had also created a King of Pop.

If you have multiple test files, run them independently before running them all as a collection. Limit the number of variables you are dealing with.

If groups of test pass independently but fail as part of a wider suite, you have a problem — you could solve this by re-writing the tests (see 2), but it’s also worth considering what it reveals about your program’s functionality (see 3).

2. Writing your tests for independence

If your tests are meant to be independent of each other, you could rely on not re-using variable names and keeping track of everything in memory, but you’re safer resetting the state of any collections, objects, and anything else shared across tests, before each test (or before each group as needed).

In the case of testing with RSpec in Ruby, you can use its before hooks to wipe collections or initialize objects before each example — or before larger groups of examples if you know you want values to persist across certain tests:

• before :each or before :example to execute a block of code before each example, i.e. before every it "does a thing" do block
• before :all or before :context to execute a block of code before each group of tests, i.e. before each describe "a group of things" do block
• before :suite to execute a block of code before everything.

You are better off using before :each unless you know you want something to persist through multiple tests. The official documentation is here.

3. Ensuring consistency of intent across your code

If you do find that the interaction between tests is causing unexpected results, before you jump to re-writing the tests, it’s worth asking yourself: should this result be allowed to happen at all?

In my example, the tests failed because there were two Artist objects created with the same name. But elsewhere in the code, I had a custom constructor which was supposed to prevent this exact thing from happening:

The tests were circumventing this protection by creating an Artist using the standard #new initializer directly.

This revealed that an intent of my code — ‘no two artists should share the same name’ — was being undermined, and was not robust.

While I could edit the tests to avoid it happening in this specific context, it was still open to happening again, whether by my own hand or someone else’s.

It was therefore better to take advantage of the fact that the clash between tests had revealed it was even possible, and fix the vulnerability, by preventing name duplication in the default initializer:

Thus the final lesson: if a test is behaving unexpectedly, your program has the potential to behave unexpectedly — and you should put a stop to it.