I’ve been studying unit testing for a few months, and trying to practice a more TDD approach to writing code.  The most influential book that inspired this is Roy Osherove’s The Art of Unit Testing.  It’s a great introduction to the topic of unit testing and serves as a fantastic stepping stone to more advanced discussions on the topic.

One of the things that Osherove warns against is multiple asserts in unit tests.  That is, one should generally avoid writing tests that can fail for more than one reason:

[Fact()]
public void Test_Some_Condition()
{
    Foo() sut = new Foo();
    bool[] expected = { true, false };

    bool[] actual = sut.GetSomeResult(SomeInput());

    Assert.Equals(2, actual.Length);
    Assert.Equals(expected[0], actual[0]);
    Assert.Equals(expected[1], actual[1]);
}

I agree with this sentiment.  Sticking to one assert per test tends to make it easier to figure out what is wrong when a test fails.  If you have multiple asserts, the first one to fail tends to end the test; perhaps 2 or more assertions would have failed but you only get information about one.  (My example is poor for this: if Length is incorrect then clearly the rest of the assertions will fail.)  I’m not really sure I like any of the solutions to this problem.  I’ll look at a few solutions and explain why.

Messages with the asserts

This solution involves attaching string messages to the assertions to make it clear which assertion failed for what reason.  This can help if you have several of the same kind of assertion and the values don’t make it clear which one fails.  I don’t like this technique for a personal reason: I use XUnit .NET and it only allows string messages on its true/false assertions:

[Fact()]
public void Test_Some_Condition()
{
    // ...

    Assert.True(
        2 == actual.Length, 
        string.Format("The length should have been {0}, but was {1} instead.",
            2,
            actual.Length
            )
        );
    // ...
}

This looks atrocious and is difficult to write.  It’s easier in test frameworks like NUnit that provide the string message as part of the assertion.  It’s also tedious to write multiple failure messages; I’d rather write “Arrays were not equal” than multiple individual strings, particularly when dealing with larger arrays that will require string interpolation to give me expected/actual values.

Custom Assert Methods

This is an idea I picked up from XUnit Test Patterns.  Generally, these groups of multiple assertions tend to represent one logical assertion, so you wrap these multiple assertions in a single custom assertion method:

[Fact()]
public void Test_Some_Condition()
{
    // ...

    AssertArraysAreEqual(expected, actual);
}

private void AssertArraysAreEqual(bool[] expected, bool[] actual)
{
    Assert.Equals(expected.Length, actual.Length);
    Assert.Equals(expected[0], actual[0]);
    Assert.Equals(expected[1], actual[1]);
}

This doesn’t address the problems that arise when the order of the assertions hides that multiple failures are happening.  It also doesn’t shed light on the fact that the custom assertion exists; “Assert.Equals failed, Expected: 0 Actual: 1” is not as helpful here as “Arrays were not equal.”  Adding strings to the messages can help a little bit, but I’ve already noted why this isn’t a universally convenient solution.

More Verbose Custom Assert Methods

This is something I tried recently and it burns like fire.  Instead of making multiple asserts inside of the custom assert, I make multiple tests and build a string that describes what tests fail:

private void AssertArraysAreEqual(bool[] expected, bool[] actual)
{
    bool lengthsAreSame = (expected.Length == actual.Length);
    bool elementsAreSame = true;
    for (int i = 0; i < elementsAreSame.Length; i++)
    {
        if (expected[i] != actual[i])
        {
            elementsAreSame = false;
            break;
        }
    }

    if (!lengthsAreSame)
    {
        Assert.Fail("Array lengths are not equal.");
    }
    else if (!elementsAreSame)
    {
        Assert.Fail("Some elements held unexpected values.");
    }

    Assert.Pass();
}

If that doesn’t make your eyes hurt, you aren’t looking hard enough.  It’s big.  It has logic that should probably require its own tests to verify.  It was a pain in the butt to write.  (And what makes it worse is the code I’m testing that spawned this post actually works with 2D rectangular and jagged arrays.)  Note that XUnit .NET doesn’t have an Assert.Fail() equivalent, so I’d really end up using something nasty like “Assert.False(true, “…”)”.  I’m starting to question why I’m using it again.

Multiple Tests

On the “pain of implementation” meter this is pretty high.  To implement this solution, you create an individual test for each assertion:

[Fact()]
public void Foo_GetSomeResult_ValidInput_CorrectLength()
{
    // setup code

    Assert.Equals(2, actual.Length);
}

[Fact()]
public void Foo_GetSomeResult_ValidInput_CorrectElements()
{
    // setup code

    Assert.Equals(expected[0], actual[0]);
    Assert.Equals(expected[1], actual[1]);
}

That doesn’t even fully solve the problem!  Note that “CorrectElements” had to make multiple asserts, and without adding a custom string it’s hard to figure out which elements are wrong if that information is needed.  The setup logic is heavily duplicated; this tends to lead to the need to implement setup methods, which can cloud the scenario under test (particualrly if you use your test framework’s automatic setup/teardown mechanism.)

Am I interpreting a guideline as a rule?

I’m not sure I need to solve the problem every time I see it.  Since all of the solutions I know about have some tradeoffs I don’t want to deal with, perhaps sometimes it’s best to just live with multiple asserts?

I’m not happy with that answer.  I made this post because of some “do something fancy with multi-dimensional arrays” code I’m testing.  I decided to live with multiple asserts at first.  Then, I had a couple of tests fail where the diagnosis would have been easier had I known that more than one assertion was failing.  I tried moving to the verbose custom assert technique outlined above, but the custom assert ended up as a > 30-line behemoth that feels many different flavors of wrong.

Maybe I should just live with multiple asserts in this case, since the techniques to avoid them would cause more problems than they solve?