Test-Driven Hypocrisy? Who tests the test?

An oft heard mantra in Test-Driven Development is "if it's not tested, it's broken" and I have to admit that this slogan makes me cringe -- and leads to some of my own hidden objections to TDD.

"If it's not tested it's broken" -- okay it's a blatant exaggeration -- yet this seems to be lost on a lot of people. What it really means is something more like:

"If it's not tested with unit tests then it's unlikely to be tested elsewhere and hence we'd be making a fairly safe bet, to assume that it contains bugs."

But somehow this has less punch than the exaggerated mantra

"If tested it ain't, broken it is."

I like exaggeration -- hell, i probably like it a thousand more times than you do -- but i get annoyed when people take exaggerations literally.

For kicks, let's apply the principle literally and see where it gets us:

Say you write some code. Oops. You should've written tests first. Your code's broke.

So now you write code to test your code.

Oops, the code you wrote to test your code is broke. Fool! You didn't write any tests to test the tests that test the code that broke.

Stack overflow. Goodnight.

Discuss.

(First -- thanks to Punky for mentioning this conundrum in the comments to the previous post)

Okay -- wait a moment -- i think i see what went wrong. Let's investigate.

In the following dialogue there are two parts:

Agl: An Agilista.
and Dvl: The Devil's Advocate.

Agl: We don't write tests to tests our unit tests -- so the 'stack overflow' doesn't occur.

Dvl: But that means your tests are broken -- by your own definition.

Agl: No it only means there's a chance that they're broken. So we work to make this chance as small as possible. Firstly, we write very simple unit tests -- the smallest things that can possibly break...

Dvl: Yeh yeh, small unit tests are less likely to contain bugs -- or likely to contain less bugs -- but you need lots of tests, thus lots of code and at least a few bugs. i say again, your unit tests are broke and the whole approach is bunk.

Agl: No, we start off with small unit tests that fail. On purpose -- we write the unit tests before we write the code. The tests fail at first. Then we write the simplest code we can to pass the unit tests. We run it again and now the tests pass. By doing that, we've gotten a hidden benefit -- we've tested our unit tests. And it's at that stage that we sometimes find our unit tests were broken.

Dvl: Nonsense -- just because the tests failed at first and then passed doesn't mean the tests were correct. It might be that there's a bug in both the test and the code being tested. I can easily write up a situation like that.

Agl: True, you can, and it's definitely possible. But it's all about probabilities. We wager that it's unlikely that a pair of bugs together will display that behaviour. And even if they do, we've reduced the likelihood of bugs by an order of magnitude. Certain combinations of bugs will slip through -- but that's a much better situation than the situation we had before, where all bugs slipped through.

Dvl: Weeeel, I'm unconvinced.

Agl: Look, just try it. Plenty of people have tried it and found that their projects are more succesful, with less bugs because of it.

Dvl: It's impossible to know that for certain unless they've performed a repeatable, placebo-controlled double-blinded experiment. And the economics of the situation dictate that no-one would ever pay for that kind of experimentation. If they have, show me the journal and i'll show you the experimental flaws.

Agl: You're just being an a**hat now. Talk to people who've used it and they'll tell you how effective it is.

Dvl: Oh I see, it's like magnets and crystals for healing. Anecdotal evidence a-plenty.

Agl: Then try it for yourself! People who try it seldon turn back.

Dvl: I've heard the same thing about heroine and nicotine. Maybe it creates a psychological dependency. Seems feasible. I wonder if being 'test-infected' would respond to OCD medication?

Agl (closing eyes, covering ears and shouting): SHUT UP! SHUT UP! SHUT UP!

lb: Okay... anyone got a better response?

My book "Choose Your First Product" is available now.

It gives you 4 easy steps to find and validate a humble product idea.

Learn more.