Test SCUM


Font is "Pen of Truth"

Source is Ben Rady and Rod Coffin of Improving Works, given in an agile presentation at Agile2009.

In sharp contrast to the FIRST principles of good unit tests, Rady and Coffin give us these properties of completely bad (scummy) unit tests.

  • Slow: We discussed now important speed is in unit tests. I find that when tests are more than 40 seconds, people have to decide to run them, rather than feeling free to run them immediately after every small step. Rady's Infinitest runs tests automatically and analyzes the tests to selectively exclude those which have no (discernable) transitive dependency on the system under test, just to eek out some extra speed. While it is preferable to run all the tests all the time, in many circumstances culling the unrelated tests will give back some speed.
    Tests are generally slow because thy provide insufficient insulation from the environment. They tend to spend a lot of time waiting on clocks, file systems, databases, web services, or the like. Slow tests are best rewritten with isolation from the environment.
    How fast is fast? I find that I can tolerate a usefully large battery of tests if each of them takes about .002 seconds. I can tolerate any number that starts with a dot and two zeroes. I might be able to tolerate a very few with only one zero immediately after the dot. More than that is "slow". It doesn't take many multi-second tests before developers feel reluctance to run them.
  • Confusing: We have emphasized that good tests isolate errors and their causes. A confusing test is one that fails to isolate a specific error. This may be because it has too many things going on, or because it is simply unreadable. It is preferable that the combination of the test class name, the test method name, and the assertion gives all the information a developer needs in order to correct a misstep. In a confusing test, those three items are not enough an the entire body of the test may not shed light on the failure mode. Confusing tests are best rewritten as smaller, more clear tests.
  • Unreliable: Tests should be repeatable: they should always fail the same way, or always pass for the same reasons. Tests that run only in isolation or on certain days or times of day are not reliable tests. One of the best ways to judge reliability is to run tests repeatedly. Note that overspecification often causes tests to fail in unexpected and non-useful ways.
  • Missing: The tests that you don't run provide you no benefit, and quite a bit of harm. Tests help you to design your production code, and also help you to tell if you are making decisions that break the system in unexpected ways. Not having the tests means that you just don't know. Maybe your design is iffy in ways you hadn't noticed. Maybe you've broken other code with your last line of code. Maybe the behavior you hand-verified last hour is no longer occurring as you expected it to. The tests you did not write are the ones that are really holding back your productivity.
While we could list other unwanted qualities of tests, these seem to describe the most painful sins.

4 comments:

  1. All very true, but somehow a repetition of the FIRST card, isn't it?

    ReplyDelete
  2. Mostly, yes. "Missing" is an addition. I'm mostly including it because Rady & Coffin came up with this without having seen FIRST, and I attended their talk at Agile2009 and was stricken by similarities.

    ReplyDelete
  3. This is slowly going off the deep end! don't Agile developers have anything todo that they sit there and come up with these 'intelligent' terms? When do they get time to actually develop - debug - test and actually deploy software in the 'REAL' world? Agile is getting to be the laugh stock of the development world!

    ReplyDelete
  4. It's not like it took hours of thinking to come up with it. It's more like people noticed trends and in a burst named them. It was independent discovery, and more credible for being like FIRST.

    BTW: Ben & co build and market software, and it's good stuff. I've used it. Yeah, they really do program, and so do Jeff and I.

    If one works a sustainable pace and reflects on their work regularly (as Agile requires) then one tends to come up with observations about what works and what doesn't.

    ReplyDelete

Note: Only a member of this blog may post a comment.