Is Your Unit Test Isolated?

(Kudos to the great software guru Jeff Foxworthy for the card phrasing.)

An effective unit test should follow the FIRST prescriptions in order to verify a small piece of code logic (aka “unit”). But what exactly does it mean for a unit test to be I for Isolated? Simply put, an isolated test has only a single reason to fail.

If you see these symptoms, you may have an isolation problem:

Can't run concurrently with any other. If your test can’t run at the same time as another, then they share a runtime environment. This occurs most often when your test uses global, static, or external data.

A quick fix: Find code that uses shared data and extract it to a function that can replaced with a test double. In some cases, doing so might be a stopgap measure suggesting the need for redesign.

Relies on any other test in any way. Should you reuse the context created by another test? For example, your unit test could assume a first test added an object into the system (a “generous leftover”). Creating test inter-dependencies is a recipe for massive headaches, however. Failing tests will trigger wasteful efforts to track down the problem source. Your time to understand what’s going on in any given test will also increase.

Unit tests should assume a clean slate and re-create their own context, never depending on an order of execution. Common context creation can be factored to setup or a helper method (which can then be more easily test-doubled if necessary). You might use your test framework's randomizer mode (e.g. googletest’s --gtest_shuffle) to pinpoint tests that either deliberately or accidentally depend on leftovers.

You might counter that having to re-execute the common setup twice is wasteful, and will slow your test run. Our independent unit tests are ultra-fast, however, and so this is never a real problem. See the next bullet.

Relies on any external service. Your test may rely upon a database, a web service, a shared file system, a hardware component, or a human being who is expected to operate a simulator or UI element. Of these, the reliance on a human is the most troublesome.

SSDD (same solution different day): Extract methods that interact with the external system, perhaps into a new class, and mock it.

Requires a special environment. “It worked on my machine!” A Local Hero arises when you write tests for a specific environment, and is a sub-case of Relies on any external service. Usually you uncover a Local Hero the first time you commit your code and it fails during the CI build or on your neighbor’s dev box.

The problem is often a file or system setting, but you can also create problems with local configuration or database schema changes. Once the problem arises, it’s usually not too hard to diagnose on the machine where the test fails.

There are two basic mitigation strategies:

1. Check in more often, which might help surface the problem sooner
2. Periodically wipe out and reinstall (“pave”) your development environment

Can’t tell you why it fails. A fragile test has several ways it might fail, in which case it is hard to make it produce a meaningful error message. Good tests are highly communicative and terse. By looking at the name of the test class, the name of the method, and the test output, you should know what the problem is:

CSVFileHandling.ShouldToleratedEmbeddedQuotes -
   Expected "Isn't that grand" but result was "Isn"

You shouldn't normally need to dig through setup code, or worse, production code, to determine why your test failed.

The more of the SUT exercised by your test, the more reasons that code can fail and the harder it is to craft a meaningful message. Try focusing your test on a smaller part of the system. Ask yourself “what am I really trying to test here?”


Your test might be failing because it made a bad assumption. A precondition assertion might be prudent if you are at all uncertain of your test’s current context.


Mocks indirect collaborators. If you are testing public behavior exposed by object A, and object A interacts with collaborator B, you should only be defining test doubles for B. If the tests for A involve stubbing of B’s collaborators, however, you’re entering into mock hell.

Mocks violate encapsulation in a sense, potentially creating tight coupling with implementation details. Implementation detail changes for B shouldn’t break your tests, but they will if your test involves test doubles for B’s collaborators.

Your unit test should require few test doubles and very little preliminary setup. If setup becomes elaborate or fragile, it’s a sign you should split your code into smaller testable units. For a small testable unit, zero or one test doubles should suffice.



In summary, unit tests--which we get most effectively by practicing TDD--are easier to write and maintain the more they are isolated.

Test Abstraction Smells

In our article Test Abstraction: Eight Techniques to Improve Your Tests (published by PragPub), we whittled down a convoluted, messy test into a few concise and expressive test methods, using this set of smells as a guide. We improved the abstraction of this messy test by emphasizing its key parts and de-emphasizing its irrelevant details.

Stripped down, the "goodness" of a test is largely a matter of how quickly these questions can be answered:

• What's the point of this test?
• Why is the assertion expecting this particular answer?
• Why did this test just fail?

That's a nice short-form summary, but it is descriptive rather than prescriptive. In the Pragmatic Bookshelf article, a real-world code example was used to painstakingly demonstrate the techniques used to improve the clarity of tests by increasing their level of abstraction. Here we provide the "cheat sheet" version:

• Unnecessary test code. Eliminate test constructs that clutter the flow of your tests without imparting relevant meaning. Most of the time, "not null" assertions are extraneous. Similarly, eliminate try/catch blocks from your tests (in all but negative-case tests themselves).
• Missing abstractions. Are you exposing several lines of implementation detail to express test set-up (or verification) that represents a single concept? Think in terms of reading a test: "Ok, it's adding a new student to the system. Fine. Then it's creating an empty list, and then adding A to that list, then B, and ..., and then that list is then used to compare against the expected grades for the student." The list construction is ugly exposed detail. Reconstruct the code so that your reader, in one glance, can digest a single line that says "ensure the student's expected grades are A, A, B, B, and C."
• Irrelevant data. "Say, why does that test pass the value '2' to the SUT? It looks like they pass in a session object, too... does it require that or not?" Every piece of data shown in the test that bears no weight on its outcome is clutter that your reader must wade through and inspect. "Is that value of '2' the reason we get output of '42'?" Well, no, it's not. Take it out, hide it, make it more abstract! (For example, we'll at times use constant names like ARBITRARY_CHARGE_AMOUNT.)
• Bloated construction. It may take a bit of setup to get your SUT in the state needed for the test to run, but if it's not relevant to core test understanding, move the clutter out. Excessive setup is a design smell, showing that the code being tested needs rework. Don't cope with the problem by leaving extensive setup in your test, but tackle the problem by reworking the code as soon as you have reasonable test coverage. Of course, the problem of untestable code is largely eliminated through the use of TDD.
• Irrelevant details. Is every step of the test really needed? For example, suppose your operational system requires the presence of a global session object, pre-populated with a few things. You may find that you can't even execute your unit test without having it similarly populate the session object. For most tests, however, the details of populating the session object have nothing to do with the goal of the test. There are usually better ways to design the code unit, and if you can't change your design to use one of those ways, you should at least bury the irrelevant details.
• Multiple assertions. A well-abstracted unit test represents one case: "If I do this stuff, I observe that behavior." Combining several cases muddies the abstraction--which setup/execution concepts are relevant to which resulting behaviors? Which assertion represents the goal of the test case? Most tests with multiple assertions are ripe for splitting into multiple tests. Where it makes sense to keep multiple assertions in a single test, can you at least abstract its multiple assertions into a single helper method?
• Misleading organization. You can easily organize tests using AAA (Arrange-Act-Assert). Remember that once green, we re-read any given test (as a document of SUT behavior) only infrequently--either when the test unexpectedly fails or when changes need to be made. Being able to clearly separate the initial state (Arrange) from the activity being tested (Act) from the expected result (Assert) will speed the future reader (perhaps yourself in 5 months) on his way. When setup, actions, and assertions are mixed, the test will require more careful study. Spare your team members the chore of repeatedly deciphering your tests down the road--spend the time now to make them clear and obvious.
• Implicit meaning. It's not all just getting rid of stuff; abstraction requires amplifying essential test elements. Imagine a test that says "apply a fine of 30 cents on a book checked out December 7 and returned December 31." Why those dates? Why that amount? If we challenged you to tell us the rules that govern the outcome of this test, you'd likely get them wrong. A meaningful test would need to describe, in one manner or another, these relevant concepts and elements:
  
  • books are normally checked out for 21 days
  • Christmas--a date in the 21-day span following December 7--is a grace day (i.e. no fines are assessed)
  • The due date is thus December 29
  • December 31 is two days past the due date
  • Books are fined 20 cents for the first day late and 10 cents for each additional day.
  • 30c = 20c + 10c
  (Consider that each of these concepts could represent a separate unit test case, and that this detailed scenario represents more of an end-to-end test case.)
  A unit test that requires you to dig through code to discern the reasons for its outcome is wasteful.

You could easily survive without our list of test abstraction smells--and figure out on your own what to do about your tests--as long as you keep one thought in your head: "My goal is to ensure that each unit test clearly expresses its intent, and nothing else, to the next person who must understand this system behavior."

The 4 Ts of Engaging Management

Teams self-organize and self-direct, so much of the Taylorist views of what a manager can do seem to not apply at all. What is the working arrangement between an agile team and their manager?

• Time (schedule) - As a steward of the schedule, a manager needs to know where the teams' effort stands--are they behind or ahead the expected schedule? Does the release need to be delayed or de-scoped? The manager is best positioned to communicate issues and evaluate the larger impact to the organization.  When the schedule needs change, we find that your chance of success is greater the earlier the matter is communicated to management. While human nature may lead one to avoid delivering bad news, it is usually better to share problems with people who are in a position to help.
• Talent (talent pool) - A manager may bring in additional talent in the form of trainers, consultants, staff augmentation contractors, or new employees. Sometimes having an eLearning course, a visiting expert, or a short-term consultant can make a huge long-term difference in the team's technical proficiency.
  A manager may change hiring criteria to bring in the kinds of developers that will help the team to work in a more fluent and productive way.
  Likewise, a manager can remove or reassign a team member who just isn't working for the current team. These kinds of issues require coordination with HR and possibly even legal representation, but are a good use of your manager's time. See also Robert Sutton's research on maintaining a productive workplace.
• Target (direction, goals) - Agility (strictly defined) is the ability to change direction gracefully. Sometimes a change of technology or target audience can greatly improve a product's chance of being accepted in the market. Additionally, agile teams would rather face failures early when they're inexpensive (fail fast) than have hopeless projects run on until resources are exhausted. Any a significant change in direction, whether cost-saving or value-creating, will need to involve management.
• Treasury (funding) - There are times that spending a few hundred or a few thousand dollars can make the difference between the team working fluidly versus struggling and slogging along. Is your biggest need for a distributed build product? A better build server? An alternative testing tool? A local version control server? More reliable intranet? A better library or framework? A minor hardware upgrade? An online training course?

When a problem can only be solved using one or more of the 4Ts, then it is clearly a management problem. Make use of your manager's sphere of influence to improve the team's chance of success.

Agile supports the empowered, competent team. The team should own their problems and push forward with solutions and data-based decision-making.  On the other hand, a team can hardly be called "resourceful" when their management assets go unused. Remember that the manager is a member of the team, and often managers pride themselves on problem-solving.

For problems in the gray area where some solutions are purely technical, it may be wise to involve management in triage. You would be surprised how many times teams have struggled with a work-around only to find that a manager would have been happy to solve the problem outright through his contacts and resources.

In your next retrospective, consider adding solution category called "take it to management." See if it helps your velocity over time.

Management Theater

Great managers can improve teams in meaningful ways: smoothing the workflow, selecting targets worth hitting, wisely managing budget and schedule, and working to align teams with organizational goals. We have fond memories of great managers, technical or otherwise, who led and mentored us, who helped us reach new plateaus of understanding and productivity.

We're not talking about those great managers today.

Instead, we'll discuss a particular form of management dysfunction often seen in development shops. Daniel Pink (in Drive) points out that programming shops are full of people who are motivated by the work and excited to make progress. Intrinsic motivation tends to be quite high, though exceptions exist (see Esther Derby's Stop Demotivating Me). Most shops face problems with procedure, organization, technological limits, overly ambitious schedules, and shortage of knowledge or practice. Less astute managers don't understand the problems in their teams, and misinterpret these as motivational issues. When the problem is technical, it does not help to attempt solving it through motivation.

You've probably been a witness to most of these. Just in case they're not obvious:

• Begging: "Please, I just really need you to work harder to get this done. Just this one time."
• Browbeating: "Stop your whining and get it done before I replace you with someone who gives a darn about their job!"
• Cheerleading: "I have faith in you, you're the best! Go Team!"
• Punative Reporting: "I want to see daily status reports! Twice daily!"
• Publicity Stunts: "I want every last one of us in this meeting. We need a show of force!"

Such motivational tactics tend to be ineffective. To people struggling with difficult organizational and/or technical problems, emotional appeals seem to be a kind of buffoonery. Of course, if the team succeeds despite the management theater, it merely strengthens the causal connection in the manager's mind. By simply not failing, the team locks their manager into a pattern that ensures that all future crises will be met with emotional and ineffective responses.

We should not be asking how to make managers behave. We should be asking what a team can do to ensure that a manager can provide effective servant leadership. Management theater is not a manager's first choice of action, but rather a tactic of last resource. When a manager does not have sufficient information or timely opportunity to be effective, she must use whatever ethical means remain. Management theater is, therefore, primarily a process smell not of management but of the development team.

Worldwide shortage

I'm told that there are shortages of AgileInAFlash, some channels having projected waits of as much as two months for replenishment.

You can still get AgileInAFlash physical decks at the Pragmatic Programmers site, and you can also pick up electronic copies there to tide you over. The electronic version is not as handy in mentoring settings, but is fine for reading or projecting.

Second Printing

Thanks to the demand of our readers, Agile In A Flash is going to its second printing only 8 months after its initial release. Thanks to all of you!

Mobile Version

Thanks to our hosts at blogger.com, we now have a mobile version, so accessing Agile In A Flash from your mobile device is finally easy and even pleasant.

Doctor Dan's Prescription

Dr Dan's Prescription: An Agile Flashcard a Day keeps the Fail away...

This photo of card #33 was received via twitter from Daniel Thomas (@geekdan) in Brisbane, Australia, complete with the quote above.

I like how they have a low-tech "frame" for the cards, and a place of honor on the team whiteboard. I hope they get some help, humor, or inspiration from the deck.

Dan tweets a lot of interesting links. Give him a follow.

A Card-Enriched Workspace

Friend of the blog Brian Kelly offers us a glimpse of his workspace. I notice that the books on organizational change are above the card on overcoming organizational obstinance, and that the rest are values and practices cards.

The new website-only card, Five Rules for Distributed Teams, is shown on the monitor.  It was released coincidentally with Lisa Crispin and Nanda Lankalapalli's StickyMinds article on Tele-teams. I guess the distributed team meme is in the air.

Brian is the fellow who has been posting the excellent near-daily meditations on the 52 cards of the official Agile In A Flash deck released by Pragmatic Programmers in late January of this year.

We are very proud, indeed.

Rules for Distributed Teams

As the Greatest Agile-In-A-Flash Card Wielding Coaches So Far^(tm), we're often asked for advice (or drawn into arguments) about how to make agile work with distributed teams. Sometimes it's for more humble reasons, though: We've both been remote members of a team, pair-programming with peers daily. We prescribe the following rules for distributed development:

1. Don't. Traditional organizations should avoid the extra strain, trouble, and expense of remote members without a significant reason. For example, building a better team using remote rockstars might provide some justification, but you might also be better off with a capable, local team that works well together. It's often out of your control, though, and in the hands of a really big boss, bean counter, or entrenched culture. You can make it work: Virtual organizations, startups, and the like find great success using virtual tools, pairing, and non-traditional communication pathways. Make sure you really mean it, because it's not a trouble-free add-on to the way your large organization does things now.
   
2. Don't treat remotes as if they were local. Treat your "satellite" developers as competent people with physical limitations. A remote is visually impaired, since he can only see through a web cam, and only when it is on. He cannot see the kanban board, the other side of the camera, and so on. Likewise, he only hears what is said into the microphone and not at all if simultaneous conversations occur. A remote cannot cross the room and talk to people, so interoffice chat (Skype, Jabber, etc) is essential. The local team has to make concessions, repeat conversations, and be the eyes and ears of the remote employees. Do not treat unequal things as equal--accept that there are compromises.
   
3. Don't treat locals as if they were remote. You certainly can install electronic kanban boards, online agile project management tools, instant messaging, cameras, email, and shared document management so that every local can sit alone in a cubicle or office and behave just like a remote employee. Rather than being empowered, you are all equally limited (see point #2). Never limit so that all are equal. Allow all to rise to greatness. The power of people working closely together in teams is significant (see first bullet above).
   
4. Latitude hurts, but longitude kills. Just being remote hurts (see first two bullets), but the complications can be overcome when employees share the same time zone and working hours. The further you move the team across time zones, the fewer common hours they have, and the longer any kind of communication takes to make a round-trip. Agile is predicated on short feedback loops, so 24-hour turn-around is out of the question. If you can't be a single team that works together, create separate agile teams.
   
5. Don't always be remote. Begin your engagement with a nice long on-site visit. A week is barely enough. Two weeks starts to make it work. Visiting for one week a quarter or even a week a month can keep a feeling of partnership alive. Dealing with difficulties is easy among people who know and respect each other. While constant telepresence (Skype, Twitter, IM, etc.) can minimize the problem of "out-of-sight, out of mind," studies show that distributed team success requires teams with strong interpersonal relationships, built best on face-to-face interaction. The bean counters may not be able to comprehend it, but the investment is well worth the return.

Tim: Would I remote again? In fact, I do most of my Industrial Logic work remote. We are in constant touch with each other and pair frequently. I converse with Joshua Kerievsky more in the course of a week than I have any other "boss" (he'll hate that I used that word!) in my employment history. We even work across time zones. We would work more fluently and frequently side-by-side, but we'd probably not get to work together if we all had to relocate. It is a compromise that has surprisingly good return on investment for us. We also have the advantage that we are all mature agilists; it would be very hard for first-timers. It's hard for me, as I have a tendency to "go dark" sometimes.

Jeff: I enjoyed a year of remote development with a now-defunct company one time zone to the east. Pairing saved it for me. The ability to program daily in (virtually) close quarters with another sharp someone on the team helped me keep an essential connection with them. On the flip side, however, I never felt like I was a true part of that team. I missed out on key conversations away from the camera, and I felt that debating things over the phone was intensely ineffective. You do what you must, but I'd prefer to not be remote again.