Jeff Langr and Tim Ottinger
Acceptance tests (ATs) are as enduring and important an artifact as your code. The proper design of each will minimize maintenance efforts. You'll recognize some familiar concepts--Kent Beck's rules for simple design and some classic OO principles apply well to the design of acceptance tests.
Abstract. A test is readable as a document describing system behavior. Uncle Bob's definition for abstraction applies equally well to tests: Amplify your test's essential elements, and bury its irrelevant details and clutter. Anyone, including non-DBA and non-programming staff, should be able to follow the steps taken in the test, and understand why it passes. Extracting duplicate behavior to a common place--the AT analog of programming's extract method--is the main workhorse that allows you to increase abstraction at the same time you remove duplication.
Bona fide. To ensure continual customer trust, a test must always truly exercise a system as close to production as possible. Passing acceptance tests tell the customer that what they asked for is complete and working. But once the customer has doubt as to what your tests exercise, you have severely damaged your ability to continue using them as contracts for completion.
Cohesive. A test expresses one goal accomplished by interacting with the system. Don't prematurely optimize by combining multiple scenarios into a single test. Keep single-goal tests simple by splitting common content into separate fixtures. Yes, this will mean your acceptance test suite runs more slowly, but it's far more important to avoid compromising clean test design. (It's also why your unit test suites should be as fast as possible.)
Decoupled. Each test stands on its own, not depending upon or being impacted by results of other tests. A failure caused by problems in another test can be difficult to decipher.
Expressive. A test is highly readable as documentation, not requiring research to understand. Name it according to the goal it achieves. As with unit tests, refactor each test to improve the ability of a third party to understand its intent. You should always eliminate magic numbers, replacing them with constants as appropriate. Improve visual accessibility by formatting your test using Arrange-Act-Assert (AAA). You should also make it clear what context is being set up in the test; one way is to incorporate additional assertions that act as preconditions.
Free of duplication. Eliminate duplication across tests (and even within the same test) before it eliminates you! Duplication increases risk and cost, particularly when changes to frequently-copied behavior ripple through dozens or more tests. Duplication also reduces the use of abstraction, making tests more dense and difficult to follow.
Green. Once a story is complete, its associated ATs must always pass. A failing AT should trigger a stop-the-line mentality. Don't allow your test suite to fall into disarray by allowing failures to be ignored!
Acceptance tests (ATs) are as enduring and important an artifact as your code. The proper design of each will minimize maintenance efforts. You'll recognize some familiar concepts--Kent Beck's rules for simple design and some classic OO principles apply well to the design of acceptance tests.
Abstract. A test is readable as a document describing system behavior. Uncle Bob's definition for abstraction applies equally well to tests: Amplify your test's essential elements, and bury its irrelevant details and clutter. Anyone, including non-DBA and non-programming staff, should be able to follow the steps taken in the test, and understand why it passes. Extracting duplicate behavior to a common place--the AT analog of programming's extract method--is the main workhorse that allows you to increase abstraction at the same time you remove duplication.
Bona fide. To ensure continual customer trust, a test must always truly exercise a system as close to production as possible. Passing acceptance tests tell the customer that what they asked for is complete and working. But once the customer has doubt as to what your tests exercise, you have severely damaged your ability to continue using them as contracts for completion.
Cohesive. A test expresses one goal accomplished by interacting with the system. Don't prematurely optimize by combining multiple scenarios into a single test. Keep single-goal tests simple by splitting common content into separate fixtures. Yes, this will mean your acceptance test suite runs more slowly, but it's far more important to avoid compromising clean test design. (It's also why your unit test suites should be as fast as possible.)
Decoupled. Each test stands on its own, not depending upon or being impacted by results of other tests. A failure caused by problems in another test can be difficult to decipher.
Expressive. A test is highly readable as documentation, not requiring research to understand. Name it according to the goal it achieves. As with unit tests, refactor each test to improve the ability of a third party to understand its intent. You should always eliminate magic numbers, replacing them with constants as appropriate. Improve visual accessibility by formatting your test using Arrange-Act-Assert (AAA). You should also make it clear what context is being set up in the test; one way is to incorporate additional assertions that act as preconditions.
Free of duplication. Eliminate duplication across tests (and even within the same test) before it eliminates you! Duplication increases risk and cost, particularly when changes to frequently-copied behavior ripple through dozens or more tests. Duplication also reduces the use of abstraction, making tests more dense and difficult to follow.
Green. Once a story is complete, its associated ATs must always pass. A failing AT should trigger a stop-the-line mentality. Don't allow your test suite to fall into disarray by allowing failures to be ignored!
These are nice guidelines, but I would love to see some examples of tests that demonstrate these principles, especially where you might need to make tradeoffs on one for the other. IMO this is really the crux of good tests - adapting them over time to be maintainable and work for you.
ReplyDeleteHi Chris-- Thanks for the comment, examples are a great idea but not the focus of this blog. I may put together some entries for my own blog (langrsoft.com, although I've been lazy there lately) that demonstrate examples.
ReplyDeleteAs far as needing to make tradeoffs, the bulk of ATs I've seen can meet all of the principles. The tensions I've seen are where we had to make concessions. Two typical examples: 1) inability to replicate production elements (e.g. we had to mock third-party systems); 2) combining test cases/assertions because the common setup is excessively slow (over 15 minutes in one case, with a few dozen subsequent verifications needing to run--too much time!). There are always tradeoffs, but I don't see them as being due to the other principles.
Can you provide some examples?
Jeff
Hey Jeff,
ReplyDeleteIf we take out references to Acceptance Test items such as "assertions" and such then would you say this is also relevant to Acceptance Criteria (for those orgs not yet committed to AT's)?
Dave Updike
Hi Dave--
ReplyDeleteGood question. The Bona fide and Green principles don't seem to have a lot of value when applied to criteria, not tests. I figure the rest are perfectly applicable, although 'free of duplication' may not be as desirable.
Jeff