22 Oct 2008

Automated Test Problems?
Address the Root Problem!

I've just finished a gig helping a team get to grips with automated testing on an existing project. Most focus was on creating and running Junit based developer tests.

By the end of the assignment we had nearly 1000 fast running developer tests. The tests were ran every five minutes or so by each pair, and again as part of continuous integration build on each checkin. Code coverage was moving towards 100% on many packages. The team had the essence of a design language which made discussing both existing and new work much easier. We also had over 50 automated customer tests.

Previous efforts to create a suite of automated tests had failed. Most importantly because nobody was running the tests, and there were many good reasons why not:

  • The tests are irrelevant to what I'm working on
  • The tests are always broken
  • It takes too long to run the tests
  • We are supported to check log files for test failures, I don't know what to look for
  • The tests are out of date and don't reflect the current system requirement

However, I have come to believe that these are just symptoms of the real barriers to automated testing. The diagram below shows my view of the dependancies (click to enlarge)...


These symptoms are present in most clients I visit. Projects that are having problems with automated testing exhibit a combination of:
  • poor communication,
  • poor design
  • and poor rhythm.
The great thing is that we can apply standard XP practices to address these problems. This is what we did...

To address poor team communication we introduced:
  • Collective code ownership to motivate information sharing.
  • Pair programming so that all design comes out of conversations.
  • User Stories visible to the whole team.
  • Face to face conversations to minimize assumptions.
  • Customer Tests to document and clarify specific requirement and defect cases.
  • Daily Standup to broadcast progress and issues.
  • Iterations to broadcast progress outside of the development team.
  • Retrospectives to identify and apply lessons learned from each iteration.

To address poor test design we introduced:
  • Collective Code ownership to fix issues as they arise.
  • An "all tests pass" quality gate on checkin to catch stale tests.
  • Test driven development to align the tests with the features we are developing.
  • Test coverage metrics (and daily discussion of current results) to communicate the quality of our tests.

To address poor production code design we introduced:
  • A very simple quality gate (Kent Beck's Good Code) as the basis of our design decisions.
  • Test Driven Development as an easy way to create testable production code.
  • Dependancy injection to control infrastructure dependancies.
  • Standard object oriented principals (SRP, DI, DRY, LSP, Demeter, etc).
  • Quick design sessions and UML (as a sketch) to visualize the design.
  • Pair programming for "real time" design conversations.

To address a lack of rhythm we introduced:
  • Test Driven Development to provide a individual/pair high frequency minute by minute rhythm.
  • Continuous Integration for an hourly rhythm.
  • Daily Standup for a team wide daily cycle.
  • Customer Tests to provide a moderate frequency delivery rhythm.
  • User Stories and Engineering Tasks for a multi-day team with rhythm.
  • Iterations to provide a weekly rhythm.