14 Dec 2007

Keeping Customer Tests Interesting

I've gone back using GUI driven tests for the majority of Customer Testing situations. This comes after spending a lot of time and effort introducing "under the GUI" testing tools like Fit/FitNesse into projects, only to see them fall into disuse through lack of (ongoing) interest from the Customer.

In an Agile/XP project, Customer Tests are those tests produced for (and ideally by) the customer. The defining characteristic of a Customer Test is that it is interesting to the Customer. A test that isn't interesting to the customer isn't really a customer test. So how do we make Customer Tests that are interesting to our Customers?

Traditionally Customer Tests are manual tests, using the user interface to exercise specific test cases. I find Customers like this approach because it mirrors exactly how they will use the system. Unfortunately, as the system grows and flexes, Regression Testing becomes ever more onerous, until it becomes so backbreaking that it consumes the project. At this point either milestones or (and?) quality start to slip.

Automated testing tools automate this manual task with user actions are recorded as macros. These automated tests can then be replayed without human involvement. Unfortunately these captured tests are often expressed in strange macro languages that are hard maintain, making the tests brittle.

Fit/FitNesse type tests that run "just under the GUI " allow specific features and functionality to be isolated. This helps us reduce the overall brittleness of our automated test suite. Unfortunately that same isolation of features can alienate some Customers.

For many projects, using this type of testing requires a tech savvy customer, and even then are really only suited to a small subset of customer focused tests. Without real customer involvement, I'm often better off just expressing these business tests with xUnit. As practices like Test Driven Development have matured, most problems I see these days are now integration and configuration based problems.

More recently I've been using Ruby with the Watir library and rSpec to create user interface driven customer tests:

  • These tests are scripted in Ruby, and drive functionality through GUI using the Watir object. This familiar GUI driven approach to testing keeps customer interest.
  • Whilst tests are Ruby code and not English, using rSpec gives the test scripts a “plain english", easy to read and write feel.
  • Since we are using a first class scripting language we can ensure that our script code is GoodCode. This significantly reduces the brittleness of the test scripts.

This isn’t to say that there isn't a place for Fit/FitNesse style "just under the GUI" tests. For the moment though, for me, in most cases both me and my customers are happiest with "via the GUI" testing using Ruby+Watir+rSpec.