By John Hughes
Testing is typically something most people do not really like to do. Testing has this reputation of not being very fun. Why is testing hard? For n single features, there are O(n) test cases. However, you need to combine multiple features to find real bugs. So, how about k features combined, you have to write O(n^k) tests. So, this does not scale. Border line: don’t write test, but generate them! That is the idea of QuickCheck. It is a random testing tool: given an API under test, random tests are generated, until a test fails – given a (formal) model of the specification. From the random tests, it bundles and shrinks the tests that failed and presents this to the user. As a model they use state machine as their model. Important lessons: the same properties can find many different bugs, and minimal failing tests help to find bugs.
Example 1: for a large car manufacturer, they found 200 problems, of which 100 problems in the standard. And they tested 1M LOC from a spec (the standard) of about 3000 pages (resulting in a model of 20k lines)
Example 2: for a key-value store, bugs turned out to be in racing conditions. By random tests and shrinking one finds the minimal test, and that really helped finding the bugs: random tests “outsmarted” smart people staring at a production environment…
Message: do not write tests, generate them!