I’ve been a fan of testability for a few years now. I can’t actually remember where I first read about it (probably BBST or RST) but it was about 6 years ago, way before it became cool in testing circles.
I think one of the next things we could do with talking about more is test coverage, or as I’ll describe later, risk coverage. And by test coverage I don’t mean code coverage.
Code Coverage versus Test Coverage
A quick Google provides a lot of results that describe code and test coverage as the same thing.
Code coverage is roughly the amount of code that has been covered by some tests. When we talk about test coverage it tends to be a bit more vague, although still mostly based on how many tests have been executed against different parts of the system.
Code coverage is useful to see what parts of the code haven’t been tested or maybe as a low level measure of quality but I’m not sure it’s useful for the purposes of testing, other than unit testing or test driven development maybe.
Test Coverage, I think, is a different thing, but ends up being too loose and vague.
How many tests have you run? How many of the regression tests did you run? Have you tested “everything”? These are the sorts of questions you might get, but providing a good response can be tricky. Maybe a graph of the number of tests against parts of the system? We’ve all done it.
Just providing a percentage or number of tests as a basis for test coverage lacks any contextual meaning and doesn’t relate to risk or quality, that should drive our testing.
Test Coverage and Risk
If we say that testing is basically the quest of finding information about risks and quality, how does code coverage help? Code coverage to me seems to be useful sometimes (as described about), but that’s not really about risk and quality anyway.
It feels like Test Coverage should directly relate to risk. So how do we express the degree to which our tests address risk, given that part of what we are doing is uncovering risks not identified yet, so the scope of our testing is never fixed (our shouldn’t be!)?
One day during a project we may have a handle on the risks and our tests to the degree we can confidently express how effective our testing is and what the quality looks like, but the next day that might change. Stakeholders rarely like moving targets!
A better way of talking about test coverage
Let’s face it, people want to see graphs about percentage compete so they know how much longer testing is going to take, not to form a better opinion about quality.
A good start to have a better conversation about test coverage is to stop using numbers. Numbers give an inaccurate representation of the complexity and subtlety we are generally dealing with.
RAG statuses can work well. It takes away the misrepresentation that numbers often leads to. They’re also less about how complete something is and more about the current status, which allows test scope to change as the risk profile changes. Even better, split out the status by risk category / test type so there’s a status for usability, performance etc.
If you must include some numbers, use something that isn’t about effort but reflects risks and quality. Bugs is a reasonable start, although be careful not to just provide a count without considering severity and impact.
Have a conversation. Maybe ditch the reports altogether and just have a chat.
A different approach to Test Coverage – Risk Coverage
While writing this the name Risk Coverage cropped into my head. I like it more than test coverage, although it’s not perfect. It expresses what we are actually trying to do a little more.
So rather than trying to express the status and our progress during testing using test coverage, why not just talk about risk coverage?
Risk Coverage to me is the degree to which we are covering risks in our testing. If we uncover new risks we evolve our testing and the coverage changes. The code coverage could be more, or less, or the same as it was before.
If you are actually tracking risks and taking to stakeholders, presenting a report or table that describes those risks and the associated testing should be easy. If not, maybe you should be?
Any thoughts about how we can start to use Risk Coverage in our discussions about testing rather than Test Coverage?