Understanding Coverage

While it is common for projects to measure what percentage of their statements are executed by tests, this single number is woefully inadequate at providing a detailed understanding of the extent to which a project’s code is tested, if there are gaps in the tests, and if these tests are useful in any meaningful way. For instance, seemingly simple changes in one part of the codebase may reduce the efficacy of existing tests that seem otherwise unrelated.

Code coverage can be useful to track long-term trends in how tested a project is, but on a day-to-day basis, can’t serve as an indicator for the change in test suite quality. In particular, moving the coverage needle even 0.01% can be extremely difficult in a project with millions of lines of code. At such a large scale, focus often drifts from which lines of code are covered to simply the number of lines covered. However, a change in the coverage of several hundred critical lines of code might be important for developers to take notice of. Over time, these small changes to which lines are covered add up to form a coverage debt, and can lead to a dangerous reduction in test suite effectiveness.

We are building tools and techniques to help every developer track and manage their coverage debt, easily answering questions like: Which lines are no longer covered, even though I didn’t change them? Which lines are non-deterministically covered, perhaps indicative of flaky tests?  By answering these questions with hard data, we can provide developers with a rich understanding of the impact of their actions on test suite effectiveness.

If you have any feedback, please, email Jon, at bellj@gmu.edu.

Michael Hilton

Hayder Al Haddad

Jonathan Bell

Darko Marinov