Feature Coverage

Feature Coverage

For some time now I have talked about testing software in terms of "Feature Coverage" but Feature Coverage is not a term that is widely recognized or one that I have even ever defined particularly well.

In this case, a "feature" is a path through your software application. These paths start at a particular place and they end at a place where the user of the software accomplishes some goal in a satisfactory and meaningful way.

For example, many paths through software applications start at login. Having logged in successfully, a user might do any number of things. They might:

• Create a new thing

• Edit a thing that already exists

• Change their Settings so as to change the behavior of the software for themselves or for the people who use the software.

• Delete something they no longer wish to have

What this means is that there are a finite number of paths through any given software application. There are a finite number of features of the application that may be tested. And there are definitely some design guidelines to test features well.

RISK AND COMPLEXITY

We can evaluate software features in terms of risk and complexity, and we can prioritize testing on the basis of risk and complexity. A feature that is high risk might not be complex. A simple feature that generates revenue will always be high risk. A complex feature important to the users may be of low risk to the business itself. It is the features that are both high risk and highly complex that deserve the most testing.

TREE VS WEB

A feature is a unique path through a software application. Feature coverage is the set of tests that exercise each unique path. An important design guideline for feature coverage is to avoid duplicating paths being tested.

Consider the simplest case, for example a calculator. An excellent feature test would be to add 2 plus 2 minus 1 and confirm that the result is 3. Any feature tests beyond this would be redundant, since we know that a path through the application combining multiple functions yields a correct result in one reasonable case. Other reasonable tests for this application would not be feature tests, but probably unit tests or integration tests.

For a more complex example, you may have to test an application that leads your users through eight introductory screens before they can choose a unique destination in the application. This may be a design flaw in the application itself. If you have to create feature tests under such circumstances, consider taking special measures to bypass the redundant steps. For example if this were a web application you might be able to navigate to specific URLs in order to bypass the redundancies.

However, if the application you are testing offers opportunities for unique paths but you create tests that duplicate many steps before diverging to unique end states, that is definitely a design flaw in the tests you have created.

A poor approach to feature testing is a tree-like structure. If every test has to climb up the trunk, then any breakage at the trunk will cause all the tests to fail without providing useful information. If many tests have to go down one or two branches before arriving at a unique leaf, then any breakage on a branch will cause many tests to fail.

A design flaw related to the tree structure pattern is for many feature tests to follow the same path through the application but yield different results. For the calculator example above. If a test for 2 plus 2 minus 1 succeeds, creating another test for 4 plus 4 minus 3 is a pointless waste of time. Again, some sort of tests other than feature tests may be appropriate, but feature coverage aims to test valid results for unique paths through the application.

A much better approach to feature testing is a web-like structure of tests. Visualize each test entering the application and then as soon as possible taking a unique path through the application to a satisfying destination. In this sort of organizational structure, any test failures may be isolated and examined without compromising the other tests in the suite.

GROWTH AND MATURITY

To build a feature test suite we prioritize the tests we create based on risk and complexity. But unique paths through the application are finite, and at some point the opportunities for creating new tests diminish. As a feature test suite matures, we invest more time in maintenance and in exploring new features than in creating new tests.

When our investment in the feature test suites shifts from an emphasis on creating new tests to maintaining existing tests and exploring new features, then we can say that our feature test coverage is high.

Many software test automators are tempted at this point to add more tree-like tests to the test suite in the guise of adding more "coverage". These tests follow the same paths through the application as existing tests but use different sets of data. While it is tempting to do this, it is almost certainly a waste of your time, as the maintenance burden increases but the feature coverage does not.

ALWAYS BE EXPLORING

As an end-to-end test suite matures, the smart tester will add a new test only if that test follows a new unique path through the application. This increases feature coverage, increases the chances of identifying important problems, while minimizing the maintenance burden for the entire suite.

It follows then that a smart tester will invest significant time in discovering and documenting (in tests) new unique paths through the application. This is exactly the same thing as Exploratory Testing.

We explore in order to automate tests; we automate tests in order to have time to explore new features. In a mature test practice, the two activities are inseparable.