In the past year, I got laid off from a job, got a new job that was terrible, then got another new job that I like very much. In my career I am often in a senior role where I can decide what work I should do with very little guidance. I know of a number of other mid- to senior QA/test people in a similar situation in recent times, so I decided to create a brief guide for deciding where to put your efforts as a senior QA/test professional. There are three criteria that guide my work and that I believe should guide yours, and there are three activities that guide the process by which the work gets done.

Complexity

When faced with an unfamiliar software application in need of extensive testing, the first place where I always start is with complex features. It is pretty simple to evaluate the complexity of a feature: if it looks complex, it almost certainly is complex. You might see a web page with many elements and controls, or you might see a configuration file of unusual size. When you tackle testing complex features first, not only are you providing valuable test coverage right away, but you will also learn about the entire application more quickly than if you tackle simpler features first.

Risk

When I am exploring an application new to me I am constantly asking myself "Does this feature make money?" Any feature that is directly responsible for customers paying money to the org should go under a regression test regime as soon as possible. Other good questions might be:

• If this feature stops working, could someone be injured or die? 

• If this feature stops working, could someone lose their money unintentionally?

• If this feature stops working, could someone complain to the media and hurt the organization?

Features whose failures have important consequences should always have reliable tests in place.

Churn

Any time you change working code, you have a risk of unforeseen problems in the code. While you are building out a regression test suite based on complexity and risk, you also want to be on top of testing ongoing changes to the code. These new features and enhancements are always a potential source of new problems, and keeping up with current development is an important part of testing the application and understanding the application.

Three Activities

This may seem counterintuitive, but the process of building out an effective regression test suite is independent of tools and even independent of any other context at all. In my case I am usually using Ruby with my favorite set of open source helper libraries. Someone else may be using Java and building out their own DSLs from scratch. Someone else may even be creating a scripted suite of tests intended to be run manually!

These are all the exactly the same process.

The process of creating an effective regression test suite involves a cycle of exploring the application, creating the tests based on the exploring, and evaluating the results of the tests as you create them. Some ways of doing this may be more or less efficient, or more or less practical, but any test suite that does not involve a constant design cycle of exploring, creating, and evaluating will not be effective. It is a powerful and meaningful approach that any number of people may find surprising.

Finally, once created, tests should remain "green" until such time as the application being tested changes. In other words: do not create flaky tests. Create tests such that every time one fails, you get new information about the application being tested. Your test failures should tell you that either you have a bug; or that you need to update the test for new behavior. And of course, both of those cases are opportunities for a new round of exploring, creating, and evaluating with your test suite.