The test strategy is the set of ideas that guide your test process to fulfill your mission. The test strategy is shaped by asking yourself what your mission is, who’s affected, and how much time do you have. Ultimately, we have a finite time and we can’t test for everything so we’ll discuss a few things that need to be considered when thinking of a test strategy.
- First and foremost, what is your mission?
- What is the context?
- You are wrong
What is your mission?
Before starting off in any testing, you should understand what your mission is.Why does your company exist? Why do we have this release? Why should you care? Testing is expensive and you may end up testing something that is useless.
Why bother with why? I find myself questioning why I do things a lot. From my daily routine to every single thing I have to test because of a recent release. It seems that there is a never ending pipeline of things to do for the development team, which is good. One of our recent releases but the feature ended up being useless. The root cause is the false identification of the business problem at hand, there was no conversation on why we need this. It was just a let’s do this because we can do it. Testing and Development is expensive. Before and during testing, activities testing must only include ones that matter enough to spend time testing.
We test for the values and specifications of people who matter. When we are testing we must ensure that we are serving somebody’s interest. Failure to do may result in testing not being as optimal as possible.
“I’m testing because I was told to do so” Testing is a service. Testers are there to provide an excellent service. We serve many stakeholders and give them our thoughts and opinions of the matter at hand.
- Management - To management, you are there to create reports in a concise manner. This will enable them to make decisions based on your findings.
- The end user - The one you are truly serving. There is a satisfaction I find when I keep telling myself that I pretend to be in the end users shoes. Is there bad usability? Something not working as expected?
- You care - As the tester, despite not having full responsibility of releases and development, you have the urge to do the best job you possibly can. Get the mindset of “being responsible for everything that happens around you” this has been one of the things that has improved my testing alot.
There are many possible test strategies
There is no one size fits all strategy. The test strategy is an outline that describes your test approach for the specific release at hand. It will inform developers, your product manager, and other stakeholders with your testing process. This will include things like objectives, resources required, and estimated time to complete. Here are some example of strategies that I’ve used.
1 - We will release the product to our customer success team after a brief review to find any glaringly obvious problems. These users will tell us about their thoughts on the feature and any improvements they’d like us to make.
2 - We will perform automated regression testing and manual testing in parallel. The manual testing will be done using a set of test cases that will focus on testing the functionality of the product that have not been covered by the regression testing. The automated regression testing will first, focus on basic functionality to provide an early warning on major functional failures.
3 - We will run our automated smoke testing to qualify the build, This will let us know if any of the major functionalities that we’ve agreed upon are working as expected. Once this has qualified, we will run our automated regression tests alongside our customer success team testing the product. Our top priority is finding fundamental deviations from specified behaviour, and we’re concerned in which this release might violate user expectations.We are also concerned about performance but that is still in discussion.
The strategy is more than the test
What has been mentioned above is a strategy, and they’re all different. Each strategy tells a little story and have a different emphasis, they tell a story that explain and justify the testing that needs to be done. What is written above is quite basic, in a real case, we would use very specific business requirements and rules, what part will be tested and what different testing approaches will be used.
Oh the dreaded, D, Documentation. I have been in involved in projects with long winded test strategies, to test strategies written in a few sentences. We all know that good documentation takes time is hard and takes time to write. Moreover, the maintenace is quite difficult. When writing test plans, don’t overdo it with documentations, please.
I’ve seen a lot of good testing done with a lot of documentation and some without. Whatever you use, make sure that your approach has been approved to make sure you avoid backlash.
Fitting to your context
One way to visualise test planning situation is shown in the text below. This is the satisfice context model, I’ve used this model many times and have added one thing. The five points of the star represent what you’ve been given: resource and constraints.
The givens are:
- Development - The system that produces the product to be tested. How will the product be received? When will it be received? How testable is it?
- Requirements - Is the system doing its job? What are the risks of the testing?
- Test Team - Who is available to test the product? Are they up-to-speed with the product?
- Test Lab - The materials required for testing. Is the bug tracking system in place? Do you have the equipment? How can we ensure the testers don’t get distracted?
- Mission - What is your company mission? What is the development team mission? What does this release mean for the company’s mission and the team’s mission?
- Stakeholders - What stakeholders must we identify and how do we keep them on the loop? How do we communicate to our stakeholders about the progress of the project?
Normally, I don’t believe what the stuff I read until I’ve tried them myself but the notion of testing with a more diverse strategy with less thorough is better than a not-so diverse strategy and being very thorough.
This comes from the fact that software is extremely complex products and we don’t have enough time to test everything. When you test, you are testing a certain section, no single-minded viewpoint will find more important problems quickly.
Have you ever wondered why is that when you release a product then ship it, the users discover big problems immediately after release. Similar to what I’ve mentioned above, tunnel-vision is the enemy. Not that you did not test enough, you just didn’t test using different viewpoints.
Revise your initial test strategy
Strategy is like water. It should evolve and fit to the context that you need. When the vase is of a rectangular shape, the strategy becomes rectangular. When your environment is a globe, you fit the globe. The strategy should be based on risks. This comes with the problem about not knowing what your risks are.
You start the project, you have an inkling as to what might be affected and what could arise. Your strategy at the start of the project is going to suffer from not being focused on risk and focusing on items that are not of high risk.
Minimise your test by talking to your team about what the risks they think are, talk to the wider business team as to what they think about the release, and if you’re lucky, talk to some clients about the release and gather their thoughts!
What can I do now?
In the middle of the project, ask yourself this question regularly. Let’s say the project has been spec’d out, what can you do now? I would talk to the product manager and discuss the specs, and then talk to the users about their thoughts.
What I’ve found about asking this question are:
- The tunnel vision of my thoughts
- The specs not considering real-case scenarios
- The specs not taking into account how it should affect the different sections
Don’t assume that certain techniques are useful only at certain times. In any given time, test whatever is worth testing and use the techniques you deem will serve your stakeholders best.