Anyone who has tried to lose weight knows there is a big gap between “knowing” and “doing.”
We can read the books. Follow the science. Listen to the advice. The professional advice and of course the “helpful” advice from well-meaning friends, coworkers and even the cashier at the grocery store. Weight-loss is so ubiquitous in our society that everyone feels confident sharing their expert advice and experience. And yet we are a good decade or more into an obesity epidemic…
Bad Advice or Lack of Questioning?
Software testing ironically has many of the same issues. Testing is part of every organization and most any developer can tell you how it should be done (usually without having done it themselves) and there is plenty of advice, numerous books on best practices that tell you what should be done. And yet, testing doesn’t always go the way it should nor does it always deliver the results that we expect. This is not due to a lack of common rules or even expert advice, but it is indicative of the fact that each situation has unique conditions. Success is thus much more complicated than simply the following of advice. For example, telling a vegan that the perfect way to lose weight is to cut out carbs is perhaps more complex and a bigger sacrifice than for someone working (and eating!) at a BBQ shack.
In many situations testing does not work for similar reasons to diet advice: it is not appropriate to the specific organization and the implementing individuals. You need a solution that works for you, for your organization, for your specific advantages, and for your limitations. Drugs (pharmaceuticals) are another parallel. Not every drug is right for every person. Some people experience side effects or allergies, where others don’t. Some drugs are only effective for some people. Some drugs make great claims, but it is unclear whether they are even actually any more efficient than a placebo.
Purpose: A prescription medicine that contains no nicotine, Chantix can help adults stop smoking. Studies have shown that after 12 weeks of use, 44% of users were able to quit smoking.
Side Effect: Some of the most common side effects of using Chantix include trouble sleeping, as well as vivid, unusual or increased dreaming. In February, the FDA also released a public health advisory stating that severe changes in mood and behavior may be related to use of the drug.
Only 44% effective. Trouble sleeping. Vivid dreams. Changes to behavior. Wow. Are we sure this drug is actually is worth the investment and risk? Is it really the optimal method to quit smoking?
We should be asking similar questions when we select testing methods.
Time and money are easily spent on ineffective methods. Ineffective testing creates frustration, scheduling problems, budget problems and it often results in a lack of morale for the various stakeholders. And so, the next time you consider implementing the latest advice on engineering productivity or a different idea for reducing testing cycles, think carefully about what its side effects might be for your specific situation. Is the testing method truly optimal for your organization? Similarly, if you just completed a testing cycle and your result was moody engineers and not meaningful data, consider writing up a disclaimer for the next group or project that might fall into a similar trap.
One-size-fits-all testing disclosure:
Problems may occur if testing is poorly integrated or a one-size-fits-all approach is taken. Some of the most common side effects of a “one-size-fits-all approach” include unrealistic expectations, inadequate identification of defects and overextended budgets. A NIST study indicated that 25% to 90% of software development budgets may be spent on testing.
For a field where we have a digital track record for everything from requirements to code to bugs to releases, we don’t have much information about what works and what does not work. Everyone tries to share the advice that they believe is right; however, before implementing the newest tip make sure to ask questions. Ask the person sharing the advice, if she has any real world experience with the method. And, then ask yourself or your team, if the advice is truly appropriate to and optimal for your testing situation.
Remember your organization is unique and not a one-size-fits-all.