A Look at Why I can’t follow advice and neither can you…

Anyone who has tried to lose weight knows there is a significant gap between “knowing” and “doing.”

We can read the books. Follow the science. Listen to the advice. The professional advice and of course the “helpful” information from well-meaning friends, coworkers and even the cashier at the grocery store. Weight-loss is so ubiquitous in our society that everyone feels confident sharing their expert advice and experience. And yet we are a good decade or more into an obesity epidemic…

Lousy Advice or Lack of Questioning?

Software testing ironically has many of the same issues. Testing is part of every organization, and most any developer can tell you how get things done (usually without having done it themselves) and there is plenty of advice, numerous books on best practices that tell you what should be doing. And yet, testing doesn’t always go the way it should nor does it consistently deliver the results that we expect. This is not due to a lack of standard rules or even expert advice, but it is indicative of the fact that each situation has unique conditions. Success is thus much more complicated than simply the following of advice. For example, telling a vegan that the perfect way to lose weight is to cut out carbs is perhaps more complicated and a more significant sacrifice than for someone working (and eating!) at a BBQ shack.

In many situations testing does not work for similar reasons to diet advice: it is not appropriate to the specific organization and the implementing individuals. You need a solution that works for you, for your group, for your particular advantages, and for your limitations. Drugs (pharmaceuticals) are another parallel. Not every drug is right for every person. Some people experience side effects or allergies, where others don’t. Some drugs are only effective for some people. Some drugs make grand claims, but it is unclear whether they are even actually any more effective than a placebo.

Let’s consider the disclaimer on Forbes for Chantix a drug that is supposed to help folks quit smoking.

Purpose: A prescription medicine that contains no nicotine, Chantix can help adults stop smoking. Studies have shown that after 12 weeks of use, 44% of users were able to quit smoking.

Side Effect: Some of the most common side effects of using Chantix include trouble sleeping, as well as vivid, unusual or increased dreaming. In February, the FDA also released a public health advisory stating that severe changes in mood and behavior may be related to use of the drug.

Only 44% effective. Trouble sleeping. Vivid dreams. Possible unknown changes to behavior. Wow. Are we sure this drug is worth the investment and risk? Is it the optimal method to quit smoking?

We should be asking similar questions when we select testing methods.

Time and money can be spent quickly on ineffective methods. Ineffective testing creates frustration, scheduling problems, budget problems and it often results in a lack of morale for the various stakeholders. And so, the next time you consider implementing the latest advice on engineering productivity or a different idea for reducing testing cycles, think carefully about what its side effects might be for your specific situation. Is the testing method truly optimal for your organization? Similarly, if you just completed a testing cycle and your result was moody engineers and not meaningful data, consider writing up a disclaimer for the next group or project that might fall into a similar trap.

One-size-fits-all testing disclosure:

Problems may occur if testing is poorly integrated or a one-size-fits-all approach is taken. Some of the most common side effects of a “one-size-fits-all approach” include unrealistic expectations, inadequate identification of defects and overextended budgets. A NIST study indicated that 25% to 90% of software development budgets might be spent on testing.

For a field where we have a digital track record for everything from requirements to code to bugs to releases, we don’t have much information about what works and what does not work. Everyone tries to share the advice that he or she believes is right; however, before implementing the newest tip make sure to ask questions. Ask the person sharing the advice, if she has any real world experience with the method. And, then ask yourself or your team, if the advice is truly appropriate to and optimal for your testing situation. 

Remember your organization is unique and not a one-size-fits-all.

One comment

  1. This is why I think Frameworks vs. Methodologies are the best approach.

    Scrum, for instance, is a framework but many perceive it as a methodology.

    Frameworks are flexible by definition,
    and should be extensible.

    Kind of like an interface analogy. The interface has a signature that must be implemented and so long as you implement the interface then you are following the framework.

    In this sense your testing approaches are polymorphic which connotes that one size not necessarily fit all.

Leave a Reply