Why I can’t follow advice and neither can you…

Posted by

Anyone who has tried to lose weight knows there is a big gap between “knowing” and “doing.”

We can read the books. Follow the science. Listen to the advice. The professional advice and of course the “helpful” advice from well-meaning friends, coworkers and even the cashier at the grocery store. Weight-loss is so ubiquitous in our society that everyone feels confident sharing their expert advice and experience. And yet we are a good decade or more into an obesity epidemic…

Bad Advice or Lack of Questioning?

Software testing ironically has many of the same issues. Testing is part of every organization and most any developer can tell you how it should be done (usually without having done it themselves) and there is plenty of advice, numerous books on best practices that tell you what should be done. And yet, testing doesn’t always go the way it should nor does it always deliver the results that we expect. This is not due to a lack of common rules or even expert advice, but it is indicative of the fact that each situation has unique conditions. Success is thus much more complicated than simply the following of advice. For example, telling a vegan that the perfect way to lose weight is to cut out carbs is perhaps more complex and a bigger sacrifice than for someone working (and eating!) at a BBQ shack.

In many situations testing does not work for similar reasons to diet advice: it is not appropriate to the specific organization and the implementing individuals. You need a solution that works for you, for your organization, for your specific advantages, and for your limitations. Drugs (pharmaceuticals) are another parallel. Not every drug is right for every person. Some people experience side effects or allergies, where others don’t. Some drugs are only effective for some people. Some drugs make great claims, but it is unclear whether they are even actually any more efficient than a placebo.

Let’s consider the disclaimer on Forbes for Chantix a drug that is supposed to help folks quit smoking.

Purpose: A prescription medicine that contains no nicotine, Chantix can help adults stop smoking. Studies have shown that after 12 weeks of use, 44% of users were able to quit smoking.

Side Effect: Some of the most common side effects of using Chantix include trouble sleeping, as well as vivid, unusual or increased dreaming. In February, the FDA also released a public health advisory stating that severe changes in mood and behavior may be related to use of the drug.

Only 44% effective. Trouble sleeping. Vivid dreams. Changes to behavior. Wow. Are we sure this drug is actually is worth the investment and risk? Is it really the optimal method to quit smoking?

We should be asking similar questions when we select testing methods.

Time and money are easily spent on ineffective methods. Ineffective testing creates frustration, scheduling problems, budget problems and it often results in a lack of morale for the various stakeholders. And so, the next time you consider implementing the latest advice on engineering productivity or a different idea for reducing testing cycles, think carefully about what its side effects might be for your specific situation. Is the testing method truly optimal for your organization? Similarly, if you just completed a testing cycle and your result was moody engineers and not meaningful data, consider writing up a disclaimer for the next group or project that might fall into a similar trap.

One-size-fits-all testing disclosure:

Problems may occur if testing is poorly integrated or a one-size-fits-all approach is taken. Some of the most common side effects of a “one-size-fits-all approach” include unrealistic expectations, inadequate identification of defects and overextended budgets. A NIST study indicated that 25% to 90% of software development budgets may be spent on testing.

For a field where we have a digital track record for everything from requirements to code to bugs to releases, we don’t have much information about what works and what does not work. Everyone tries to share the advice that they believe is right; however, before implementing the newest tip make sure to ask questions. Ask the person sharing the advice, if she has any real world experience with the method. And, then ask yourself or your team, if the advice is truly appropriate to and optimal for your testing situation. 

Remember your organization is unique and not a one-size-fits-all.

One comment

  1. This is why I think Frameworks vs. Methodologies are the best approach.

    Scrum, for instance, is a framework but many perceive it as a methodology.

    Frameworks are flexible by definition,
    and should be extensible.

    Kind of like an interface analogy. The interface has a signature that must be implemented and so long as you implement the interface then you are following the framework.

    In this sense your testing approaches are polymorphic which connotes that one size not necessarily fit all.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s