Effective QA is Not an Option it’s a Necessity: Here’s How to do it Right

As has long been the case in the software and technology industry, immigration is an important source for accessing Quality Assurance people.

Part of the reason is that, as per usual, QA isn’t always a job that attracts US-based developers. The US job market is already competitive for developers and moving developers on your team to quality assurance is often undesirable for a number of reasons. The result is that relocating QA developers to the USA is in many cases the most satisfactory solution for the industry.

Unfortunately, barriers to immigration continue to drive-up demand for qualified QA, which only further exacerbates the shortage of people with an appropriate development background already in the US.


Quality Assurance is not Optional

Compounding the problems experienced by a physical shortage of qualified developers for QA is the fact that some decision makers continue to consider QA an “optional” line item in their budget. Unfortunately for them, QA and the surrounding processes are anything, but “optional” and a good quality assurance engineer is a key player on any team.

In companies where QA is an accepted need, it is still often considered more of a necessary evil than a definite benefit. QA is too easily blamed when a project that’s been on schedule suddenly gets hung up in testing. Another common problem appears when development teams pick up speed after implementing workflow changes, only to discover that QA is still a bottleneck keeping them from delivering a product to their customer.


A Look at the Current Situation

As we’ve seen, numerous factors contribute to the myths surrounding Quality Assurance that contribute to this questionable climate that makes some engineers shy away from the moniker or even the field.  Moreover, we are experiencing an actual shortage of qualified engineers, which means that QA in many instances ends up being not an afterthought, but rather a luxury.

Immigration and work status has been hot-topics for the last few years. And regardless of where you fall on the political spectrum, if you work in software, you’ve likely experienced first hand the effect on the jobs market.

For those of you that have been lucky enough to hire engineers, many of you may have also been unlucky enough to discover that said engineer has to go back to India (or wherever his or her country of origin might be). And what started out as a short trip, quickly devolves into a lengthy, even a 6-month long, process to renew or regularize an H-1B visa, following changes made to the requirements in 2017.

Whether your experience with the dwindling QA applicant pool is first hand or anecdotal,  here are some statistics for you to munch on:

The US Bureau of Labor and Statistics expects the market for software developers to blow-up by 24% from 2016 to 2026. That means a need for approximately 294,000 additional software developers in an already tight market. If you think it’s hard to convince an engineer to join your QA team now, just wait and see what it will be like in 2026.

We can’t know for sure the number of H-1B visas currently held up due to changes in requirements, but this article does a decent job of discussing the demand and actual need for H-1B Visa’s in for the USA with a focus on the State of Massachusetts. If you’d like to know more, I’d suggest taking a look; however, for our purposes, I don’t think importing QA staff is necessarily the answer.

So, you need to have QA, but you can’t hire qualified staff to take care of the work. What can you do?

The Foundations of Quality Assurance

Before I answer this question, let’s take a look at why Quality Assurance exists in the first place. From a business perspective, there are a few things that pretty much every customer expects from a software product. The following three expectations are the crucial reasons to the story behind why quality assurance is not optional::

  1. Customers expect that the programs will work as requested and designed, and within the specified environments;
  2. They hope that software will be user-friendly;
  3. And, they assume that software will have been successfully debugged: meaning that QA must deliver a product that is at the least free of the bugs that would result in numbers 1 or 2 becoming false.

Historically, software teams were small, but since the early 80s, due to the need to scale quickly and keep up with changing requirements and other advances in technology and globalization, we’ve experienced rapid growth in the size of development teams and companies. This growth has led to implementing a wide variety of tactics from workflow solutions, think Waterfall or Agile, to different methods of increasing productivity and efficiencies, such as offshore teams and microservices.

Take a look at this simple chart approximating the increase in development team size from the original release of Super Mario Brothers in 1985 to Super Mario World in 1990 and Super Mario 64 in 1996. (Noting that in the credits, by 1996 the occasionally thank entire teams, not just individuals, so actual number is likely even higher).

Super Mario Release Team Size

© Possum Labs 2018

Where we haven’t (at least in the USA) been able to keep up is in the training of new software engineers. QA departments, regardless of size, are challenged to carry-out all the different required processes to follow software through the development lifecycle to delivery and maintenance (updates), while also keeping abreast of changes to technology and integrations.

A misfortunate result of this shortage of QA engineers is that the point in the development cycle where most companies fall short is in testing. And, yet, the ability to provide useful and meaningful testing is crucial to the successful delivery of quality assurance to one’s client, whether building an in-house product, such as for a financial institution, or a commercial product for the public market.

While offshore teams may be a solution for some companies, many companies are too small to make offshore building teams practical or cost-effective.

What’s more is that many engineers tend to be good at one thing — development — they may not have a good sense of your organization’s business goals or even an understanding of what makes a good customer experience. And while your high paid development staff might excel at building clever solutions, it doesn’t necessarily mean that they also excel at testing their own goods. And do you really want to pay them to do your testing, when their time could be better invested in innovations and features? At Possum Labs we’ve determined that it is often most efficient to design workflows and teams to adjust to the people you have.

This disconnect between development requirements and a full understanding of business goals is in fact often the culprit in a pervasive disconnect between testing and business outcomes. What do I mean by disconnect? Let’s consider the four following statements and then talk about some real-life examples:

  • Users prefer seamless interfaces, intuitive commands, and technology that makes them feel smart, not dumb.
  • Businesses prefer software that assures their business goals are met and that technology allows their employees to work smarter with greater efficiency thus promoting growth and profit.
  • Today the average person is adequately adept and familiar with technology to know when your software exhibits even moderately lousy UX. And companies can also experience public shaming via social media when they make a particularly dumb or inopportune mistake.
  • And then there are the security risks.

In 2017 we saw several major security snafus experienced by large corporations that from the few details publicized were the direct result of inaction on the part of the decision-makers, despite being notified by engineering.

One might think that the decision makers, while moderately acknowledging the risk, may have simply gambled that nothing would come of the risks while taking steps to protect themselves.

I would like to go a step further. I’d wager that everyone involved fell victim to a set of common testing pitfalls.

Indeed, one of the most challenging aspects of testing is figuring out not only how to effectively and efficiently create and run tests, but most importantly to figure out how to confidently deliver meaningful results to the decision makers. Whether you are a software business or a business that uses software, successful quality assurance is crucial to the long-term health and success of your business.


Let’s do a quick recap of what we’ve covered so far:

  1. There is a shortage of qualified test engineers.
  2. Users want products they can rely on and that are friendly to use.
  3. Companies want products they can trust that improve efficiencies, their bottom line and that, of course, make their clients happy.
  4. It is difficult to create tests that deliver meaningful results when the testing is done by engineers that don’t necessarily understand the businesses end goals.
  5. Decision makers don’t want to understand the tests; they want to have meaningful results so that they can make effective decisions.


So what if we could solve all of these problems at once?

This type of solution is what Possum Labs achieves through the clever use of solutions and tools that integrate with your existing systems, processes, and people. We build out quality assurance so that anyone who understands the business goals can successfully carry-out testing and efficiently uses the results.

Not only does this solve problems 1 to 5 above, but it is also, in fact, a super solution in that it prevents most companies from having to hire or train new developers. Instead, you need to hire for people who with a keen understanding of your business and that can be trained to work with your tools. Possum Labs methods allow you to implement and upgrade your quality assurance — sometimes even reducing your staffing load — while delivering better and more meaningful results, so that your end of service recipients get better services or better products than before.


How does Possum Labs do this?

Each of our solutions varies a bit from company to company, but in general, we use several tools, including proxies and modules (think Lego) to make it so existing tests can be modified and new tests written with simply reorganizations of the “bricks.” This focus on custom solutions allows a non-technical individual with a solid understanding of business goals to generate tests and result that deliver meaningful results with confidence for him or her to share with decision makers.

The result is that testing bottlenecks open up allowing for a more efficient flow of information and better feedback through all channels. Products are delivered faster. Information flows smoothly. Better decisions are made, and efficiencies are gained. Developers can focus on development and decision makers in achieving their strategic goals. Meanwhile, you’ve got happy customers, and everyone can get a good night’s rest.

Why Negligence in Software should be of Urgent Concern to You

The future of Liability in Software:

Many things set software companies apart from other businesses. A significant, but an often overlooked difference, is that the manufacturers of software exhibit little fear of getting sued for negligence, including defects or misuse of their software. For the moment, the legal consequences of defective software remain marginal.

After more than a decade, even efforts to reform the Uniform Commercial Code (UCC) to address particularities of software licensing and software liabilities remain frozen in time. As Jane Chong discusses in We Need Strict Laws, the courts consistently rule in favor of software corporations over consumers, due to the nature of contract law, tort law, and economic loss doctrine. In general, there is little consensus regarding the question: should software developers be liable for their code?

Slippery When Wet

If you go to your local hardware store, you’ll find warning signs on the exterior of an empty bucket. Look around your house or office, and you see warning labels on everything from wet-floors to microwaves and mattresses.

In software, if you are lucky you might find an EULA buried somewhere behind a link on some other page. Most products have a user manual, why is it not enough to print your “warning” inside? An easy to find, easy to implement as a standard.

Legal issues in software development: why is there no fear?

Fear is socially moderated and generated. Hence the term “mass hysteria.” We fear things that we’ve already experienced or that have happened to personal connections. We all too easily imagine horrible events that have befallen others. In an age of multimedia, this is a rather flawed system that has gone berserk or as they say “viral.” What we see has left us with some pretty weak estimates on the odds of events, like shark attacks or kidnappings.

One reason we don’t fear lawsuits around software is that we don’t see them in the public sphere. They do happen, but all too often the cases never make it very far. Or the judge rules in favor of the software developer. Interpretation of the laws makes it difficult to prove or attribute harm to a customer.

To date, we’ve yet to see a Twittergate on negligence in software development. This lack of noise doesn’t mean that no one has reasons to sue. Instead, it is more of an indicator that for the moment, the law and the precedent are not written to guide software litigation. And with little news or discussion in the public sphere, no one has the “fear.”


A Matter of Time

Frankly, it is not a matter of will it happen, but when? What is the likelihood that in the next five years there will be successful suits brought against software firms? Should we make a bet?

Software development is an odd industry. We leave an incredible electronic trail of searchable data for everything that we do. We track defects, check-ins, test reports, audit trails, and logs. And then we back up all of them. Quite often these records last forever. Or at least as long, if not longer than the company that created the record.

Even when we change from one source control system to another, we try to make sure that we keep a detailed record intact just in case we want to go back in time.

This level of record keeping is impressive. The safety it provides and the associated forensic capabilities can be incredibly useful. So what is the catch? There is a potential for this unofficial policy of infinite data retention to backfire.

Setting the Standard

Most companies follow standard document retention policies that ensure businesses save both communications and artifacts to meet fiscal or regulatory requirements for a required period then eventually purged after some years.

Ironically, even in companies who follow conservative document retention policies, the source control, and bug tracking system is often completely overlooked if not flat out ignored. From a development perspective, this makes sense: data storage isn’t expensive, so why not keep it all?

The Catch & The Cause

The reason that document retention policies exist is not merely to keep companies organized and the IRS happy, it’s because of the potential for expensive lawsuits. Companies want to make sure that they can answer categorically why certain documents do or do not exist.

For instance, let’s say your company makes widgets and tests these before shipping them on; you don’t want to say that “we don’t know why we don’t have those test results.” By creating a documented process around the destruction of data (and following it) you can instead point to the document and say the data does not exist — it’s been destroyed according to “the policy.”

Policy? What Policy?

This example takes us back to the story of liability in software. In the software business we often keep data forever, but then we also delete data in inconsistent bursts. Maybe we need to change servers, or we are running out of space, or we find random backups floating around on tapes in vaults. So we delete it or shred it or decide to move to the Cloud to reduce the cost of storage.

This type of data doesn’t tend to carry any specific policy or instructions for what we should and shouldn’t include, or how best to store the data, and so on.

What’s more, when we document our code and record check-ins, we don’t really think about our audience. Or default audience is likely ourselves or the person(s) in the cube next to us. And our communication skills on a Friday night after a 60-hour-week won’t result in the finest or most coherent checking comments, especially if our audience ends up being someone besides our cubemate.

The reason that this type of mediocre record keeping persists is that it remains difficult to sue over software. There really are no clear-cut ways to bring a suit for defective services provided. If you live in the US you know the fear of lawsuits over slips and falls; this fear does not exist for creating a website and accepting user data.

Walking a Thin Line

My guess is that this indifference to record keeping and data retention will persist as long as potential suitors must do all the groundwork before seeing any money. And, as long as judges continue to side with corporations and leave plaintiffs in the lurch. However, as soon as that first case sneaks through the legal system and sets a precedent, anyone and everyone just may reference that case.

Ironically, patents or copyright protection don’t travel with theories presented in a trial, which means that once a case makes it through the system, the case only needs to be referenced. Suggesting that if one lawyer learns how to sue us; they all do. Think of it as an open source library you can reference, once it exists anyone gets to use it.

I expect that there will be a gold rush, we are just waiting for the first prospector to come to town with a baggy of nuggets.

As to what companies can do? For now, create an inventory of what data you keep and how it compares to any existing policies. This may involve sitting down in a meeting that will be an awkward mix of suits and shorts where there likely will be a slide titled “What is source control?” There is no right answer, and this is something for every company to decide for themselves.

Where does your development process leave a data trail? Has your company had discussions about document retention and source control?

How to Effortlessly Take Back Control of Third Party Services Testing

Tools of the Trade: Testing in with SaaS subsystems.

For the last few years, the idea has been floating around that every company is a software company, regardless of the actual business mission. Concurrently even more companies are dependent upon 3rd party services and applications. From here it is easy to extrapolate that the more we integrate, the more likely it is that at some point, every company will experience problems with updates: from downtime, uptime, and so on.

Predicting when downtime will happen and or forcing 3rd party services to comply with our needs and wishes is difficult if not impossible. One solution to these challenges is to build a proxy. The proxy allows us to regain a semblance of control and to test 3rd party failures. It won’t keep the 3rd parties up, but we can simulate failures whenever we want to.

As an actual solution, this is a bit like building a chisel with a hammer and an anvil. And yet, despite the rough nature of the job, it remains a highly useful tool that facilitates your work as a Quality Assurance Professional.

The Problem

Applications increasingly use and depend upon a slew of 3rd party integrations. In general, these services tend to maintain decent uptime and encounter only rare outages.

The level of reliability of these services leads us to continue to add more services to applications with little thought to unintended consequences or complications. We do this because it works: it is cheap, and it is reliable.

The problem (or problems) that arise stem from the simple nature of combining all of these systems. Even if each service maintains good uptimes, errors and discordant downtime may result in conditions where the time that all your services are concurrently up is not good enough.

The compounding uptimes

Let’s look at a set of services that individually boast at least 95% uptime. Let’s say we have a service for analytics, another for billing, another for logging, another for maps, another for reverse IP, and yet another for user feedback. Individually they may be up 95% of the time, but let’s say that collectively the odds of all of them being up at the same time is less than 75%.

As we design our service, working with an assumption of around a 95% up-time scenario feels a lot better than working with a chance of only 75% uptime. To exacerbates this issue, what happens when you need to test how these failures interact with your system?

Automated Testing is Rough with SaaS

To create automated tests around services being down is not ideal. Even if the services consumed are located on site, it is likely difficult to stop and start them using tests. Perhaps you can write some PowerShell and make the magic work. Maybe not.

But what happens when your services are not even located on the site? The reality is that a significant part of the appeal of third-party services is that businesses don’t really want onsite services anymore. The demand is for SaaS services that remove us from the maintenance and upgrade loop. The downside to SaaS means that suddenly turning a service off becomes much more difficult.

The Solution: Proxy to the Rescue

What we can do is to use proxies. Add an internal proxy in front of every 3rd party service, and now there is an “on / off switch” for all the services and a way to do 3rd party integration testing efficiently. This proxy set-up can also be a way to simulate responses under certain conditions (like a customer Id that returns a 412 error code).

Build or buy a proxy with a simple REST API for administration and it should be easy to run tests that simulate errors from 3rd party providers. Now we can simulate an entire system outage in which the entire provider is down.

By creating an environment isolated by proxies, test suites can be confidently run under different conditions, providing us with valuable information as to how various problems with an application might impact our overall business service.

Proxy Simulations

Upon building a proxy in between our service and the 3rd party service, we can also put in the service-oriented analog of a mock. This arrangement means we can create a proxy that generates response messages for specific conditions.

We would not want to do this for all calls, but for some. For example, say we would like to tell it that user “Bob” has an expired account for the next test. This specification would allow us to simulate conditions that our 3rd party app may not readily support.

Customer specific means better UX

By creating our own proxy, we can return custom responses for specific conditions. Most potential conditions can be simulated and tested before they happen in production. And we can see how various situations might affect the entire system. From errors to speed, we can simulate what would happen if a provider slows down, a provider goes down, or even a specific error, such as a problem that arises when closing out the billing service every Friday evening.

Beyond Flexibility

Initially, the focus might be on scenarios where you simulate previous failures of your 3rd party provider; but you can also test for conditions for which your 3rd party may not offer to you in a test environment. For instance, expiring accounts in mobile stores. With a proxy, you can do all of this by merely keying off of the specific data that you know will come in the request.

In Conclusion: Practical Programming for the Cloud

This proxy solution is likely not listed in your requirements. At the same time, it is a solution to a problem that in reality is all too likely to arise once you head to production.

In an ideal world, we wouldn’t need to worry about 3rd party services.

In an ideal world, our applications would know what failures to ignore.

In the real world, the best course is preparation: this allows us to test and prevent outages.

In reality, we rarely test for the various conditions that might arise and cause outages or slowdowns. Working with and through a 3rd party to simulate an outage is likely very difficult, if not impossible. You can try and call Apple Support to request that they turn off their test environment for the App store services, but they most likely won’t.

This is essentially a side-effect from the Cloud. The Cloud makes it is easy to add new and generally reliable services, which also happen to be cheap and makes the Cloud an all-around good business decision. It should not then be surprising that when you run into testing problems, an effective solution will also come from the Cloud. Spinning up small, lightweight proxies for a test environment is a practical solution for a problem in the Cloud.

Unrealistic Expectations: The Missing History of Agile

Unrealistic expectations or “why we can’t replicate the success of others…

Let’s start with a brain teaser to set the stage for questioning our assumptions.

One day a man visits a church and asks to speak with the priest. He asks the priest for proof that God exists. The priest takes him to a painting depicting a group of sailors, safely washed up on the shore following a shipwreck.

The priest tells the story of the sailors’ harrowing adventure. He explains that the sailors prayed faithfully to God and that God heard their prayers and delivered them safely to the shore.

Therefore God exists.

This is well and good as a story of faith. But what about all the other sailors who have prayed to God, and yet still died? Who painted them?

Are there other factors that might be at play?

When we look for answers, it’s natural to automatically consider only the evidence that is easily available. In this case, we know that the sailors prayed to God. God listened. The sailors survived.

What we fail to do, is look for less obvious factors.

Does God only rescue sailors that pray faithfully? Surely other sailors that have died, also prayed to God? If their prayers didn’t work, perhaps this means that something other than faith is also at play?

If our goal is to replicate success, we also need to look at what sets the success stories apart from the failures. We want to know what the survivors did differently from those that did not. We want to know what not to do, what mistakes to avoid.

In my experience, this is a key problem in the application of agile. Agile is often presented as the correct path; after all lots of successful projects use it. But what about the projects that failed, did they use Agile, or did they not implement Agile correctly? Or maybe Agile is not actually that big a factor in the success of the project?

Welcome to the history of what is wrong with Agile.

Consider this, a select group of Fortune 500 companies, including several technology leaders decides to conduct an experiment. They hand pick some people from across their organization to complete a very ambitious task. A task of an order of magnitude different from anything they’d previously attempted and with an aggressive deadline.

Question 1: How many do you think succeeded?

Answer 1: Most of them.

Question 2: If your team followed the same practices and processes that worked for these teams do you think your team would succeed?

Answer 2: Probably not.

The Original Data

In 1986, Hirotaka Takeuchi and Ikujiro Nonaka published a paper in the Harvard Business Review titled the “The New New Product Development Game.” In this paper, Takeuchi and Nonaka tell the story of businesses that conduct experiments with their personnel and processes to innovate new ways to conduct product development. The paper introduces several revolutionary ideas and terms, which most notably developed the practices that we now know as agile (and scrum).

The experiments, run by large companies and designed for product development (not explicitly intended for software development), addressed common challenges of the time regarding delays and waste in traditional methods of production. At the root of the problem, the companies saw the need for product development teams to deliver more efficiently.

The experiment and accompanying analysis focused on a cross-section of American and Japanese companies, including Honda, Epson, and Hewlett-Packard. To maintain their competitive edge each of these companies wished to rapidly and efficiently develop new products. The paper looks at commonalities in the production and management processes that arose across each company’s experiment.

These commonalities coalesced into a style of product development and management that Takeuchi and Nonaka compared to the rugby scrum. They characterized this “scrum” process with a set of 6 holistic activities. When taken individually, these activities may appear insignificant and may even be ineffective. However, when they occur together as part of cross-functional teams, they resulted in a highly effective product development process.

The 6 Characteristics (as published):

  1. Built-in instability;
  2. Self-organizing project teams;
  3. Overlapping development phases;
  4. Multilearning;
  5. Subtle control;
  6. And, organizational transfer of learning.

What is worth noting, is what is NOT pointed out in great detail.

For instance that the companies hand-picked these teams out of a large pool of, most likely, above average talent. These were not random samples, they were not even companies converting their process, these were experiments with teams inside of companies. The companies also never bet the farm on these projects, they were large, but if they failed the company would likely not go under.

If we implement agile, will we be guaranteed success?

First, it is important to note that all the teams discussed in the paper delivered positive results. This means that Takeuchi and Nonaka did not have the opportunity to learn from failed projects. As there were no failures in the data set, they did not have the opportunity to compare failures with successes, to see what might have separated the successes from failures.

Accordingly, it is important to consider that the results of the study, while highly influential and primarily positive, can easily deceive you into believing that if your company implements the agile process, you are guaranteed to be blessed with success.

After years in the field, I think it is vitally important to point out that success with an agile implementation is not necessarily guaranteed. I’ve seen too many project managers, team leads, and entire teams banging their heads up against brick walls, trying to figure out why agile just does not work for their people or their company. You, unlike the experiments, have a random set of people that you start with, and agile might not be suited for them.

To simplify this logical question; if all marbles are round, are all round things marbles? The study shows that these successful projects implemented these practices, it did not claim these practices brought success.

What is better: selecting the right people or the right processes for the people you have?

Consider that your company may not have access to the same resources available to the companies in this original experiment. These experiments took place in large companies with significant resources to invest. Resources to invest in their people. Resources to invest in training. Resources to invest in processes. Resources to cover any losses.

At the outset, it looks like the companies profiled by Takeuchi and Nonaka took big gambles that paid off as a result of the processes they implemented. However, it is very important to realize that they, in fact, took very strategic and minimal risk, because they made sure to select the best people, and did not risk any of their existing units. They spun up an isolated experiment at an arm’s length.

If you look at it this way, consider that most large multinational companies already have above average people, and then they cherry pick the best suited for the job. This is not your local pick-up rugby team, but rather a professional league. As large companies with broad resources, the strategic risks they took may not be realistic for your average small or medium-sized organization.

The companies profiled selected teams that they could confidently send to the Olympics or World Cup. How many of us have Olympians and all-star players on our teams? And even if we have one or two, do we have enough to complete a team? Generally, no.

The Jigsaw Puzzle: If one piece is missing, it will never feel complete.

Takeuchi and Nonaka further compare the characteristics of their scrum method to that of a jigsaw puzzle. They acknowledge that a single piece of the puzzle or a missing piece mean that your project will likely fail. You need all the pieces for the process to work. They neglect to emphasize that this also means that you need the right people to correctly assemble the puzzle.

The only mention they make regarding the people you have is the following:

“The approach also has a set of ‘soft’ merits relating to human resource management. The overlap approach enhances shared responsibility and cooperation, stimulates involvement and commitment, sharpens a problem-solving focus, encourages initiative taking, develops diversified skills, and heightens sensitivity toward market conditions.”

In other words, the solution to the puzzle is not only the six jigsaw puzzle pieces, but it is also your people. These “soft merits” mean that if your people are not able to share responsibility and cooperate, focus, take the initiative, develop diverse skills and so on, they aren’t the right people for an agile implementation.

If you don’t have all the pieces, you can’t complete the puzzle. And if you don’t have the right people, you can’t put the pieces together in the right order. Again, you might be round, but you might not be a marble.

Human-Centered Development for the People You HAVE

As with any custom software development project, the people who implement are key to your project’s success. Implementing agile changes the dynamics of how teams communicate and work. It changes the roles and expectations of all aspects of your project from executive management to human resources and budgeting.

Agile may work wonders for one company or team, but that success doesn’t mean that it will work wonders for YOUR team. Especially if all stakeholders do not understand the implications and needs of the process or they lack the appropriate aptitudes and skills.

In other words, if these methods don’t work for your people, don’t beat up yourself or everyone else. Instead, focus on finding a method that works for you and for your people.

Agile is not the only solution …

Why do people select agile? People implement agile because they have a problem to solve. However, with the agile approach managers need to step back and let people figure things out themselves. And that is not easy. Especially when managers are actively vested in the outcome. Most people are not prepared to step back and let their teams just “go.”

Maybe you have done the training, received the certifications, and theoretically “everyone” is on board. And yet, your company has yet to see Allstar success. Are you the problem? Is it executive management? Is it your team? What is wrong?

I cannot overemphasize that the answer is as simple as the people you have. Consider that the problem is unrealistic expectations. The assumption when using agile and scrum is that it is the best way to do development, but what if it is not the best way for you?

If you don’t have the right people or the right resources to implement agile development correctly, then you should probably do something else. At the same time, don’t hesitate to take the parts of agile that work for you. 


Nonaka, H. T. (2014, August 01). The New New Product Development Game. Retrieved July 19, 2017, from https://hbr.org/1986/01/the-new-new-product-development-game

Correction Vs. Prevention

Correction vs Prevention in Software Development

The desire to prevent adversity is a natural instinct.

As humans, as individuals, we generally do what we can to avoid something going wrong. This is especially true when we invest a lot of our time and effort in a project.

Say you spend 40hrs per week on a particular project for a year. The project may become a part of your identity. At the least, you will be personally vested in a positive outcome.

The more time we invest, the greater our fear that something might go wrong. In this way, investing time and resources in a project is almost like raising a kid.

We do everything we can to set our project “children” up to avoid and prevent problems with the intent that they achieve success. Similarly, we do everything possible within reach of our finances and power (and sometimes beyond) from schooling to extracurricular activities, to ensure that our kids have the best chance in life as is possible.

Defect Prevention

Investing in our kids is a bit like defect prevention in software process improvement. Just as many theories of parenting existing, many methods of defect prevention exist. Some of them, such as Six Sigma look for the root causes of failures with the intention of preventing failures before they occur.

In prevention, we must continually ask why something might break and then fix the underlying causes. On one hand, this is very much like parenting.

A challenge in prevention work is that to be effective, detailed and good requirements are a necessity. For example, if 6 months ago someone rushed the requirements and left out a key step or valuable information, you will likely soon discover a slew of bugs in your project.

This kind of noise easily distracts us from the underlying issue that the problem came from faulty requirements. Either way, your project is delayed. And, if you have to hand over issues with requirements to another department, your work on a feature may suddenly grind to a halt.

At this point, the best you can do is note why the problem happened and resolve to do better next time. Prevention work is not always timely.

Appropriate applications for prevention work…

Prevention work generally means lessons learned for the next iteration.  We might learn how to more accurately prepare requirements during implementation. Or we might learn to focus better attention on our code after we’ve already deployed.

In prevention, we learn from problems we encounter so that we can prevent those problems in the future.  By paying more attention to difficult steps or stages, we learn what to avoid next time. Effective prevention work, in effect, creates a system that can create successful future projects, and in then it follows a company that can consistently launch successful products.

Shortcomings of Defect Prevention: Defect Prevention and Rare Events

Let’s return to the parenting analogy. Defect prevention, when applied to parenting, shifts our target goal away from our current child so that our end goal becomes setting up a system to be better parents to our future children.

I’ve chosen the parenting analogy because it clearly highlights a shortcoming of the defect prevention method. In real life, humans are highly invested in carrying out tasks that will ensure the success of already existing offspring. Unlike in prevention work, most parents don’t (intentionally) use their first child as a test case, to learn how to be successful with their future offspring.

What’s more, many families may choose only to have a single child or they may space their children 5 or 10 years apart. Defect prevention is a waste of resources if you lack a future (and immediate) application.

Prevention work is not appropriate for rare events.

Lots of small and medium-sized companies are not software companies, but companies that also do software. Software for them is a necessity, but not a business. For these companies, a software project is likely a rare event.

In prevention the entire feedback mechanism is reactive. If you want to use prevention, you will make your next project better with the lessons you learn, not your current project.

As we know, many companies may support only one or two software products and or they may only have a software development project come about every few years. Defect prevention that demonstrates how requirements could have been better 6 months ago may give comfort in the form of an explanation, but they will not solve existing and relevant problems.

Please, have a seat on the couch: explanations vs. solutions

Prevention methods may help you change your perception through the receipt of explanations, but they won’t give you solutions to an active problem. Let’s say the parent of a college student visits a psychologist to discuss problems they encountered raising their now young adult. The discussion and insight help the parent to understand and accept what they may have done right or wrong. But this explanation and acceptance will not fix a problem, it simply will change your perception of a problem. Of course, perception is significant, but it’s not a solution.

The discussion and insight may help the parent to understand and accept what they may have done right or wrong. But this explanation and acceptance will not fix a problem, it simply will change the parent’s perception of the problem. Of course, perception is significant, but it’s not a solution.

Resilience through rapid corrections…

An alternative is to simply pursue resilience through rapid corrections. Think of this like driving. Seated behind the wheel of a moving car you constantly process multiple pieces of information and feedback, adjusting your speed, direction, and so on. Driving requires constant attention and focus.

It’s a given that the closer we pay attention when driving, the better our results. Paying less attention, such as texting while driving, often results in the occasional need for bigger or last minutes corrections. Sometimes the information arrives too late or not at all and the result is an accident.

Attention + Discipline = Positive Results

This method again applies to raising children. Children and custom software development projects both require close attention. In child rearing there is a saying that “discipline is love.” Pay attention to your children, apply the rules consistently and thoughtfully, and generally speaking, they will grow up without too many bugs!

In correction (versus prevention) you pay constant attention to your custom software development projects. This focus allows you to react to problems and correct in a timely manner. Consistent attention and disciplined application of requirements result in a better end product. Rapid correction builds resilience.

Focus on reactive resiliency…

How we change as we go through production is a function of lessons learned, but also our ability to adapt, such as an outage to a new test case. Or perhaps some more automation to email for failures, so that we have the tools to pay closer attention and fine tune our responses as needed.

In correction, we maintain the goal of improving for future projects. We will still change systems and processes to fix defects, but we are less interested in learning how and why our problems exist. Instead, we focus on continually improving our near future. In correction, our immediate goal is to make today better than yesterday. And then tomorrow’s production a little better than today. Your ability to react quickly and appropriately builds resilience.

Instead, we focus on continually improving and resolving for our near future. In correction, our immediate goal is to make today better than yesterday. And then tomorrow’s production a little better than today. The ability to react quickly and appropriately builds resilience.

Deciding on Prevention or Correction

Know your company, know your goals: are you in the business of software development or are you a company that sometimes develops software?

In custom software development we come across different types of companies and goals. Some companies, say a mortgage company or an insurance company require software to function, but their business is not software. If your team is only involved the occasional software project, then it is likely more efficient fro you to focus on resilience trough correction over prevention.

From an industry perspective, it would be awesome if we could benefit from group intelligence and expect continuous improvement on every custom software project. Prevention across teams and companies is an ideal goal. Unfortunately, for the moment lessons learned on unique and rare projects cannot be easily shared. Concrete results or lessons learned are seldom shared across companies by anything other than word of mouth.

For infrequent projects, correction is best …

Until there is a method to consistently record and move beyond oral histories of software projects, the parties involved are most likely better off focusing on correcting problems in the projects we work on rather than preventing problems in future projects.

You can do both, but neither is free and prevention is most effectively applied to future projects. Decision makers must thus be careful to prioritize solutions that are best suited to your particular situation, project, and company.

If you are interested to know more about how we use correction at Possum Labs please feel free to contact me or start a discussion below.

Staying up all night to get Lucky: The Importance of Retrospectives

Retrospectives: every custom software development project should conclude with a retrospective to determine how and why luck contributed to its success or its failure.

My grandmother always said, “Better lucky than good.”

“Unless it is my money that you are spending.”

Grandmothers and many others happily share advice that interestingly promotes getting lucky over doing good. In custom software development the line between lucky and good is often quite blurry. And yet, when asked how a project succeeded, the answer is all too often, “well, I think we just got lucky!”

Is it better to be lucky or good?

It is true that often the answer depends on luck. Sometimes a well-managed project may simply encounter bad luck. Sometimes your project may suffer from bad management and yet it succeeds purely on dumb luck.

Does staying up all night to meet a deadline show dedication or is it a fool’s errand? Personally, I advocate for good over lucky. Consistent success depends upon an ability to look back and learn from your successes and from your failures.

Why conduct retrospectives? Because luck is not a strategy.

Outside of Agile development, it is rare in the quality assurance and custom software development industries, that we conduct retrospectives.

It is an even rarer event that we document the retrospectives of successful projects. When things go right we are more likely to pat ourselves on the back and move on. Often, it is only when things go wrong that we stop to take a deeper look.

Conducting retrospectives should be standard operating procedure.

I find the lack of retrospectives exceptionally shocking in the field of custom software development and quality assurance. Given that projects frequently deliver late and over budget, the keys to success seem overly taken for granted.

Everyone knows that it is generally more expensive to fix mistakes than to avoid them. And understanding where things have gone wrong in the past is an excellent way to anticipate future problems. Why then, is it not a given that every team should invest in a simple tactic that is proven to deliver continuous improvement?

Retrospectives are an opportunity, not a burden.

Retrospectives provide an opportunity to effectively record experiences and events. All too often the information exists, but only in people’s heads and if a key “head” leaves the organization then the lessons learned are lost overnight.

And even without employee turnover, once a project is completed and we move on to the next task, it is all too easy to fall into the same familiar patterns of development even if they don’t work or they are not the most efficient.

Furthermore, if we pass off our projects or our teams as simply “lucky” or “unlucky” we cannot effectively harness our capabilities. Perhaps we could be luckier. Or perhaps one day our luck will turn if we don’t understand why it exists.

Do you really want to gamble on every project?

Even when a successful custom software project can be chalked up to a fair bit of luck, we should take a careful look at how and why luck contributed in a positive manner. Maybe the actual contributing factor leads back to a particular developer or tactic and not to “pure luck.”

Or, maybe we will see that at a particular point in time we skirted disaster: if component A had been implemented, as intended, before component B, our project might have outright failed. In other words, a mistake contributed to our “good luck,” when the project really out to have failed. Can we really count on two wrongs making a right in the future?

There is high value in conducting retrospectives.

I strongly believe in the value of retrospectives. To me, the most valuable training a company can engage in is: “when and how to conduct a retrospective.” There is nothing more applicable to learn at a company than how to record and quantify experiences of the company.

A successful retrospective is one that is thoughtfully conducted and carefully documented. In this manner, the retrospective is sure to provide all stakeholders the opportunity to improve, while also providing the opportunity to learn. Retrospectives must be documented so that they can be shared widely and overtime.

The benefits of conducting retrospectives are extensive:

  • Improved communication between all stakeholders
  • Honest communication that demonstrates a commitment to transparency
  • Building a company specific repository of best practices
  • Improved ability to identify possible pain points for customers
  • Finish solutions faster and with better quality
  • More accurate development of requirements and adherence to schedule

When should we conduct a retrospective?

Retrospectives are valuable both during and at the end of a project. Conducting periodic smaller retrospectives during a project will certainly help to assess a project’s progression and make it easier to redirect or change paths before it is too late. Final retrospectives provide a valuable long-term tool and method to document lessons learned.

Periodic Retrospectives

Periodic retrospectives are a key component of Agile development, but it is a useful tactic across pretty much every domain. If you think about it, successful football coaches conduct retrospectives every time they call a timeout. And in accounting, monthly profit and loss statements provide a retrospective or a snapshot of where the company is on the last day of the month.

Periodic retrospectives may be less formal than an end of project retrospective, but they can still be extremely useful. That said, although I believe in periodic retrospectives, what I focus on this piece is the importance of the final retrospective.

Final Retrospective

Every single custom software development project, successful or not, should terminate with a complete and thoughtful retrospective.

The most valuable and necessary retrospective takes place at the end of a project. Again using the accounting analogy, an end of project retrospective is a bit like an end of year financial audit. This article focuses on the value of conducting final retrospectives.

Who benefits from a retrospective?

The IRS requires nonprofit organizations with budgets over a certain dollar value to conduct an annual financial audit. Audits are not only useful to the IRS, they are exceptionally useful to all stakeholders, from decisions makers, such as upper management and boards of directors to the employees. An audit assesses not only the financial state of the organization but how and why certain decisions or actions took place. Audits conclude with recommendations to ameliorate or improve decision making and record keeping in the future.

Retrospectives in custom software development are similarly valuable at multiple levels to audits in nonprofits. Everyone from business analysts to developers to QA testers will benefit from a retrospective. I cannot think of any viable reason that any software development team should not conduct a retrospective at the end of each and every project.

What does a retrospective look like?

Retrospectives have three key components: the people involved, the questions, and the final analysis or report. The goal of the retrospective is to review how a project has progressed from start to end, with a final assessment that includes lessons learned, and future recommendations. Ideally, all team members participate and share their answers to a set of questions.

1) The People: Who participates in the Retrospective?

Depending on the size and location of your development teams, ideally, everyone involved in a project should participate. If possible, local management should sit with each team and conduct a live assessment.

Hearing co-workers and talking together is often more productive than simply filling out a written survey. Given the opportunity to speak, members of your development and QA teams will feel that their opinion is both important and valued.

2) The Questions: What to ask during a retrospective?

(Don’t worry, it is not rocket science!)

The first three questions are fairly standard, but the final three entail an evolved process to effectively create and implement lessons learned from a retrospective.

  1. During this project, what did you feel worked well?
  2. During this project, what did you feel did not work well?
  3. In the future, what actions would you recommend taking to improve future projects?
  4. Retrospective ideas for identifying luck:
    1. Where did we get lucky?
    2. Was this really “luck” or a “lucky coincidence”?  
    3. Could we replicate this “lucky” success in the future (or avoid this failure)
  5. The final step is to work with all stakeholders to identify:
    1.  Lessons Learned
    2. Future Recommendations

3) Documentation

Without a little analysis and documentation to record the lessons learned and to make future recommendations, a retrospective quickly loses its value. Sure, giving everyone the opportunity to talk about their experience is of value. However, as we already discussed, memories are short and people move on.

To truly implement a valuable process you need to put some effort into conducting and concluding an effective retrospective. Maybe you need to bring in an outside consultant. Maybe you need to create a cohort or assign a thoughtful member of your team who writes well to type it up into a report. Whatever you do, make sure that you have a clear objective, an outline, and a process in place before you start.

Every time you conduct and document a retrospective, your team will get better at the process and the results will increase in value. Over time your lessons learned and your future recommendations will become more precise and targeted.

Retrospectives in action — an example of “where we got lucky.”

Let’s say that right as project X commences, you hire Bob to replace a retiring developer. Bob has followed his wife to a research position at the local university. Previously, Bob worked in a tech hub that also happened to be an early adopter of a new technology stack. Purely by coincidence, just as you hire Bob, another part of your team decides to implement this same technology stack (no one knows about Bob).

Your team has little to zero reasonable expectation that there will even be a “Bob” in town to hire when deciding to use the technology stack. Nor does anyone consider the possibility that you might need a “Bob.”

A few weeks in, when the new technology proves to have a few more rough edges than expected, weird crashes, flaky performance and so on, Bob is your “lucky” salvation.

How could a retrospective help avoid Bob’s situation in the future?

To start with, your team needs to identify that this is a “lesson learned.” Next time your company decides on using a new technology they should intentionally plan on securing a “Bob.” If you can’t find a Bob locally, perhaps this is an indicator that this is not an appropriate time to bring this tech to your company. Or maybe you need to set up an external recruitment plan.

Without retrospectives it doesn’t get better, it get’s real.

Over the years I’ve worked on a number projects that have fallen victim to the “Bob” scenario. To solve a problem the company adopts a piece of tech touted by a big company (aka Microsoft, Google, LinkedIn). And then only a few a months into the project we’ve found ourselves knee-deep in problems.

Each time I’ve watched a similar evolution take place. The first assumption is usually that the configuration of the technology is incorrect. Next, there is a search for bugs to be fixed in the next release. Finally, there is the decision to hire “Bob” or a team of “Bob’s” thereby disrupting operations and budgets by bringing on one or more individuals who actually know the ins and outs of this particular piece of tech.

In the end, only about two-thirds of these projects actually made it into production. Ouch. That is the reality.

Maybe it’s not the lack of retrospective, it’s the quest for new technology that is the real problem?

No, I’m not against new technology. Sometimes we need it and in the appropriate situations, new technology can work really well. Unfortunately, implementing new or custom technology is not an across the board success story.

And yet, this scenario often repeats itself within the very same companies that have already experienced a failure. This is why retrospectives are vital.

Learning from past mistakes

Companies need the ability to learn from and document past mistakes. Development teams need to have a method to memorize lessons learned and manage the turnover in both staff and technology.

The best bet for everyone is to conduct and widely share honest retrospectives. When this happens, we see what went well, what went wrong and where we got lucky. And, we can do our best to avoid, duplicate and or prepare in the future.

Successful Development is Intentional

Frankly, staying up all night to get lucky doesn’t work much on the social scene nor does it work on the professional level. What does work? Preparation. Analysis. Lessons Learned. Retrospectives.

At the end of every project, you should conduct an honest retrospective so that next time you’ll know what you can do to be prepared. Make sure that your success comes from intentional acts and not simply because “you got lucky.”

Retrospectives significantly improve the software testing process. Period.

The custom software development and software testing process is at odds with almost all other technologies. Can you imagine a caterer baking a custom order wedding cake and then testing it for flavor and attempting to go back and “fix” the flavor? Once the cake is baked, it’s done. It’s over.

Most industries test first and then build. Conducting retrospectives delivers a bit of this logic into the custom software development process and improves the software testing process. Retrospectives allow us to integrate lessons learned and avoid repeating the same or similar mistakes. Why fix it, if you can do it right the first time?

Ok. You convinced me to conduct a retrospective. Now What?

As I mentioned above, the most valuable training you can provide your team is how to conduct a retrospective. Fortunately, I am not the only one who is knowledgeable about retrospectives and many resources exist. If you would like to look in detail at what it is like to conduct a retrospective, I recommend this article: The 7 Step Agenda for an Effective Retrospective.

My last piece of advice is this: keep the environment surrounding a retrospective positive. If a retrospective turns negative, some people become defensive and afraid. Others simply tune out. Negative retrospectives lack the transparency and clear communication that are vital to constructing effective lessons learned.

Keep in mind that the purpose of a retrospective is not to critique or diminish your team or any one individual. The goal is to develop a culture of continuous improvement that benefits all stakeholders.

If you enjoyed this piece, please share and discuss!