Staying up all night to get Lucky: The Importance of Retrospectives

Retrospectives: every custom software development project should conclude with a retrospective to determine how and why luck contributed to its success or its failure.

My grandmother always said, “Better lucky than good.”

“Unless it is my money that you are spending.”

Grandmothers and many others happily share advice that interestingly promotes getting lucky over doing good. In custom software development the line between lucky and good is often quite blurry. And yet, when asked how a project succeeded, the answer is all too often, “well, I think we just got lucky!”

Is it better to be lucky or good?

It is true that often the answer depends on luck. Sometimes a well-managed project may simply encounter bad luck. Sometimes your project may suffer from bad management and yet it succeeds purely on dumb luck.

Does staying up all night to meet a deadline show dedication or is it a fool’s errand? Personally, I advocate for good over lucky. Consistent success depends upon an ability to look back and learn from your successes and from your failures.

Why conduct retrospectives? Because luck is not a strategy.

Outside of Agile development, it is rare in the quality assurance and custom software development industries, that we conduct retrospectives.

It is an even rarer event that we document the retrospectives of successful projects. When things go right we are more likely to pat ourselves on the back and move on. Often, it is only when things go wrong that we stop to take a deeper look.

Conducting retrospectives should be standard operating procedure.

I find the lack of retrospectives exceptionally shocking in the field of custom software development and quality assurance. Given that projects frequently deliver late and over budget, the keys to success seem overly taken for granted.

Everyone knows that it is generally more expensive to fix mistakes than to avoid them. And understanding where things have gone wrong in the past is an excellent way to anticipate future problems. Why then, is it not a given that every team should invest in a simple tactic that is proven to deliver continuous improvement?

Retrospectives are an opportunity, not a burden.

Retrospectives provide an opportunity to effectively record experiences and events. All too often the information exists, but only in people’s heads and if a key “head” leaves the organization then the lessons learned are lost overnight.

And even without employee turnover, once a project is completed and we move on to the next task, it is all too easy to fall into the same familiar patterns of development even if they don’t work or they are not the most efficient.

Furthermore, if we pass off our projects or our teams as simply “lucky” or “unlucky” we cannot effectively harness our capabilities. Perhaps we could be luckier. Or perhaps one day our luck will turn if we don’t understand why it exists.

Do you really want to gamble on every project?

Even when a successful custom software project can be chalked up to a fair bit of luck, we should take a careful look at how and why luck contributed in a positive manner. Maybe the actual contributing factor leads back to a particular developer or tactic and not to “pure luck.”

Or, maybe we will see that at a particular point in time we skirted disaster: if component A had been implemented, as intended, before component B, our project might have outright failed. In other words, a mistake contributed to our “good luck,” when the project really out to have failed. Can we really count on two wrongs making a right in the future?

There is high value in conducting retrospectives.

I strongly believe in the value of retrospectives. To me, the most valuable training a company can engage in is: “when and how to conduct a retrospective.” There is nothing more applicable to learn at a company than how to record and quantify experiences of the company.

A successful retrospective is one that is thoughtfully conducted and carefully documented. In this manner, the retrospective is sure to provide all stakeholders the opportunity to improve, while also providing the opportunity to learn. Retrospectives must be documented so that they can be shared widely and overtime.

The benefits of conducting retrospectives are extensive:

  • Improved communication between all stakeholders
  • Honest communication that demonstrates a commitment to transparency
  • Building a company specific repository of best practices
  • Improved ability to identify possible pain points for customers
  • Finish solutions faster and with better quality
  • More accurate development of requirements and adherence to schedule

When should we conduct a retrospective?

Retrospectives are valuable both during and at the end of a project. Conducting periodic smaller retrospectives during a project will certainly help to assess a project’s progression and make it easier to redirect or change paths before it is too late. Final retrospectives provide a valuable long-term tool and method to document lessons learned.

Periodic Retrospectives

Periodic retrospectives are a key component of Agile development, but it is a useful tactic across pretty much every domain. If you think about it, successful football coaches conduct retrospectives every time they call a timeout. And in accounting, monthly profit and loss statements provide a retrospective or a snapshot of where the company is on the last day of the month.

Periodic retrospectives may be less formal than an end of project retrospective, but they can still be extremely useful. That said, although I believe in periodic retrospectives, what I focus on this piece is the importance of the final retrospective.

Final Retrospective

Every single custom software development project, successful or not, should terminate with a complete and thoughtful retrospective.

The most valuable and necessary retrospective takes place at the end of a project. Again using the accounting analogy, an end of project retrospective is a bit like an end of year financial audit. This article focuses on the value of conducting final retrospectives.

Who benefits from a retrospective?

The IRS requires nonprofit organizations with budgets over a certain dollar value to conduct an annual financial audit. Audits are not only useful to the IRS, they are exceptionally useful to all stakeholders, from decisions makers, such as upper management and boards of directors to the employees. An audit assesses not only the financial state of the organization but how and why certain decisions or actions took place. Audits conclude with recommendations to ameliorate or improve decision making and record keeping in the future.

Retrospectives in custom software development are similarly valuable at multiple levels to audits in nonprofits. Everyone from business analysts to developers to QA testers will benefit from a retrospective. I cannot think of any viable reason that any software development team should not conduct a retrospective at the end of each and every project.

What does a retrospective look like?

Retrospectives have three key components: the people involved, the questions, and the final analysis or report. The goal of the retrospective is to review how a project has progressed from start to end, with a final assessment that includes lessons learned, and future recommendations. Ideally, all team members participate and share their answers to a set of questions.

1) The People: Who participates in the Retrospective?

Depending on the size and location of your development teams, ideally, everyone involved in a project should participate. If possible, local management should sit with each team and conduct a live assessment.

Hearing co-workers and talking together is often more productive than simply filling out a written survey. Given the opportunity to speak, members of your development and QA teams will feel that their opinion is both important and valued.

2) The Questions: What to ask during a retrospective?

(Don’t worry, it is not rocket science!)

The first three questions are fairly standard, but the final three entail an evolved process to effectively create and implement lessons learned from a retrospective.

  1. During this project, what did you feel worked well?
  2. During this project, what did you feel did not work well?
  3. In the future, what actions would you recommend taking to improve future projects?
  4. Retrospective ideas for identifying luck:
    1. Where did we get lucky?
    2. Was this really “luck” or a “lucky coincidence”?  
    3. Could we replicate this “lucky” success in the future (or avoid this failure)
  5. The final step is to work with all stakeholders to identify:
    1.  Lessons Learned
    2. Future Recommendations

3) Documentation

Without a little analysis and documentation to record the lessons learned and to make future recommendations, a retrospective quickly loses its value. Sure, giving everyone the opportunity to talk about their experience is of value. However, as we already discussed, memories are short and people move on.

To truly implement a valuable process you need to put some effort into conducting and concluding an effective retrospective. Maybe you need to bring in an outside consultant. Maybe you need to create a cohort or assign a thoughtful member of your team who writes well to type it up into a report. Whatever you do, make sure that you have a clear objective, an outline, and a process in place before you start.

Every time you conduct and document a retrospective, your team will get better at the process and the results will increase in value. Over time your lessons learned and your future recommendations will become more precise and targeted.

Retrospectives in action — an example of “where we got lucky.”

Let’s say that right as project X commences, you hire Bob to replace a retiring developer. Bob has followed his wife to a research position at the local university. Previously, Bob worked in a tech hub that also happened to be an early adopter of a new technology stack. Purely by coincidence, just as you hire Bob, another part of your team decides to implement this same technology stack (no one knows about Bob).

Your team has little to zero reasonable expectation that there will even be a “Bob” in town to hire when deciding to use the technology stack. Nor does anyone consider the possibility that you might need a “Bob.”

A few weeks in, when the new technology proves to have a few more rough edges than expected, weird crashes, flaky performance and so on, Bob is your “lucky” salvation.

How could a retrospective help avoid Bob’s situation in the future?

To start with, your team needs to identify that this is a “lesson learned.” Next time your company decides on using a new technology they should intentionally plan on securing a “Bob.” If you can’t find a Bob locally, perhaps this is an indicator that this is not an appropriate time to bring this tech to your company. Or maybe you need to set up an external recruitment plan.

Without retrospectives it doesn’t get better, it get’s real.

Over the years I’ve worked on a number projects that have fallen victim to the “Bob” scenario. To solve a problem the company adopts a piece of tech touted by a big company (aka Microsoft, Google, LinkedIn). And then only a few a months into the project we’ve found ourselves knee-deep in problems.

Each time I’ve watched a similar evolution take place. The first assumption is usually that the configuration of the technology is incorrect. Next, there is a search for bugs to be fixed in the next release. Finally, there is the decision to hire “Bob” or a team of “Bob’s” thereby disrupting operations and budgets by bringing on one or more individuals who actually know the ins and outs of this particular piece of tech.

In the end, only about two-thirds of these projects actually made it into production. Ouch. That is the reality.

Maybe it’s not the lack of retrospective, it’s the quest for new technology that is the real problem?

No, I’m not against new technology. Sometimes we need it and in the appropriate situations, new technology can work really well. Unfortunately, implementing new or custom technology is not an across the board success story.

And yet, this scenario often repeats itself within the very same companies that have already experienced a failure. This is why retrospectives are vital.

Learning from past mistakes

Companies need the ability to learn from and document past mistakes. Development teams need to have a method to memorize lessons learned and manage the turnover in both staff and technology.

The best bet for everyone is to conduct and widely share honest retrospectives. When this happens, we see what went well, what went wrong and where we got lucky. And, we can do our best to avoid, duplicate and or prepare in the future.

Successful Development is Intentional

Frankly, staying up all night to get lucky doesn’t work much on the social scene nor does it work on the professional level. What does work? Preparation. Analysis. Lessons Learned. Retrospectives.

At the end of every project, you should conduct an honest retrospective so that next time you’ll know what you can do to be prepared. Make sure that your success comes from intentional acts and not simply because “you got lucky.”

Retrospectives significantly improve the software testing process. Period.

The custom software development and software testing process is at odds with almost all other technologies. Can you imagine a caterer baking a custom order wedding cake and then testing it for flavor and attempting to go back and “fix” the flavor? Once the cake is baked, it’s done. It’s over.

Most industries test first and then build. Conducting retrospectives delivers a bit of this logic into the custom software development process and improves the software testing process. Retrospectives allow us to integrate lessons learned and avoid repeating the same or similar mistakes. Why fix it, if you can do it right the first time?

Ok. You convinced me to conduct a retrospective. Now What?

As I mentioned above, the most valuable training you can provide your team is how to conduct a retrospective. Fortunately, I am not the only one who is knowledgeable about retrospectives and many resources exist. If you would like to look in detail at what it is like to conduct a retrospective, I recommend this article: The 7 Step Agenda for an Effective Retrospective.

My last piece of advice is this: keep the environment surrounding a retrospective positive. If a retrospective turns negative, some people become defensive and afraid. Others simply tune out. Negative retrospectives lack the transparency and clear communication that are vital to constructing effective lessons learned.

Keep in mind that the purpose of a retrospective is not to critique or diminish your team or any one individual. The goal is to develop a culture of continuous improvement that benefits all stakeholders.

If you enjoyed this piece, please share and discuss!

prepare for failure, you will make mistakes

I made a mistake. Now what?

In the United States, we often find our motivation to achieve fueled by a culture that rewards success, while commonly denying the realities of failure. As a culture, we feel terrified to fail, despite the reality that true long-term success is often built upon the ability to recognize our failures and to learn from our mistakes.
Read more

You don't work at Google and neither do I!

Improving Software Development Outcomes when you don’t work at Google

How to Improve Software Quality: Tips for Improving Your Development Process and Outcomes

Be Realistic About Your Team, Your Company Culture, and Your Development Needs

Software is long-lived and widgets have effectively infinite lifespans. Broken things tend to stay broken for years, so if our goal is to improve our processes and develop better software we must optimize tactics with our specific organizations in mind.

Much has been written about Software Process Improvement (SPI) Best Practices, but sometimes more can be achieved by avoiding known mistakes than by trying to emulate other successful organizations and their best practices. In this piece, I will look at why it can be more effective to look at what not to do rather than to focus on best practices. To start, instead of trying to copy Google’s best practices, let’s look at the idea that better outcomes might actually be achieved by copying what successful companies like Google ARE NOT doing.

Why? You don’t work at Google.

And, neither do I.

For 99.7% of developers out there, this is an accurate statement (18 million vs 40,000). And for me, it is a 100% accurate statement. By proxy, your colleagues are also not Googlers. And, although everyone at your company is surely above average, it is highly unlikely that the majority of your developers could successfully navigate the complex and often frustrating Google interview process.

So, we don’t work at Google. Does that mean we are relegated to a life of mediocrity?

We may not work at Google (or even one of these “talent magnets”), but we do know a fair amount about how top-tier companies, like Google, develop software. Their methods are considered “state of the art” and as an industry, we often set the bar high by hoping to implement their practices and achieve their success as our goal.

Development Teams around the country dream of being Googlers.

Of course, this is in many ways comparable to your softball league trying to implement the Yankees’ training regimen and expecting to end up at the World Series or at least in the playoffs. If you think this sounds ludicrous then you are correct, it is ludicrous.

Just because it works at Google doesn’t mean it will work for you…

Why then, do we set the bar ridiculously high, despite knowing that the goals we are setting are ludicrous? We want to emulate Google because we see that what they do works. From the sidelines, we have watched their success over the years. Sometimes we catch a glimpse of what it takes, not just at the level of the players, but at the level of the organization, for Google to innovate, lead, and succeed.

Google doesn’t host open spring training or publish a “spring training guide.”

We can observe Google from the outside, but most companies and their business analysts, quality assurance teams, and software developers don’t actually get to see inside Google. For those of us that are not Googlers, we never truly get to see how the machine works from inside of Google. We are not in their meetings and we don’t get to watch how the support structures put in place play out.

All we see is the final score, which generally looks pretty good.

People like to study Google and give presentations on their successes, encouraging us to try to replicate Google’s success. And let’s be fair, on one hand, this is a great idea. At the same time, what if we try to implement these “best practices” and they don’t work? Do we keep trying and devolve into No. What I advise is that instead of looking at Google’s best practices, we should instead look at what not to do. Learn from development team failures.

The Pragmatic Solution

The pragmatic solution is actually to start looking at successful companies that also do software but that are not software companies. We should then look not only at these companies’ successes, but we should also identify what we can learn from their failures.

Working with the People we Have

Your company’s real path to success is to figure out how to work with the people already on board in the organization. We can’t work like Google when we are not Google. And we shouldn’t necessarily borrow ideas from the companies that have their pick of the talent pool when we work at companies made up of everyone else. This doesn’t mean our people are inferior, but it does mean that they are our unique to our company.

We need development and project management plans that work for REAL people, for OUR people.

Brooks and the Surgical Team

This brings us to one idea that has been around for a few years, but that is often overlooked. In The Mythical Man-Month, Brooks talked about the surgical team approach. This approach takes finding a good developer or tester and then supporting this individual with a team of other staff. Brooks bases this technique on the idea that out of a set of semi-random set of developers a few of them will be 10 times more efficient and effective than the others. Given the fact that the average company has not developed Google hiring practices, it quite likely that your company will align with Brook’s numbers and a handful of your developers will have a work history demonstrating that they are more effective than their peers.

The surgical team is inherently a non-egalitarian system…

Should the few developers that have proven track records, lead “surgical teams” and how would this create an opportunity for your company to improve outcomes? For better or worse, the surgical team is inherently a non-egalitarian system that embraces the view that developers are not created equal. Instead, it focusses on building a team around your best contributors and then supporting it with the rest of the staff.

Chances are that some people will not be willing (or able) to play second fiddle to a coworker they used to be equal with, and eventually, they may need to be replaced or given a different role. In general, however, management is never easy and this is a structure that exploits and plays the hand you’ve been dealt.

Playing the hand you’ve been dealt

The reality is that most of us don’t work at Google, but instead with your average little league aka small or medium sized company. Most of us don’t have our pick of staff, we don’t work necessarily with the best and brightest. And to be honest, you and I may not fall into the category of the best and brightest ourselves. However, this reality does not mean that we won’t be successful. There is still much we can do to develop effective procedures and processes, and develop our talent to maximize the skills and the resources that we have at hand. What we can learn from Google may indeed be of value to our teams, especially if we are able to avoid the same mistakes and pitfalls that Google has learned to avoid.

Improve your outcomes by focusing on the tactics that will work for you, for your company and for your team.

A call for a "FAA" of Software Development

Mayday! Bad UX in Airplanes

Software development teams constantly learn lessons. Some of these lessons are more significant than others.
Due to the fact that there is not a universal method or process for sharing lessons learned, many lessons are learned in parallel within different companies. Many times different teams in our very own companies make the same mistakes over and over again, simply because there is not a shared repository of knowledge and experience.

Even when a developer and or quality assurance professional attempts to research when, where and why things have gone wrong, it is very difficult to find documented and pertinent information. 

These unnecessary mistakes comprise avoidable expenses to both consumers and companies, and should at a certain price point, especially a public price point, make it very useful to have a public method for accessing “lessons learned.”

Not just a report of the problematic lines of code, but inclusive of an analysis of the effects of that code (who, what, when, where, why and, how).

What’s more, in addition to the time and financial costs of problematic code, there is also risk and liability to consider. From privacy to financial health to business wealth, the risk is great enough that I propose the creation of an organization, similar to the FAA, for documenting, reporting and making software “travel” safer.

There are many examples of bad UX to be found. Just for fun, let’s look at some real life examples of lessons learned in software code in regards to Shareholder Value and Liability in Airline Travel.

Mayday: Bad UX in Airplanes

As often happens in life, not so long ago I decided I needed a change of pace in my evening routine and one way or another and I stumbled upon Mayday a show about air crash investigations. My natural human curiosity into bad things that happen to other people caught me at first, but after a few episodes, the in-depth analysis of all the various factors that cause airplane crashes really caught my attention. As a testing and QA expert, I found it disheartening to see the frequency with which bad UX is a contributing factor to airplane crashes. In most cases, the bad UX is not the instigating cause (phew!), but the stories in the show make it evident that bad UX can easily make a bad situation worse.

Scenario #1: Meaningful Warnings

For instance, on a Ground Control display, there is a 7 character field next to each aircraft indicating its expected altitude in comparison to its published height (theoretically actual altitude). In the case of one episode, an airplane intended to be at an altitude of 36o reported flying at 370, as indicated by the display which read “360-370.”

If the broadcast stopped the display would be “360Z370.” This would indicate a final broadcast of 370 versus an expected broadcast of 360. If the broadcast stopped with this discrepancy shown the display did not set off an alarm or even display a color change, just the character “Z” in the middle of a 7 digit string that implies that half of the rest of the numbers is garbage.

This piece of information on its own is not terribly exciting nor is it something that could on its own cause an aircraft to go down. Furthermore, there is not much of a reason for a system tracking altitude to simply go off.

A bad UX design process uncovered by “hidden” warnings

That is, of course, unless the button to activate or deactivate the system is placed behind the pilot’s footrest. And the display for the error message is then placed next to the button; presumably down below the foot.

No audible alarm, no flashing lights, nothing else of note to catch a pilot’s attention. In this scenario (based on a true story) the system can easily be turned off accidentally and without warning. Then just add to the mix another plane flying in the same vicinity and the unfortunate result is many dead people spread out over the jungle. The resulting system update is the addition of an audible alarm if and when the system is switched off.

Scenario #2: How to Handle Bugs

Another episode profiles an airplane crash precipitated by an airspeed sensor that bugged. As in a bug appears to have built a nest in the sensor. In this situation, the system created contradictory warnings, while also leaving out expected warnings.

For instance, the plane went from warning of excess speed to stall warnings. The inconsistencies sadly managed to confuse the pilots into their deaths.

Now it is required (standard) flight training to include how to respond when the cockpit says: “Hey! We are receiving conflicting airspeeds!”

Scenario #3: Half-Full vs. Empty

Another show profiles a UX design process failure that came about on the maintenance side. Somehow, two similar looking modules for gauging fuel reserves came into use for similar, but different models of an airplane.

Initially, it appeared that the gauges could be installed and worked interchangeably, even going so far as to updating readings. The problem is that the readings would be a bit off — well, let’s call it like it is — completely off.

If you put the incorrect model in the wrong model of plane, the gauge will read half full, when the tank is actually empty. An easy fix turned out to be putting a key into the socket making it so the two gauges are no longer interchangeable. Proof that good UX is not just about design, but about also tracking lessons learned.

Of course, the fix, unfortunately, did not get implemented until a plane crashed into the sea before reaching land (and an airport).

Documenting Problems and Solutions

When planes go down there is a loss of human life and great financial expense. This means that all of these issues have been addressed and fixed, and they likely won’t happen again. Documentation and prevention are one of many reasons that airplanes really don’t go down very often these days. Or at least in they don’t go down very often in the markets where shareholder interest and liability make downed craft unacceptable. And, significant investment is made in the UX design process.

From my perspective, the most interesting aspect of the show Mayday is that it highlights many small UX problems discovered only because people died. The death of a person is something that has a near infinite cost associated with it in Western civilization and therefore causes a very large and detailed process to get down to root causes and implement changes. Especially when it is the death of 100 people at once. Mayday is a great show for learning how to analyze problems, while also giving viewers a deep appreciation for good UX design.

Looking at the small issues that are the root causes of these airline crashes and the unremarkable UX changes made to prevent them; it really drives home the number of small UX errors that cause small non-life threatening losses that take place every day due to lousy UX. Adding up all of these small losses might actually result in a quite significant financial cost for the various businesses involved.

And although the majority of them don’t directly cause the loss of life, life might be better (safer, less stressful, more financially secure, etc.) if these small UX design flaws could be reliably flagged and a system put in place to prevent them from recurring.

Standard tracking and reporting of software failures

This brings us back to my statement at the beginning of this piece regarding the need for a body or a system to track and report lessons learned. In airline travel in the USA, we have the Federal Aviation Administration (FAA) to make sure that airline travel is safe.

The purpose of the FAA is the Following: The Federal Aviation Administration (FAA) is the agency of the United States Department of Transportation responsible for the regulation and oversight of civil aviation within the U.S., as well as operation and development of the National Airspace System. Its primary mission is to ensure the safety of civil aviation.

Now imagine, we had a Federal Software Administration, whose primary mission is to ensure the safety and reliability of software? What if we held ourselves accountable to report and to document not only when bad UX precipitated an airplane crash, but also when people would be held to a reporting standard for all kinds of software defects that cause significant loss?

Software Asset Managment (SAM) already exists as a business practice within some organizations, but not in enough. And there is still not any central organization to pull the information documented by business with successful SAM practices.

In 2016, the Software Fail Watch identified over a billion dollars in losses just from software failures mentioned in English-language news sources and they estimate that this is only a “scratch on the surface” of actual software failures worldwide. There is much to be debated here, but if a US based or even an international agency simply started to record failures and their causes, without bothering to get into the official business of issuing of guidelines, the simple acts of investigation and reporting could create an opportunity for significant and widespread improvements in design.

I think we can all agree that our industry could greatly benefit from the creation of an efficient and productive repository for the sharing of lessons learned. The strategy of sharing lessons learned, lessons that are often relegated to oral histories shared between developers would greatly benefit our industry.

Companies may not initially be motivated to report details of their problems, however, the long-term benefits would surely outweigh any perceived negative costs. As an industry, software development can only benefit from sharing knowledge and lessons learned.

Group intelligence is known to exceed that of individual members, and in a world of increasingly unanticipated scenarios and risks, we need able to effectively anticipate and solve challenges around the security and uptime challenges faced by many companies. As an industry, perhaps instead of fearing judgment, we can instead focus on the benefits of embracing our imperfect nature, while facilitating and co-creating a more efficient and productive future.

As an industry, perhaps instead of fearing judgment, we can instead focus on the benefits of embracing our imperfect nature, while facilitating and co-creating a more efficient and productive future.

If you enjoyed this piece, please share it. 

If you have something to say, please join the discussion!