Tools of the Trade: Testing in with SaaS subsystems.
For the last few years, the idea has been floating around that every company is a software company, regardless of the actual business mission. Concurrently even more companies are dependent upon 3rd party services and applications. From here it is easy to extrapolate that the more we integrate the more likely it is that at some point, every company will experience problems with updates: from downtime, uptime, and so on.
Predicting when downtime will happen and or forcing 3rd party services to comply with our needs and wishes is generally difficult if not impossible. One solution to these challenges is to build a proxy. The proxy allows us to regain a semblance of control and to test 3rd party failures. It won’t keep the 3rd parties up, but we can simulate failures whenever we want to.
As an actual solution, this is a bit like building a chisel with a hammer and an anvil. And yet, despite the rough nature of the job, it remains a highly effective tool that facilitates your work as a Quality Assurance Professional.
Applications increasingly use and depend upon a slew of 3rd party services. In general, these services tend to maintain decent uptime and encounter only rare outages.
The level of reliability of these services leads us to continue to add more services to applications with little thought to unintended consequences or complications. We do this because it works: it is cheap and it is reliable.
The problem (or problems) that arise, stem from the simple nature of combining all of these systems. Even if each individual service maintains good uptimes, errors and discordant downtime may result in conditions where the time that all your services are concurrently up is actually not good enough.
The compounding uptimes
Let’s look at a set of services that individually boast at least 95% uptime. Let’s say we have a service for analytics, another for billing, another for logging, another for maps, another for reverse IP, and yet another for user feedback. Individually they may be up 95% of the time, but let’s say that collectively the odds of all of them being up at the same time is less than 75%.
As we design our service, working with an assumption of around a 95% up-time scenario feels a lot better than working with a chance of only 75% uptime. To exacerbates this issue, what happens when you need to test how these failures interact with your system?
Automated Testing is Rough with SaaS
To create automated tests around services being down is not ideal. Even if the services consumed are located on site, it is likely difficult to stop and start them using tests. Perhaps you can write some PowerShell and make the magic work. Maybe not.
But what happens when your services are not even located on the site? The reality is that a significant part of the appeal of third party services is that businesses don’t really want onsite services anymore. The demand is for SaaS services that remove us from the maintenance and upgrade loop. The downside to SaaS means that turning a service off suddenly becomes much more difficult.
The Solution: Proxy to the Rescue
What we can do is to use proxies. Add an internal proxy in front of every 3rd party service and now there is an “on / off switch” for all the services. This can also be used as a way to simulate responses under certain conditions (like a customer Id that returns a 412 error code).
Build or buy a proxy with a simple REST API for administration and it should be easy to run tests that simulate errors from 3rd party providers. Now we can simulate an entire system outage in which the entire provider is down.
By creating an environment isolated by proxies, test suites can be confidently run under different conditions, providing us with valuable information as to how various problems with an application might impact our overall business service.
Upon building a proxy in between our service and the 3rd party service, we can also put in the service oriented analog of a mock. This means we can create a proxy that generates response messages for certain conditions.
We would not want to do this for all calls, but for some. For example, say we would like to tell it that user “Bob” has an expired account for the next test. This would allow us to simulate conditions that our 3rd party app may not easily support.
Customer specific means better UX
By creating our own proxy we can return custom responses for specific conditions. Most potential conditions can be simulated and tested before they happen in production. And we can see how various situations might affect the entire system. From errors to speed, we can simulate what would happen if a provider slows down, a provider goes down, or even a specific error, such as a problem that arises when closing out the billing service every Friday evening.
Initially, the focus might be on scenarios where you simulate previous failures of your 3rd party provider; but you can also test for conditions for which your 3rd party may not offer to you in a test environment. For instance, expiring accounts in mobile stores. With a proxy, you can do all of this by simply keying off of the specific data that you know will come in the request.
In Conclusion: Practical Programming for the Cloud
This proxy solution is likely not listed in your requirements. At the same time, it is a solution to a problem that in reality is all too likely to arise once you head to production.
In an ideal world, we wouldn’t need to worry about 3rd party services.
In an ideal world, our applications would know what failures to ignore.
In the real world, the best course is to be prepared: this allows us to test and prevent outages.
In reality, the various conditions that might arise and cause outages or slowdowns are rarely tested. Working with and through a 3rd party to simulate an outage is likely very difficult, if not impossible. You can try and call Apple Support to request that they turn off their test environment for the App store services, but they most likely won’t.
This is essentially a side-effect from the Cloud. The Cloud makes it is easy to add new and generally reliable services, which also happen to be cheap and makes the Cloud an all around good business decision. It should not then be surprising that when you run into testing problems, an effective solution will also come from the Cloud. Spinning up small lightweight proxies for a test environment is a practical solution for a problem in the Cloud.