Risk reduction

If there is one thing that can kill a companies’ reputation, thats a poor release, either functionally or worse with a killer defect. The bigger the user base the greater the reaction. If you’ve worked on enterprise scale releases you’ll know how even a subtle change to a colour or piece of functionality can cause an immediate flurry of calls to your poor old call centre.

The solution to this is to employ a Canary release approach. Canary releases have been written about for a while, this blog on Martin Fowlers website has a great overview. The concept is simple, new features are released to a small subset of users before rolling out to the entire customer base. This gives you the ability to only p*ss off a subset of your customer base.

Joking aside, it allows to to gauge many things first – depending on your feature, that may be

• Customer reactions
• Server load and issues
• The affect on customer behaviour such as Conversion rates of visitors to purchasers.

Often this approach is associated with a blue green deployment (again via martin fowlers site – thank you!) which involves a rolling deployment of a new service – using a load balancer to redirect traffic. So lets discuss the possible approaches to getting this done.

Basic Blue Green deployment

A basic blue green – using a load balancer or api manager you switch from a running live to a running standby

Managing a gradual release in a basic blue green fashion often involves a reverse proxy or an API manager in front of your services. Lets break down the options:

API Manager

An api manager often offers a lot of additional functionality and comes at the expense of cost and complexity. Tools such as ApiGee can run into the tens of thousands a month quite quickly and add yet another network layer on top. They can however manage migrating gradually from one service to another really well – theres a short article here about that. ApiGee will use rules to direct traffic to one IP address or another. Its a broad stroke that helps migrate users to new physical servers – of course all features in the newly released service will be available to those customers – and unless you add a number of custom headers and variables the targeting is quite basic. If you have a large development however an API manager is great for managing and packaging. If you’re using one anyway it can help perform green blue you you may as well take advantage of its functionality.

Reverse proxy

If you don’t have the need for an Api Manager then a simple reverse proxy is more than capable of managing a quick switchover. Tools such as nginx, haproxy are all very capable, even managing sessions as they roll over if you must still use sessions. At featureflow we use HAProxy combined with rancher. They are both free, very solid and dependable and offer a seamless rollover experience. We also don’t have to keep double resources permanently. New containers are spun up, checked – HAProxy is switched to the new backends and the previous containers are held in reserve until we agree to roll forward – at which point the old containers are disposed. A rollback is a case of reinstating the old container. Thats great but not very gradual. So how do we make that gradual?

Blue Green dark release

In doing this you have done something wonderful, you have successfully separated the code release from the feature rollout

A dark blue green release – switch to the new service but with features toggled off

The solution is to dim the lights and perform a dark blue green release. How it works is this, you deploy your new server/service with the new features turned off. You should still migrate the new service using a reverse proxy switch over but when you get there the customer should see no change (or very little). In doing this you have done something wonderful, you have successfully separated the code release from the feature rollout. What this gives you is a number of benefits:

1. You have mitigated as much risk as possible by doing as much as possible up front
2. You have given yourself more precise control over the subsequent feature rollout

A now you can enable a specific feature within your new service for a specific subset of users. Winning.

 So now you have switched over, nothing has happened. Can we go to the pub yet? Well nearly.

Moving the control of the feature rollout to the service itself gives you ultimate control, its not just a gradual rollout from one API or one service to another, you can now enable just a single feature, or a combination of features within that one service. You may wish to try a combination of variants and target specific user types for each to assess the impact.

Using featureflow you can do this simply and safely by having your developer wrap your new feature in a line of code and then controlling it in our control panel. No API Managers required – in addition you can run dozens of scenarios – doing that with an API manager alone you might end up spinning up a new service for each AB test and each combination of gradual rollout test you wanted to try – you could end up with dozens of servers or disparate rules.

So now you can gradually turn on ‘feature a‘ in isolation – using featureflow you can simply drag a slider to increase the rollout rate to, say 10% – combine that with a role ‘of beta users’ or just a list of specific user logins – your production testers perhaps.

a rather complex rollout scenario. you get the idea.

Above and beyond ultimate control for individual rollout there are a bunch of other benefits to managing your rollout using feature controls:

1. The business team can turn on / ramp up the feature any time after the release without the engineering team getting involved (remember the code’s gone out and you’re all celebrating down the pub by now, right?)
2. You can ramp down or ‘virtual rollout’ an individual feature that may be misbehaving without having to rollout the entire service
3. You can manage not just API migrations, but internal behaviour migrations – want to add an extra item to the ‘gold’ package, it doesn’t change the API but you can roll that out gradually too.

Summary

There are a few components to mitigating risk and providing a seamless upgrade experience to your end users:

1. A reverse proxy or api manager – depending on the complexity of your solution and budget. A reverse proxy is perfectly suitable if managed well, other gateways such as Netflix Zuul are highly worth of managing switching to new code deployments and managing service migrations.
2. Dark Blue Green release – Gives you ultimate safety and control, switching seamlessly to a new software release without the new features enabled allows the development team to deploy at any time prior to actually releasing the features. It ensures that as much work has been done up front as possible.
3. Feature controls – Adding feature controls to your code allows you to manage a gradual or ‘canary’ rollout at the individual feature level. This allows you to not just gradually rollout the whole codebase or endpoint but target new features individually to specific subsets of users (such as prod testers, beta users or ‘10% of all users’) and at different times (tonight, next week when the marketing team are ready) without having to manage multiple deployments or involve engineering teams.

Sign up for a free trial account at www.featureflow.io