Insights

Ready before we needed it

Ready before we needed it

The CIAM Migration Diary · Part 5 of 6

The CIAM Migration Diary, Part 5. The work people forget about, and why doing it early is what keeps the last few weeks from becoming a mad dash.

Most migration plans are written by the people who are going to build the platform, not the people who are going to run it. You can usually tell. Operability appears near the end of the plan, after go live, as a phase called something like handover. By then it is too late to do properly, and the people who will be called at three in the morning have had the least time with the thing they are about to own.

This article is about that work, and about the decision to do most of it early. A great deal of what follows was planned and prepared months ago, which means the final stretch of the programme is execution rather than invention.

Start with runbooks. Every non functional scenario has one, written and proven in our own environment before the customer's pre-production environment existed: failover, recovery, scaling, deployment, and what to look at when something goes wrong. The dashboards were defined at the same time, so the question of how we would know something was wrong was answered before the question of what we would do about it.

These are not perfect and will need to be refined but the refinement is the easy part.

We also decided in advance what to measure during the migration itself, not just afterwards. When a wave of customers moves across, we want to know within minutes whether it is behaving as expected, which means the measures, the thresholds and the people watching them all have to be agreed before the first customer moves. Deciding what good looks like in the middle of a live migration is too late.

Then chaos testing, which is less dramatic than it sounds. It means deliberately breaking parts of the platform in a controlled way to prove that it recovers the way it was designed to. It is far better to find out how the platform behaves when something fails on a day you chose than on a day you did not.

Closely related is restoration of service. Failover is the platform protecting itself automatically. Restoration is what people do when something more serious has happened and service has to be brought back deliberately. That too has its own runbook, rehearsed rather than written and filed.

How changes reach production deserves its own explanation, because it is traditionally where identity platforms take their downtime, and it is simpler than the terminology suggests. Most changes go out as rolling updates. The platform runs as several identical copies at the same time, and they are updated one at a time while the others carry on serving customers. Nobody notices, because there is never a moment when everything is down. Major changes are handled differently, using what is called blue green. A complete second version of the platform is built alongside the live one, proven while customers are still using the first, and then customers are moved across in a single step. It is more work, which is why it is reserved for the changes that warrant it.

Data migration testing started early as well, and it involves more than copying records. Identity data does not always arrive in the shape the new platform expects, so part of the work is conversion: transforming what the incumbent holds into what the replacement needs, without losing anything a customer would notice. We have been testing that conversion repeatedly for months, so that by the time it matters it is routine rather than an event. We have also agreed with the customer to run full dress rehearsals against production data, in a controlled way, before go live, to make sure it converts as well as we expect. In my experience this is where most migrations go wrong: one unexpected character or badly formed record is enough to break the conversion.

And then knowledge transfer, which matters more on this programme than most, because we are not introducing one new thing to the bank. We are introducing three at once. A new identity platform. DevSecOps pipelines the bank had not run before. And Kubernetes, which was also new. Together they mean the people who will look after the platform are learning a new product, a new way of releasing it and a new way of running it.

Knowledge transfer does not happen overnight, and for us it is definitely not a set of slides handed over at the end. It happens by doing. It started early and it is under way across all three areas. The bank's team has been involved in defects and configuration from the middle of the project (but I wish I had involved them earlier), so they have been working with the platform for months rather than meeting it at handover, and more granular boot camps are now starting.

At this customer, the team that will run the platform in production is not the team that develops it, and their jobs are different. Developers need to understand how the platform is built and changed. The production team needs to know how it behaves, how to tell when something is wrong, and what to do about it at any hour. So there are two programmes of knowledge transfer, each designed around what that team actually does. Both will continue well after go live, because the goal is confidence and capability, not throwing something over the wall and walking away.

I don't have any specific lessons learnt to share with you this week, but I hope the above gives you an insight as to why when we do migrations, they go well.

As we do this migration, it is becoming more obvious that what we are doing goes beyond customer identity. This platform is likely to become the foundation for managing AI agents that act on individuals' behalf (customers & colleagues).

An identity platform that is well run, well understood and observable is where that starts. I write about the business side of where AI is heading in my other newsletter, Past the Easy Stuff, and if straight through processing is on your agenda, there is a calculator for estimating what it could be worth to you at stp-calculator.midships.io

All of this is the reason the capacity challenge I described in Part 4 did not significantly change our plans. When the question of where the production environments would run took weeks to settle, the runbooks, dashboards, measures, conversion testing and knowledge transfer were already prepared and waiting. We were absorbing an unexpected problem rather than scrambling to build what we should already have had. That is the real return on doing the unglamorous work early.

If you want to see what a move like this could look like in your own numbers, there is a calculator for that at keycloak-calculator.midships.io

The CIAM Migration Diary is Ajit Gupta's newsletter on LinkedIn. Subscribe on LinkedIn ↗

← All insights