Insights

Part 3: Everyone else's roadmap

Part 3: Everyone else's roadmap

The CIAM Migration Diary · Part 3 of 6

The CIAM Migration Diary, Part 3. Every programme in the bank tests against your platform, which makes your instability their problem.

There is something about migrating an identity platform that I do not think applies to most other migrations.

Every other programme in the bank depends on the IAM platform being stable in order to do its own testing. Not in production. In the test environments, where their work actually happens. If authentication is unreliable in SIT, then it does not matter what those programmes are building or how unrelated it is to identity, because none of them can prove anything.

Environments are not plentiful. I am not sure they ever were. There is usually one shared SIT, and everybody is in it.

The first time we made a significant change in SIT, we applied our own rules the wrong way round, and we brought testing across the bank to a halt. Not our testing. Everybody's.

Our approach at that point was to route traffic to the replacement platform unless something had been explicitly identified as needing to remain on the incumbent. It seemed reasonable and it was exactly the wrong default. Our understanding of what should not yet move was incomplete (we obviously did not know this was the case), so we moved flows that other teams were relying on, in the one environment they all shared.

That was ours to prevent, and it cost several people, me included, a few nights of sleep. The lesson is obvious in hindsight. Unless you are absolutely sure of the impact, take the safe route. i.e. default is not to move unless excluded. It is not to move until proven ready.

It also taught me something less technical. My programme feels important to me because I am in it every day. There are other programmes in that bank which are just as important, and probably some which matter more. You do not get to decide the priority of the thing you interrupted.

We burnt through a great deal of goodwill in that first week and it has taken several weeks to rebuild. That process is slower than it sounds, because different teams own different components of the platform and the estate around it, so you are not repairing one relationship. You are repairing several at once, with people who now have a reason to be cautious about you.

What came out of it is a method, and the method is the useful part of this article.

Nothing moves until we have proven it ourselves. Before any consuming application is pointed at the replacement platform, we exercise the journey through the same path that application will use, and confirm the traffic arrives where it should and behaves as expected. Only then does the application move. One at a time, in a known order, with a known state.

We extended the same principle to the environment itself. We have mirrored the customer's environment inside our own in Midships, so that scripts, settings and configuration are proven against a copy before anything is attempted on theirs. Problems that would otherwise surface in a shared environment, in front of everybody, now surface in ours instead. That has probably saved more goodwill than any other single decision on the programme.

There is a limit to it, and we found that too. We proved our interfaces thoroughly, but in the shape we expected them to be called. When a consuming system called the same interface slightly differently, the incumbent turned out to behave differently in that variant, and we had never exercised it. Proving it yourself first only proves what you thought to try.

The second rule came from one of the consuming teams rather than from us, and it may be the most useful testing lesson of the programme so far. Prove it on the incumbent first.

If an application fails when you point it at the replacement platform, there are immediately two possibilities. The migration may have broken something. Or the journey may already have been broken (because of other changes outside of our control - remember Tier 1 Bank, lots of change inflight). That second one is easy to underestimate, because banks do not freeze while you migrate them. Application teams keep releasing, infrastructure changes, configuration changes, and other programmes are testing in the same environments. Without a baseline you are troubleshooting several moving parts at once.

One team put it plainly. If the journey could not be demonstrated against the incumbent first, they did not want us using their limited testing window to find out whether the replacement worked. They were right. We now establish the baseline on the old platform, then move the route and run the same journey against the replacement. That sounds inefficient, because the test effectively happens twice. In practice it saves time.

The harder part is that a testing window is rarely just a testing window. Before anyone can execute the journey, the infrastructure has to be in the correct state. Routes may need to change. Other teams may need to make configuration changes. In some cases external parties are involved, with their own lead times and their own costs. A test that takes minutes to run can take considerably more coordination to make possible.

So the scarce resource is not environment time. It is consuming team time. Every unsuccessful window spends some of it, and every time you ask a team to come back because your route was not ready, you make the next request harder.

There is another cost I had not anticipated. Reverting restores the service. It does not restore the position. After the disruption, we put the affected journeys back on the incumbent, and technically that returned everything to where it had been. But the next time something failed, the obvious question was whether our migration had caused it. That was entirely rational. Once another programme has been disrupted by your changes, your migration becomes one of the possible causes of every subsequent problem until you prove otherwise.

The fix for that was not more testing. It was stability. At one point we were ready to continue and deliberately left things untouched for several days, because I had promised an affected team a period with no further changes. It cost us time and I would do it again. Winning one testing window at the expense of confidence makes no sense when you need another ten from the same people.

Two organisational observations sit alongside this, and I think they generalise well beyond identity.

The first is what deadlines do to reasonable teams. When everybody is working to fixed dates, the first instinct when something breaks is often to establish that it was not theirs. I have watched real time go into demonstrating innocence before anyone gets to cause. That is not bad faith. It is what deadlines and accountability do to otherwise rational people.

The second is that I have learned not to confuse access to escalation with a reason to use it. Escalation works on this programme, which I said in Part 1. What I did not say is how rarely we have used it. Take every disagreement upwards and you may win the immediate point, but you also make the next conversation more formal, and you will need a favour from those same people within the fortnight. We have kept it for genuine blockers and solved everything else directly.

Senior sponsorship matters precisely because of those two things. It keeps the difficult moments focused on the outcome rather than on allocating fault, and when we have genuinely needed intervention it has been available. Just as importantly, the teams involved have been willing to put ownership boundaries aside once an issue was understood. That is not true everywhere and it is worth saying plainly when it is.

None of the discipline that came out of this is technically sophisticated. Prove as much as possible before involving anyone else. Mirror their environment in yours. Establish the incumbent baseline first. Move one journey at a time. Know the state of the environment before you ask somebody to test in it. Treat every minute you ask of another team as something you are spending rather than something you are owed. Collectively that may have more influence on the timeline than most of the architecture.

Where we have got to is this. Functionality is proven, with very few defects against the replacement platform. Performance is proven against current requirements, though a full performance test is still ahead of us. What remains is operability and the rest of the non functional requirements, which we have been proving in our own environment first.

Our biggest constraint at the moment is not identity at all. It is capacity within the cloud platforms across Asia. We need more data centres, and not only for AI.

Part 3 of The CIAM Migration Diary.

The CIAM Migration Diary is Ajit Gupta's newsletter on LinkedIn. Subscribe on LinkedIn ↗

← All insights