Rendered at 14:26:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
danpalmer 7 hours ago [-]
I can't help but think this isn't the best idea, it raises so many questions. How does fail over work (it failed for me!), can you safely fail over, how does product think about the stack, is it funded well enough, how do you scale a system from zero to all traffic instantly, how do you effectively keep this environment warm, how do you determine which services are critical enough to be in it, how do you manage finding the bugs in two separate stacks, and so on and on and on.
It seems like it would be better to invest in a platform that is distributed across multiple failure domains. Some companies call this a cell architecture. For Monzo this might mean true multi cloud, but they already have bare metal services for bank integrations so they are already hybrid in some respects.
It's not the decision I would have made, and I'm not sure I agree with their reasoning, but I'll also admit that I have not run a bank, I don't know all their constraints, and there are some good tech folks at Monzo. That said, I am an SRE at Google so I'd like to think I'm not completely naïve here.
Edit: after reading in more detail I'm puzzled. The justification for Stand-In and not traditional scale-out is the difficulty of data consistency (agreed!), but then Stand-In is just eventually consistent anyway, and introduces sync problems in both directions. Admittedly I'm spoilt by Spanner, but I do wonder if Monzo is being bitten by past architecture choices here.
barnabee 2 hours ago [-]
For truly critical systems and independently developed redundant or backup system that can perform at least the most important functions is clearly superior. Regardless of how much effort goes into making one implementation reliable, there's always the possibility of an issue that takes it out, or forces it to be taken offline.
I don't know how common this is in retail banking but I'm happy to see Monzo do it.
sajithdilshan 1 hours ago [-]
I think the biggest problem is the data consistency. Since they are planning for eventual consistency, should make it easier, but they most probably had to re-write code to support atomic operations and proper transaction isolations.
But the scaling problem is still there, when they switched over I assume they would start both horizontal and vertical scale of k8 pods and maybe even bump up DB instances as well which could result in a small congestion at the start of the switch over
YawningAngel 6 hours ago [-]
I'm a bit curious about Spanner. At $EMPLOYER we were going to do multi cloud with multi-region CockroachDB clusters but gave up on it because write latencies and costs were too high. Do you not get the same thing with Spanner, where technically you can have multi region but it's very unattractive to actually do?
danpalmer 4 hours ago [-]
I'm afraid I'd have to defer to the public docs on this. I've only used it internally and a bunch of that is not public.
What I would say is that it's considered to be much better than CockroachDB, but that there's no getting around physics. Spanner is subject to the same speed of light... but in every other way it does what it can do to improve things at every level. Google's use of GPS hardware for better timing is public.
ricardobayes 6 hours ago [-]
No, it's a great idea. Any systems engineer knows redundancy is good, even better if it's across providers. I see no issues with this from a system architecture point of view. As to how the implementation is done that's up to them to do right.
janpeuker 2 hours ago [-]
Agree, I know nothing about their system but in terms of clear auditability and accountability of systems I agree it's great. Every subsystem failure or inconsistency can be clearly attributed to one or the other site/stack, and the stand-in could for example apply other rules (e.g. approve only smaller transaction amounts that are accepted by an incident-only set of risk rules).
danpalmer 6 hours ago [-]
Redundancy is great, absolutely, but there are multiple ways to do it and this is not necessarily the right one.
Which is better: 3 DCs for 2 DCs worth of traffic, running the same stack but isolated failure domains... or 4 DCs for 2 DCs worth of traffic, that work in completely different ways, where 2 are left cold until needed. I can see some advantages of the latter, but not a lot.
This also isn't about fault tolerance at the code level like in, say, avionics. In those scenarios having multiple distinct implementations makes sense, but that's very different, you run all paths at the same time and check they agree etc.
ricardobayes 4 hours ago [-]
In your Google scope this all works out however if you're a mature but not yet prevalent, neobank whose reputation rides almost solely on availability, I'd still say theirs is a great approach. Even in the unlikely event that AWS throws in the towel and has a hard failure, users can still enjoy a subset of functions without elevated latency. On top of it, it might well be that user data can never leave UK, so geographical redundancy might not even be a thing here.
And yes you are completely right a completely distinct implementation is probably a no-go, but I'd think you could deploy a similar codebase with feature flags, with some frontend differences. I'm speculating but pretty sure they didn't reimplement backend functionality for this.
hurrrr 5 hours ago [-]
I've never run a bank either, but their current setup is very cost effective. A true multi cloud architecture would be "much" (a couple of 9s) more available, but I believe it would introduce other issues. The goal was to ensure the use of a subset of features for short periods of time, which is probably already above the market standard.
danpalmer 1 hours ago [-]
The cost effectiveness doesn't really add up for me either though (and again, I lack a lot of context). 1% of cost to run everything essential doesn't pass the sniff test, I assume it's much more during an actual failover. But I'm also not suggesting running any more hardware than the number of failure domains you want to be resilient to losing, and on cloud you could choose to run less and scale (assuming you aren't concerned about stockouts).
My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.
None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).
sdcfgy 3 hours ago [-]
I suspect Monzo was bitten by complexity. The network should not be assumed to be reliable and if you make a thousand things (microservices) with network between them regardless of what you think may happen, it’s entirely impossible to test it with any level of certainty. And that’s worse if you have a complete failover scenario.
Add the instability of the average cloud control plane during an outage and things get rather nasty rather quickly in reality. We’ve been through that with AWS’ numerous outages.
My company is victim of this mindset. We sacrificed tested and assured reliability on physical hardware in multiple redundant data centres for cloud and microservices. Mostly driven by Monzo’s claims, semi religious thinking and me too.
Reliability is measurably down, cost is higher, latency is higher and we cannot fail over to another provider because someone went to an AWS conference and started using all the toys. Even if we did get rid of those dependencies, during an outage the kubernetes control plane is so tied into the vendor that we’d be fucked anyway. Last time the ingress controller couldn’t change ALBs for example.
Only winner here is Bezos and the decision makers CVs. The losers are the customers. We live in the stupidest of times.
jmkni 35 minutes ago [-]
Just pointing out this is from February 2025, so has been a thing for a while
sdcfgy 8 hours ago [-]
Due to the recent outage where it didn’t work properly, this should be renamed Monzo Limp-In.
lozenge 4 hours ago [-]
Monzo I've Fallen And I Can't Get Up
janpeuker 2 hours ago [-]
Minor thing but I do like they use the word "Stand In" here in the payments / financial sense of Stand-in processing (STIP), which is usually used when a bank is offline to keep essential services such as public transport working.
tarun_anand 1 hours ago [-]
I think this is what distributed systems has a phrase for: "graceful degradation".
qmarchi 8 hours ago [-]
Something like this has been top-of-mind for us since we have a heavy reliance on AWS, which was made evident with the VPC Origin failure earlier this year.
Multi-cloud is best cloud.
Valodim 2 hours ago [-]
Almost seems like they can do anything at monzo.com
stavros 2 hours ago [-]
The only limit is themselves.
varispeed 4 hours ago [-]
> Monzo Stand-in is an independent set of systems that run on Google Cloud Platform (GCP)
Given the US Cloud Act, how Monzo is protecting customers' financial data from being accessed by foreign government without Monzo's knowledge?
Everyone does it, does not excuse it.
Also shows lack of faith in British business.
danpalmer 1 hours ago [-]
Which British businesses provide this sort of infrastructure?
I ask this as a Brit who ran stuff on a few UK providers, and ultimately moved to a cloud platform because of the combination of terrible service and lack of features. The landscape may have changed, but in my experience it was bleak circa 2020.
ascorbic 7 hours ago [-]
[2025]
greatgib 3 hours ago [-]
There are 3 things I have in mind when reading this:
- it should be so frustrating for a dev to work on a full core banking that is mostly useless and userless except maybe 1h every 3 years.
- if you build a totally different implementation to handle high traffic transaction from third parties that don't operate for days or months, i would expect it to be fragile. Like for example suddenly you receive unicode special chars that you didn't expect in labels and boom.
- last, I think that in most incident cases, the situation is not clear enough to be able to switch to the full different system. Rarely it will be your whole infra but part of it, so there is always the question to what to do in such a case, or if your main infra goes crazy with loops for example, or if you lose control because of a network, aws, infrastructure issue thr problem might be that you have no visibility on what is working and what is not, not necessarily the control to stop the faulty services, etc...
rjh29 2 hours ago [-]
"it should be so frustrating for a dev to work on a full core banking that is mostly useless and userless except maybe 1h every 3 years."
When other banks go down and Monzo stays up, even for 1 hour, that is a massive PR win and will bring in new customers. It's also been used multiple times as of the article (2025), and they are always exposing some users to it for testing.
guiriduro 7 hours ago [-]
By using another US hyperscaler for your backup platform you're still unhedged against US governance failure/breakdown.
sdcfgy 7 hours ago [-]
An increasing risk for sure.
lmz 7 hours ago [-]
Surely by being a bank, you are already exposed to that.
shawabawa3 2 hours ago [-]
Monzo are a UK bank that briefly but afaik no longer operate in the US
guiriduro 7 hours ago [-]
Exposed yes, but not fatally if you're working with alternative de-dollarised banking systems in the eurozone, for example.
IshKebab 6 hours ago [-]
Monzo is great. Rare example of a consumer banking company that makes things easy. I just signed up for their phone service in literally 3 minutes. Didn't even need to talk to anyone to port my old number. I'm going to move all my ISAs and SIPP to them next I think.
danpalmer 6 hours ago [-]
> Didn't even need to talk to anyone to port my old number
That seems like pretty low stakes – I wouldn't expect to talk to anyone to sign up and transfer to a consumer service in the UK. I don't think I've spoken to anyone for something like this in the last 10 years or so.
OJFord 5 hours ago [-]
I've never talked to someone for it, and I would've first done it in something like 2008. (It's 'always' been done by SMS, well before 'apps' that might let you (and I don't recall it on websites).)
IshKebab 3 hours ago [-]
Well I haven't moved my number for a couple of decades at least, but the last time I did you had to phone them and go through the whole "are you sure? Let me see if I can get you a better deal" nonsense.
danpalmer 2 hours ago [-]
Cancellation is different to porting a number. You'll still need to cancel your previous plan, but legislation fixed carriers holding numbers hostage a while ago.
IshKebab 2 hours ago [-]
No, when you port the number it automatically cancels the service.
rjh29 2 hours ago [-]
Compared to Lebara or other cheap MVNOs their offering is not great. While they eventually give you 30% off (after 6 years) their starting price is 30% higher.
spidermonkey23 5 hours ago [-]
they don't offer a phone contract..? this makes no sense
This is either a bot comment, or you have a very poor understanding of the UK financial market. They are quite far down the leaderboard for ISA rates and SIPPs.
StevenWaterman 6 hours ago [-]
The parent comment didn't mention anything about ISA rates and SIPPs in terms of performance, they said that it was easy. You might disagree with their priorities but that doesn't change whether it's the right decision given those priorities
rjh29 2 hours ago [-]
While Monzo are expensive their approach of rewarding loyalty (5% off per year for Monzo Mobile) is a good idea compared to the endless switching you have to do to get the best rates.
IshKebab 3 hours ago [-]
I don't think they actually are. Platform fees are capped at £250 and their default managed fund fees is 0.14% which is pretty normal. Also I deliberately want a platform that doesn't allow trading individual shares because otherwise I have to report those to my employer every quarter and I can't be bothered with that admin.
It seems like it would be better to invest in a platform that is distributed across multiple failure domains. Some companies call this a cell architecture. For Monzo this might mean true multi cloud, but they already have bare metal services for bank integrations so they are already hybrid in some respects.
It's not the decision I would have made, and I'm not sure I agree with their reasoning, but I'll also admit that I have not run a bank, I don't know all their constraints, and there are some good tech folks at Monzo. That said, I am an SRE at Google so I'd like to think I'm not completely naïve here.
Edit: after reading in more detail I'm puzzled. The justification for Stand-In and not traditional scale-out is the difficulty of data consistency (agreed!), but then Stand-In is just eventually consistent anyway, and introduces sync problems in both directions. Admittedly I'm spoilt by Spanner, but I do wonder if Monzo is being bitten by past architecture choices here.
I don't know how common this is in retail banking but I'm happy to see Monzo do it.
But the scaling problem is still there, when they switched over I assume they would start both horizontal and vertical scale of k8 pods and maybe even bump up DB instances as well which could result in a small congestion at the start of the switch over
What I would say is that it's considered to be much better than CockroachDB, but that there's no getting around physics. Spanner is subject to the same speed of light... but in every other way it does what it can do to improve things at every level. Google's use of GPS hardware for better timing is public.
Which is better: 3 DCs for 2 DCs worth of traffic, running the same stack but isolated failure domains... or 4 DCs for 2 DCs worth of traffic, that work in completely different ways, where 2 are left cold until needed. I can see some advantages of the latter, but not a lot.
This also isn't about fault tolerance at the code level like in, say, avionics. In those scenarios having multiple distinct implementations makes sense, but that's very different, you run all paths at the same time and check they agree etc.
And yes you are completely right a completely distinct implementation is probably a no-go, but I'd think you could deploy a similar codebase with feature flags, with some frontend differences. I'm speculating but pretty sure they didn't reimplement backend functionality for this.
My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.
None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).
Add the instability of the average cloud control plane during an outage and things get rather nasty rather quickly in reality. We’ve been through that with AWS’ numerous outages.
My company is victim of this mindset. We sacrificed tested and assured reliability on physical hardware in multiple redundant data centres for cloud and microservices. Mostly driven by Monzo’s claims, semi religious thinking and me too.
Reliability is measurably down, cost is higher, latency is higher and we cannot fail over to another provider because someone went to an AWS conference and started using all the toys. Even if we did get rid of those dependencies, during an outage the kubernetes control plane is so tied into the vendor that we’d be fucked anyway. Last time the ingress controller couldn’t change ALBs for example.
Only winner here is Bezos and the decision makers CVs. The losers are the customers. We live in the stupidest of times.
Multi-cloud is best cloud.
Given the US Cloud Act, how Monzo is protecting customers' financial data from being accessed by foreign government without Monzo's knowledge?
Everyone does it, does not excuse it.
Also shows lack of faith in British business.
I ask this as a Brit who ran stuff on a few UK providers, and ultimately moved to a cloud platform because of the combination of terrible service and lack of features. The landscape may have changed, but in my experience it was bleak circa 2020.
- it should be so frustrating for a dev to work on a full core banking that is mostly useless and userless except maybe 1h every 3 years.
- if you build a totally different implementation to handle high traffic transaction from third parties that don't operate for days or months, i would expect it to be fragile. Like for example suddenly you receive unicode special chars that you didn't expect in labels and boom.
- last, I think that in most incident cases, the situation is not clear enough to be able to switch to the full different system. Rarely it will be your whole infra but part of it, so there is always the question to what to do in such a case, or if your main infra goes crazy with loops for example, or if you lose control because of a network, aws, infrastructure issue thr problem might be that you have no visibility on what is working and what is not, not necessarily the control to stop the faulty services, etc...
When other banks go down and Monzo stays up, even for 1 hour, that is a massive PR win and will bring in new customers. It's also been used multiple times as of the article (2025), and they are always exposing some users to it for testing.
That seems like pretty low stakes – I wouldn't expect to talk to anyone to sign up and transfer to a consumer service in the UK. I don't think I've spoken to anyone for something like this in the last 10 years or so.