Patrick McKenzie (@patio11) shares his remarks to the Bank of England on critical vulnerabilities in financial infrastructure. Below is his presentation for your perusal.

Sponsor: Mercury
This episode is brought to you by Mercury, the fintech trusted by 200K+ companies — from first milestones to running complex systems. Mercury offers banking that truly understands startups and scales with them. Start today at Mercury.com 
Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group, Column N.A., and Evolve Bank & Trust; Members FDIC.

Timestamps

(00:00) Intro
(01:48) The importance of implementation-level understanding
(03:00) Single points of failure
(04:25) Can a 22-year-old engineer close all the banks?
(05:18) The CrowdStrike incident: A case study
(08:34) The culture of "shut up and shuffle"
(09:54) Blameless postmortems
(12:25) What actually happened during CrowdStrike
(18:01) Five whys: Root cause analysis
(19:03) How software monocultures are created
(22:54) Understanding endpoint monitoring software
(25:25) Distributed systems and the nature of CrowdStrike
(31:22) The economics of software monocultures
(33:29) Why wasn't there defense in depth?
(37:05) Why was recovery so difficult?
(40:32) The domino effect across financial institutions
(43:36) What went right: Electronic systems remained up
(45:10) This was a near miss
(49:29) Potential policy responses
(54:03) Switching gears: Stablecoins
(01:01:37) The elephant in the room: Tether
(01:15:32) Who loses if Tether implodes?
(01:16:59) AI and the future of trading
(01:26:47) AI risks in the trading space
(01:30:41) Closing

Transcript

 [A brief note on our transcripts] 

Hideho everyone. My name is Patrick McKenzie, better known as patio11 on the Internet.

I recently received an email from the Bank of England, which was somewhat surprising, because I'm not normally the kind of person that gets emails from central banks. The Bank of England said "Dear Mr. McKenzie, we were wondering if you could please give us a private seminar on things that software engineers understand that central bankers might not."

And so I skedaddled down to Threadneedle Street over in London to give them a private seminar. The bank graciously gave consent to have me give a derivative of that talk to the public Internet.

I'll make two mandatory disclaimers. One, these are only my own views and perceptions, and they are not endorsed in any fashion by the Bank of England. Two, I previously worked at Stripe, and I'm still an advisor there. Stripe does not necessarily endorse what I say in my own spaces or in my own capacity.

So I'm not necessarily the kind of person that you would expect to address a central bank. I can barely tell CAMELS from horses, and I haven't met Basel yet, but I assume he's a very nice chap. In fact, the one time I thought I would come down to Threadneedle Street was due to the demonetization of the old pound note. You had to exchange it with your high street bank or with Bank of England directly. Not being banked in the United Kingdom, I thought I would have to come down, but eventually talked a teller into exchanging seventy old pounds for new pounds.

But I have learned a few things about a few things in the course of a long and interesting career around financial infrastructure. I’d like to share how things are actually built and, particularly, how they break.

The Importance of Implementation-Level Understanding

I'd encourage you, as policymakers and people who are responsible for this for the entire nation of the UK, to spend more time with engineers and other people who are close to the implementation level of the policies that you oversee. I think that often in policy circles, we spend our heads a bit in the clouds and devalue implementation into being somebody else's problem.

But the financial infrastructure of the world—the technical substrate that we all exist on and that our economies exist on—cannot be separated from the implementation level. You cannot understand what is actually happening without some level of technical understanding. And so I think it's incumbent upon various actors to both have engineers and other experts in the room when these policies are drafted, debated and discussed, but also to make connections with people such as myself or any of the other million engineers that you can find on the Internet, to get some color on the actual detail of the things that are being built.

And so with that, you've asked me to address three sub-topics under the broad heading of software that undergirds financial markets. And I've brought my own personal views on them to you.

Single Points of Failure

You probably have forgotten more about financial stability and systemic risks than I will ever know. But we do have one interesting concept in engineering land about how failures tend to propagate out from where they happen. There is one bit of engineering jargon I'd like to leave you with: a Single Point Of Failure (SPOF).

Broadly, when you're scaling engineering systems, you have two choices. You can either scale vertically—just buy an increasingly big server to run your database, it'll work out—or scale horizontally. Have multiple independent copies of a thing, and when you want to scale, just increase the number of copies that you have. One advantage from a resiliency perspective of scaling horizontally is that you can lose some of the nodes in the system and have that be a non-event for the overall availability of the system, because you have many nodes backing them up.

There are some things in systems which have the characteristic where there is only one copy of them, a single point of failure where the unavailability or downtime or brown time of that particular node means the system is likely to grind to a crashing halt. Sometimes this is unavoidable. Sometimes this is a considered engineering choice. The cost of resilience is a cost, and sometimes it's not a cost that you want to bear for a particular application.

But there's a dangerous pattern where you have a single point of failure and are not aware of the fact of having that single point of failure. I want to present an example to you.

Can a 22-Year-Old Engineer Close All the Banks?

Help the American out with a remedial civics lesson. Can the monarch of England unilaterally choose to close the banks? I was told that yes, they can, but it hasn't happened since Charles II, and that authority has been circumscribed by Parliament in the intervening years. Okay, fair enough.

Question for you now: Can anyone in this room unilaterally close the banks? Do you have a big red button sitting on your desk somewhere where you slam it and all the deposits and loans stop immediately? No? I was hoping to see that.

All right, follow up question for you: Can a 22-year-old engineer close all the banks? What's your answer? And why do you think you're so confident in that answer?

In the United States, we thought we were pretty confident that no 22-year-old actually had the authority to close all the banks. And it turned out they had the power to do something which looked strikingly similar to that.

The CrowdStrike Incident: A Case Study

I want to read you a headline from July 2024—ancient history—from the Wall Street Journal: "Major tech outage grounds flights, hits banks and businesses worldwide."

I want to talk about CrowdStrike and CrowdStrike Falcon. They resulted in a large portion of the United States banking infrastructure hitting a Blue Screen of Death in Microsoft Windows, thus failing in its goals for being available to the population of the United States and the worldwide economy that relies upon us.

If you haven't been familiar with the acronym BSOD, it stands for engineering jargon, Binary Search Over Device. No, I lied—it stands for Blue Screen of Death. You're probably familiar with this if you're of a certain age like myself. We very frequently saw these in the Windows 95 days and Windows 3.1 days.

We haven't seen them since those days in any quantity. [Patrick notes: Actually it was probably Windows XP, which switched to the NT kernel, that saw the major inflection point in stability. This was in the early 2000s. One went from several involuntary system resets a day to a handful monthly.]

But suddenly, on that day, a large portion of the largest U.S. banks—the Wall Street Journal cites five of the top ten—saw the Blue Screen of Death on almost every PC in certain lines of business, starting at a particular point in the morning on a Friday and continuing through basically the end of the day, completely stopping the ability of those lines of business to work.

The Limited Official Response

Now, I regret that you're going to have to take some representations I make here on faith. I would hope to be able to point you to a postmortem conducted by, for example, the Federal Reserve Board of Governors. Unfortunately, the Fed is a little bit busy at the moment, as you might see if you've read headlines such as "Fed Independence Reaches Its Moment of Truth as Supreme Court Weighs Cook's Fate," and other various sessions happening in Washington at this time of year.

But be that as it may, there was actually one postmortem conducted by a financial regulator on the CrowdStrike outage of July 2024. It was actually conducted by the Financial Conduct Authority here in the UK. That postmortem says: "As CrowdStrike is widely used, we saw varying degrees of operational impact on regulated firms with no sector more impacted than others and minimal consumer harm."

I'm going to lightly push back against the FCA's characterization that there was minimal consumer harm. I think there was actually quite a cognizable amount of consumer harm to consumers, particularly in the United States of America. And moreover, this was a very near miss sort of situation. We got lucky in having only as much impact as we saw.

Why Media Coverage Focused on Airlines

Much of the media coverage of this issue focused on the grounding of airline flights, partly because that is maximally disruptive to business travelers on a day when they're traveling, and partly because it's very easy for people to verify independently that airline flights did not happen, where it's relatively difficult for journalists to verify independently that things like withdrawals did not happen. [Patrick notes: There is no FlightAware that aggregates routine operational failures in banks and could be used to substantiate a claim that they rose by 60,000% on Friday.]

This epistemic logic of how media happens and its complicated relationship with the financial industry is an interesting thing to keep in the back of your head to conduct sense-making as the state, because the newspapers are probably not lying to you per se, but they're also not telling exactly the truth as regulated firms understand it.

The Culture of "Shut Up and Shovel"

With the greatest amount of deference to the FCA, I don't think that the culture at financial institutions necessarily tells you their internal truth 100% of the time. I think that the culture at financial institutions interacting with the regulators frequently trends towards a "shut up and shovel" when an issue is discovered—where if we were to calculate to the depth of how bad things got in a particular engineering incident, that would invite regulatory scrutiny, invite an inquiry, and invite throwing this firm or a regulator or some other powerful institution under the bus.

And so it's always easier to say, well, no big problem happened here, and to continue saying that until the evidence becomes absolutely undeniable that there was in fact a gigantic problem.

If you want to look close to home, the experience of the UK subpostmasters who were sent to prison essentially by a non-functioning software system is extremely instructive here. The UK Post and the consulting firm that created the software system were adamant for many years that there was no problem. "Horizon software is the best software. It was adding numbers. How hard could that be?" And a tremendous miscarriage of justice happened.

Blameless Postmortems

We have a relatively new culture in engineering land for what's called blameless postmortems, where instead of conducting an inquiry and finding whose political career we need to sacrifice or who we need to throw under the bus for some engineering issue, we focus on just getting to engineering truth as quickly as possible and as reliably as possible. This decouples the quote-unquote "accountability" from the statements of engineering fact. Our goal is to get systems back up and running from the condition of being under incident to the condition of being working again as quickly as possible.

We collect a reliable corpus of information from those incidents that can inform the future craft of engineering, both at our firms and in the industry broadly. And we establish this common language for how we talk about incidents, both during the progress of the incident and afterwards.

I think we will find that many financial technology companies have conducted many, many more incident reviews than you would guess, and that these are not considered supremely sensitive. In fact, some of the best information you'll find about the CrowdStrike Falcon vulnerability was the public postmortems that were published by CrowdStrike and by Microsoft respectively.

An Olive Branch Approach

I bet that supervised firms would happily give you their best log of incident reviews at a variety of scales—how impactful the incident was, how concerning it was, how much remediation was required, and how much customer harm actually happened.

I think they will be particularly interested in giving these to you if you extend them an olive branch and say, "Look, we're not trying to bust your chops here. We are all on the same team, and we'd like to understand both what generally happens when the excrement is hitting the oscillating blade and how you generally act with regards to that, because we also have responsibilities when you are acting to do incident response."

We would like to better understand the landscape that we're working on so that we can both handle our own responsibilities as the central bank and also socialize these responses within the broader community of practice. We think there is something that we can actually learn from financial technology companies. We think that some of the more old-line, extremely storied traditional financial companies could learn something of technical practice from you. We would like to collect this and make some sense of it.

I think you would get a huge amount of uptake on this outreach.

What Actually Happened During CrowdStrike

I wrote for Bits About Money on July 31st a postmortem of what I understood to have happened during the CrowdStrike incident of July 2024. I think this postmortem holds up very well after almost a year later. There were subsequent postmortems conducted both by Microsoft and by CrowdStrike, and they don't probably change the upshot here.

While I was comfortable saying the names of the affected institutions in the room with the Bank of England, I'll leave it carefully vague for the broader audience.

Impact on Teller Operations

I understand one of the largest financial institutions in the world to have entirely grounded their tellers for the entire day of Friday due to the inability of their tellers to log into their systems because the systems were completely down—Blue Screen of Death. And that meant that all bank functions which were gated on speaking to a teller could not happen.

The most consequential of those functions on a Friday is large cash withdrawals, particularly by business customers. Many people may not appreciate that businesses still do large cash withdrawals to fund payday. Given that 97% plus of paychecks in the United States are direct deposit at this point, and of the remainder many of them are paid as checks rather than in cash [Patrick notes: I said 93% in the audio, mentally conflating the correct stat and the 4% still on paper checks], there are particular subsectors of the economy which are still heavily cash-based.

For example, in my hometown of Chicago, for a variety of reasons, the construction industry tends to be heavily cash-based. That was the thing that drove me to look all over Chicago for options for withdrawing a large amount of cash on a Friday, because I had a milestone payment committed to a general contractor who himself had committed to paying his employees on that day. From a justice angle: workmen deserve their wages. Just because financial infrastructure was being impacted, it would be a bad thing to deny people money that they had earned from the practice of their profession.

I eventually found a workaround, but that's neither here nor there.

Cascading Effects and Communication

There are other lines of business at these banks that were less impacted, and some lines which were not directly impacted but which had cascading impacts. One thing that the banks did relatively effectively was communicating via non-computerized systems of communications. This could have been through people's personal handsets or through telephone trees, as I understand happened at one of the largest banks.

It is difficult to keep in contact with a branch network of several thousand branches during an almost complete interruption of your ability to see all the screens in the organization. And yet people had done the right amount of scenario planning for this—to have printed lists of what the various phone numbers for the bank branches were and to organize call trees to get the information out to the bank branches.

"Do Not Close Branches"

One of the things that They got out to the bank branches was the instruction—my sources left carefully ambiguous as to who "They" were—"We understand that you are unable to service customers who are coming into the branch right now, but do not close branches, because that would cause a panic."

We want to avoid the panic because the panic would cause people to attempt to increase their withdrawals from the banking system, which is dangerous for all of the usual reasons, plus the operational reasons. There's a limited amount of cash in the ATMs right now. If people surged to get cash out of the ATMs, they will deplete the cash on those ATMs. We will be unable to restock it from cash that is on hand because we cannot ledger the withdrawal of cash from the vault or elsewhere on site to the ATM right now because the system capable of making ledger entries is down.

Given that banks don't love the notion of having bank staff walk out with $200,000 in cash under their arm without creating a record of that contemporaneously, we couldn’t replenish the ATMs on the day of the event.

ATM Depletion and Fraud Detection Issues

Indeed, one thing that we saw all across Chicago was that banks saw higher than expected demands on their cash at the ATMs and had to turn people away on that basis.

Another thing that happened was that there was a sudden spike of usage of ATMs from people with accounts that did not use ATMs frequently because they had previously seen the teller for those withdrawals. This caused some automated systems to heuristically flag those accounts as potentially being under attack, subject to fraud, or subject to other sorts of bad action the bank wants to interdict.

A particularly large bank which has invested billions of dollars into its IT over the last few years had an automated system which closes a fraud loop automatically with the customer, allowing them to receive an email or text message and thumbs up or thumbs down that the transaction actually was initiated by themselves. In the course where they didn't initiate the transaction, it clears the fraud alert and generally speaking allows them to go through with their withdrawal at the ATM.

Unfortunately, we discovered on that day that the system where you click the big green button in the email—you get redirected to a subdomain of that financial institution, that subdomain runs on Microsoft Windows, and that computer was hard down as well. And so we saw increasing demand on some subsystems where those subsystems failed due to the same thing that was causing the increase in demand for those subsystems.

Five Whys: Root Cause Analysis

We have a concept in doing root cause analysis in our postmortems called Five Whys. This is variously attributed in U.S. engineering culture or sometimes attributed to Japan. Having been a Japanese salaryman, I think many of my colleagues would have said this is not specifically a Japanese thing—this is just the common practice of engineering at firms that are thoughtful.

Here are five questions you might want to ask yourself:

1. Why did several banks share a software monoculture?
2. Why was this configuration change not caught in testing?
3. Why was this configuration change so hard to roll back and/or mitigate?
4. Why did this configuration change have such a large blast radius?
5. Why did this configuration change substantially disrupt important banking services?

"Important banking services” are an area of regulatory concern by the Financial Conduct Authority and other regulators, and their availability is an obligation of chartered institutions.

How Software Monocultures Are Created

I want to read you a brief excerpt from a recipe for software monocultures. There exists in the United States a poorly understood regulator, and it is a sort of meta-body sitting on top of five American financial institution regulators. It's called the FFIEC—Federal Financial Institutions Examination Council.

What the FFIEC exists to do (and it is a creature of statute) is that there are many regulators in the United States that have the primary task of going into banks and other financial institutions and ensuring that they are up to snuff via quarterly examinations, most typically. We wanted those examinations to be standardized across the various regulators so that institutions wouldn't be incentivized to pick the regulator with the weakest controls, and so that there could be shared learning, and so that we could avoid duplicating the amount of work done across the United States federal government and state governments by just writing down all the rules in one place once.

The IT Examination Handbook

The FFIEC published the Information Technology Examination Handbook: Information Security in September of 2016. I believe that this document or its successor document is sort of the Bible for conducting these examinations, specifically in regards to information security, which is one part of a large regulatory tapestry in the United States.

This tapestry is not specifically a creature of statute. This tapestry is not, to my understanding, specifically a creature of regulation. This is a phenomenon of the fact of advice that regulators give for passing your examination with flying colors. The recommendations that it gives examiners for things to look for are phrased in the terms of recommendations, but they wander their way into becoming effectively having the force of law behind them, simply because a suggestion from your regulator with enough force behind it is—unless you really want to put your points in that relationship to denying the suggestion—effectively an order.

Endpoint Monitoring Requirements

Here's the thing that the FFIEC understands to be a suggestion: "Management should implement defense in depth to protect, detect, and respond to malware. The institution can use many tools to block malware before it enters the environment and to detect it and respond if it's not blocked. Methods or systems that management should consider include the following..."

There's a list of 15 bullet points at this place. I'd like to read you one bullet point in specific: "Monitoring for unauthorized software and disallowing the ability to install unapproved software."

We mentioned CrowdStrike before. CrowdStrike Falcon is endpoint monitoring software. Endpoint monitoring software, simply described, is sort of like antivirus, except the antivirus that you know (and there's probably no person who actually loves their antivirus software, but you've probably seen it run on your computer at some point) is designed to protect your machine.

In the context of a large financial institution, you might be managing as the IT department a fleet of 100,000 machines. You know to a moral certainty that somewhere within that fleet right now, there exist problems. You think it's not possible to zero the problems, and it's not even desirable from the point of view of the institution to say, "Yes, my machines have no problems on them."

The Purpose of Endpoint Monitoring

I want to institute a process which will detect which systems under my control are out of compliance with my general mandates and to quickly bring those systems either back into compliance or limit the access those systems have to the rest of my network and other subsystems, such that, for example, an attacker that gains a foothold on one of our systems through someone installing malware inadvertently or hooking up an unauthorized USB disk to the computer can't pivot from that system to more valuable systems within the enterprise.

That pattern of attack—where first you've compromised someone in a customer support or sales role, and then you pivot to the computers that can actually physically move the money—is something that you see over and over again in, for example, North Korea's exploitation of financial companies and the cryptocurrency community.

This endpoint monitoring software is designed to run on all of your computers all of the time.

Why Not All Systems Were Affected

In fact, CrowdStrike Falcon did not execute on all of the computers all of the time in the U.S. financial industry. And it's a good thing too, because they all would have been bricked on one particular Friday. (“Bricked” is engineering slang - the computer becomes an expensive paperweight. “Bricked” usually means permanently dead, but these could be recovered with extensive manual intervention.)

Indeed, it required an extensive remediation, and it's one reason why, well, CrowdStrike reckons that the outage lasted only 90 minutes, it took down much of the financial system for the entire working day through the hours of Friday evening, with some teams working throughout the weekend to bring up systems almost entirely for Monday.

Who Gets Endpoint Monitoring Installed?

So endpoint monitoring is working on some subset of systems. How did we decide what that subset was? Well, you would have to ask the actual financial institutions how they decided what that subset is. But let's talk a little bit about the push and pull within an organization and who gets this installed on their computers and who doesn't.

You want to install it on the most vulnerable systems first. The most vulnerable systems are ones that are either exposed directly to customers or exposed to staff who directly interact with customers, particularly where the staff are—I have to phrase this bluntly—institutionally less trusted.

In the financial industry, there are customer-facing staff who are institutionally less trusted, such as, for example, tellers. Tellers are in the physical proximity of customers. They pass documents and sometimes little electronic devices over the desk in the routine practice of business. If you are a security engineering manager at a large financial institution, you would say tellers have perhaps the most vulnerable computers in this entire enterprise. I would definitely want to protect the tellers' computers. And that is indeed why we saw at some of the largest financial institutions in the world, all of the teller computers go down.

The Trading Department Exception

Now, contrast that with, say, the trading departments of large American banks. Traders are high status within the institution, and there is a bit of a speed premium in executing their work in a way that there's not a speed premium in executing teller work. Certainly, tellers want to expeditiously help the customers that talk to them, but no teller has ever said, "For want of two milliseconds, I lost millions of dollars." And traders say that all the time.

And so you might imagine traders institutionally pushing back against, "Well, okay, IT security wants to install some cruft on our laptops, but we want to trade from our laptops, and they can get around to that when they get around to that. We're not very enthusiastic about this requirement. And every time it comes up in a meeting, we're going to stall for as many months as we can."

One would predict as time goes to infinity, the institution will bring all laptops into compliance. But in actual fact, in July of 2024, the trading operations at the largest financial institutions in the world were substantially less impacted than the teller operations.

You could come up with a theory under which that is not good from an IT security perspective. I'm Switzerland with respect to that theory. I think from the perspective of the central bank of the United Kingdom, you're very glad that large U.S. counterparties were able to continue buying and selling pound-denominated assets on that particular Friday that was otherwise a trading day, because otherwise there would have been systemic risk to the United Kingdom, even if no computers in the United Kingdom were directly impacted by the CrowdStrike vulnerability (which, by the way, is contrary to fact).

Distributed Systems and the Nature of CrowdStrike

There's a yarn in the engineering community that the words "distributed system" mean that a computer that you don't know exists can break in a fashion that renders your own computer unable to do work. In this fashion, CrowdStrike does run a distributed system, and they don't run this distributed system maliciously.

How Endpoint Monitoring Works

The thing that an endpoint monitoring system does is it bundles two things:

First, it's software that is similar in character to antivirus, continuously scanning your system for threats and reporting to the centralized IT team or security organization when it is found that your laptop is out of compliance. One thing that you could do to put your laptop into a state of non-compliance is to ignore the warnings that you have been given and install, I don't know, Minecraft to play it with your family. Minecraft has many great things about it. I love playing it on my personal laptop. But your security department has not rigorously approved Minecraft and all the mods that could potentially be installed with it as a thing that can be safely installed on a laptop that touches money.

Given that there's no business purpose to have Minecraft on your laptop, the security department would greatly prefer that Minecraft not exist on any laptops. So if Minecraft is installed on any of 100,000 laptops, flag it immediately to security. So that's half of the job of endpoint monitoring.

The Software-as-a-Service Component

The other half is that endpoint monitoring software is software-as-a-service, and the service component is that in lieu of or in addition to your organization having a threat detection and threat modeling team, CrowdStrike is going to employ people who aspirationally are the best in the business at detecting emerging threats. They are going to reduce those emerging threats to signatures that will allow a computer to efficiently determine whether an emerging threat has interacted with the computer or not.

That emerging threat might be state-sponsored hackers. It might be malware which is circulating around the internet. It might be a virus that is going around, etc.

The Testing Gap

As of July 2024, the signature definition files that CrowdStrike created were tested to a far lesser degree than CrowdStrike Falcon itself was tested. And they could be pushed out in an automated fashion to all of the enterprise customers that have CrowdStrike Falcon.

To the best of my understanding from reading the postmortems, there was no capability for the IT teams at the enterprise customers to review or reject the new definition files that were pushed out to them. Those were just fanned out to all of their endpoints automatically in the ordinary operation of the system.

And that's what gave one engineer who was relatively young and probably regrets the day that they did it the ability to push out a definition file which caused many computers to interact poorly in a very synchronized fashion over many different firms that have no obvious connection with each other other than being under similar regulatory regimes.

The Economics of Software Monocultures

It isn't just the fault of the regulatory regime that we created a monoculture in the deployment of endpoint monitoring systems. It's also an unplanned interaction of how software businesses work.

A question that I've gotten from the Bank of England is: Is it effectively the case that there's only one endpoint monitoring system in the world? No, there are probably at least four providers that you can go and buy that from. However, those providers have sales and marketing teams and they are in vicious competition with each other, attempting to divvy up the entire world.

CrowdStrike's Sales Strategy

A particular thing that CrowdStrike was very good at was having their sales and marketing teams realize: "We've had some amount of success with selling to U.S. financial institutions, and we've come up with a theory of why we are getting that success. They have particular needs. We are good at explaining those needs to them, creating collateral which explains that these requirements from regulators mean that you need to buy something. And we will describe to you exactly why our thing that we have to sell is the thing you need to buy."

One thing that you're going to be required to do during this whole regulatory dance is to write up a risk analysis and then describe the countermeasures you are taking that you have identified in your risk analysis. We've done a lot of that work for you. You can copy-paste large portions of our template risk analysis to talk about how malware is the risk to your systems and then identify that CrowdStrike Falcon is a responsive control to malware.

Market Segmentation by Industry

When you iterate this game for a while, what happens is that the various providers of endpoint monitoring systems divvy up the world based on industry and customer archetype and have effectively cut the cake such that lots of the frosting ends up in CrowdStrike's slice, where frosting is the U.S. financial system.

That meant that even if you view it on a worldwide lens, endpoint monitoring might be a vibrantly competitive market. Seen from within the perspective of a particular industry, it tended far closer to monoculture than the aggregate view would suggest.

Why Wasn't There Defense in Depth?

You might expect there to be something like defense in depth. Why wasn't there a second engineer that caught this issue? Why wasn't it caught by automated testing?

CrowdStrike, like essentially every professionally operated engineering organization in the U.S., makes extensive use of automated testing. However, at the time of this incident, they did much more automated testing with regards to the review of code rather than the review of data.

If we were in a computer science class right now, we could have a discussion on whether a malware vulnerability signature is closer to code or closer to data, but it is treated by the system as data, and so we will call it data.

The 21st Parameter Problem

It turns out that in this bit of data that was pushed down to hundreds of thousands of clients of the system, there were 21 input parameters, and the 21st one had a slight issue. In almost all deploys to date, that 21st parameter had been a wildcard—match anything for whatever the 21st parameter defined. For the first time ever, this new definition included a non-wildcard there.

That caused the system that was reading this file and attempting to interpret it to read memory out of bounds within the startup of the CrowdStrike Falcon software.

Why CrowdStrike Runs in Kernel Space

Now, CrowdStrike Falcon is designed to run in kernel space and is designed to run on boot. Why? Well, it's security software.

Broadly speaking, modern operating systems have two places where programs can run. They can run in the kernel, or they can run in what we sometimes call user land.

Programs that are operating in the kernel have the most ability to inspect things that are happening on the system and sort of the highest level of privilege afforded to them. Programs that are operating in user space are, to an increasing degree, firewalled from each other (firewall not being used as the term of art here). They are discouraged from reading each other's information. They are discouraged from interacting with each other except in ways that are blessed by the operating system manufacturer and/or the user and/or the IT department that has configured the computer. And they broadly operate with fewer permissions.

The reason that CrowdStrike chose to make its security product operate in the kernel is for a couple of reasons. One is that security products are in a constant cat-and-mouse game with bad guys. One worry is that if you are not in the kernel and the bad guy figures out a way to get their vulnerability into the kernel, then that is the ball game. They will be able to introspect to your attempt to look at the current operation of the system and defeat it by operating at a plane at which you simply cannot introspect the operational system anymore. (I'll spare you a long list of very fun papers in theoretical computer science about this, starting with the problem of trust.)

CrowdStrike made the decision to have this program execute in the kernel. To be fair, Microsoft was relatively scathing in its postmortem about that decision. They believe that a properly executed software security program can still execute in user space and leave the security of the kernel to Microsoft alone. The reason Microsoft would prefer that you do that is that when things go bad in the kernel at boot, there are not great options for recovery.

And so indeed, we saw the Blue Screen of Death. There was no option for recovery whatsoever.

Why Was Recovery So Difficult?

Okay, so we've defeated our defense in depth via automated testing. We've defeated the potential two-man system for approving a change to the computer code, because in high likelihood, a young technician was able to push out this file without much review from other engineers at all.

Why weren't engineering teams at the various financial institutions capable of quickly bringing this problem to heel?

Again, CrowdStrike cites that this issue was only an issue for about 87 minutes, but it took most of an eight-hour or longer working day for financial institutions to bring this issue under control. What accounts for the difficulty?

Control Plane Failures

Well, many engineering teams found themselves locked out of their computers because they were showing the Blue Screen of Death. Rather than being able to get onto Slack or Microsoft Teams or similar and discuss, "So what are we going to do with the CrowdStrike thing that is idling all the tellers?", they found themselves idled as well.

Even getting understanding with sister teams as to what is causing my computer to not work right now, and how do I recover my computer as quickly as possible, is relatively difficult for engineers who might not have practiced, say, doing phone trees to get around the complete downtime of computer systems.

We call this class of issues control plane failures. The control plane is the part of a computer system which does what governments might call the command and control of other parts of the computer system. To the extent that human engineers and other professionals in your organization are critical to the operation of your systems, things which cause your human engineers to be unable to communicate with each other are issues that impact your control plane. And control plane issues are frequently—here's a bit of engineering jargon for you—Sev-0, which is the highest possible level of severity.

Is there an easy fix for this? Well, no. You will never have an easy time if all of your engineers are unable to do work. You should attempt to avoid to the maximum extent possible that particular thing.

I think various financial institutions will have a different point of view as to what degree they want to avoid this. For example, one control that you could have instituted is to have a multiplicity of operating systems available in your engineering department. If some engineers were working on top of MacBooks and some were working on top of Microsoft Windows and some were working on top of Linux machines, the MacBook engineers and Linux engineers would have been unaffected by this issue because their systems are physically not capable of having the Blue Screen of Death. They have their own issues that are analogous, but not that one in particular.

They would have been able to cover for colleagues whose systems were bricked as a result of the CrowdStrike Falcon issue.

However, maintaining multiple operating systems within one engineering department has costs all of its own and imposes some potential for vulnerability and potential for needing duplication of effort, potential for needing to make the engineering organization larger, or just to serve the needs of the increasing size of the engineering organization.

In retrospect, it's obvious to say, "Well, if you just had MacBooks around the engineering department, this would have been much easier for you." But I would caution you against that sort of retrospective thinking.

The Domino Effect Across Financial Institutions

There are almost 4,500 banks in the United States, and you wouldn't expect 4,500 banks to have correlated failures, right? This is one of the bright parts of the American policy predilection to having as many financial institutions as we do. And in addition to 4,500 banks, we've got about 4,500 credit unions.

But there was a bit of a domino effect that happened here. This selected for institutions that had the best compliance posture and the best IT spend, which preferentially selects for the largest institutions in the country as a just fact of nature for the distribution of deposits in the United States. Deposits are heavily weighted towards the head of the distribution of financial institutions.

The Wall Street Journal reported Chase, Bank of America, Wells Fargo, and TD Bank as being heavily impacted by the CrowdStrike Falcon issue. Those account for a large share of all deposits. And in many markets, when those banks need to turn customers away because their tellers are unable to service them, that large portion of the deposit business of all banks in the city immediately falls over to the large regional banks that are in the city and the smaller financial institutions that are in the city. And those banks do not have enough standing capacity to quickly redirect all the deposit business and, most relevantly in this case, withdrawal business of the largest banks in the nation.

Impact on Smaller Banks

So even if their bank did not personally have CrowdStrike Falcon installed, they had a very similar situation where "our tellers are unable to do withdrawal transactions because we have physically depleted just the amount of physical U.S. specie that we have on the premises, because we have the only money that is available in the city of Chicago or any other city in the United States right now."

I personally observed with my own eyes failures in this fashion at several of the largest banks which have branches in Chicago, one large regional which is concentrated in the Midwest and which was carefully vague as to whether they had CrowdStrike installed, but said that they could not service my withdrawal because of issues caused by the failure of Microsoft.

Interestingly, many of the financial institutions threw Microsoft under the bus, where Microsoft is all but utterly blameless—it was CrowdStrike's fault. And then the smallest financial institutions that have the weakest compliance posture and probably the least well-resourced engineering teams were most likely to be unaffected by the technical bug, but least likely to be able to service a sudden, unexpected spike in withdrawal volumes coming from businesses in their neighborhood. And so they ran out of cash almost immediately.

I collected a series of anecdotes from small financial institutions all across Chicago: "We didn't even understand why it happened, but by 10 a.m., we were out of dollars."

What Went Right: Electronic Systems Remained Up

Now, fortuitously, many of the automated systems for banking continued operation throughout the duration of this. They did not have the endpoint monitoring software installed. They ran on Linux systems, etc. And so the thing that banks were told to do by higher-ups, the thing bank branch staff were told to do, was to redirect customers from the teller transaction which they wanted to do and which would fail to: "If you can use the website right now or use the ATM right now, that will work for you. And so if you were going to pay someone via cash, could you try paying them via Zelle instead, because Zelle is still up?"

For your context, Zelle is an American bank-to-bank transfer method which provides guarantees similar to bank-to-bank transfers in the United Kingdom, in that you can connect them for free and approximately instantaneously. There are some ways in which Zelle is different than UK bank-to-bank transfers, but they're outside the scope of this presentation.

Importantly, it is not guaranteed that in every future coordinated failure those secondary systems or even primary systems will remain up. We could very easily have had a coordinated failure which shut down both the tellers and the electronic payment systems at the same time. And if that had hypothetically happened, that would have been much, much worse for the U.S. economy than simply taking down the tellers, even though taking down all the tellers in the nation for a day was impactful for people who need money to pay their employees, to pay rent, etc. over the course of a weekend.

This Was a Near Miss

I want to emphasize this: This was a near miss, and it was a near miss including for you folks in the United Kingdom.

We largely restored normal banking operations at the end of extended hours on Friday. The most affected service—teller transactions—while decidedly critical, could be deferred for a short while. And there were competent and immediate efforts to divert transactions to electronic channels.

I often think that we under-emphasize the degree to which controls were effective when we're discussing failures. And so this was a control that was absolutely effective. We should send to the financial institutions that were most affected a brief note saying: "Congratulations for getting that right. That is what we expect of you. And you did an excellent job."

It was a good thing that they were able to have consistent messaging across all of their bank branches very, very quickly to tell them to divert customers from the teller window to the ATMs and to web applications to do their banking business.

Saved by Incomplete Deployment

We have not yet had CrowdStrike completely deployed within the banks. Wonderful—we were saved by our own slowness and incompetence. Many IT departments within banks would tell you that, yes, we are aware we need to have endpoint monitoring deployed on 100% of the fleet. Eventually we just haven't gotten to it by Q3 2024, but we're going to get there.

In a hypothetical world where we had gotten there, I want you to consider a hypothetical universe in which all U.S.-based counterparties of UK financial institutions went dark for a day or maybe even longer. That would be an enormously disruptive world state for you, even if the UK was virtually unaffected as first party by CrowdStrike (and I do not understand that to be the case).

The Timing Was Lucky

This happened on a randomly selected Friday, and that was astoundingly good luck for us. We might experience a technical crisis when the broader economic and business environment weather is not normal. It could be a technical crisis which is incidental to market stress or has a complex causal relationship with market stress.

One thing that happens often during market stress is that transaction volumes skyrocket when the market panics, and skyrocketing transaction volumes have a way of bringing bugs out of the woodwork in computer systems. I candidly think that is less likely to happen these days. We've gotten better as an engineering community at dynamically scaling systems. But historically, things break when transaction volumes skyrocket.

The combination of a market stress event and a large portion of the financial system being down would be an extraordinarily disruptive combination and would tend to reinforce in a very negative way the psychology of market stress, such as market panics.

What If This Had Happened During the 2023 Banking Crisis?

Looking back, for example, at the 2023 miniature banking crisis in the United States concentrated in our large regional banks, that was primarily a deposit run by very sophisticated depositors and deposit brokers against a relatively small number—only dozens—of large regional banks. It was not a deposit run broadly against the entire U.S. financial system, and the panic largely did not spread to retail. [Patrick notes: I commend this paper [PDF link], out of the Federal Reserve Bank of Chicago, as the best available public explanation. It is substantially better than many contemporaneous statements by regulators or policymakers in the U.S., which border on disinformation. It is unclear to me whether misinforming the public was a contemporaneous strategic goal.]

However, if there are lines going out the door at every financial institution, if tellers are saying "We can't help you, use the ATMs," and the banks and the ATMs have a big red error sign on them, then very possibly something which starts in one section of the financial industry can metastasize to the others. And even if those other portions were unaffected by the underlying technical problem or were unaffected by the balance sheet problems that beset, say, Silicon Valley Bank, they would have a classic deposit flight issue. And that could be extremely negative, as we saw during what regulators called the "percussive runs" of March 2023.

I would like you to be aware of the fact that the kinds of risk that you deal with on a day-to-day basis for financial stability are not divorced from technical risks. Those are tied at the hip, particularly in the worst scenarios.

Potential Policy Responses

Some potential policy responses for you:

1. Import the Culture of Blameless Postmortems

One, if you can import the culture of blameless postmortems from the industry into the financial industry, you should do that and/or grease the wheels to having that happen again. I think you can probably reach out to fintechs and just ask if you can see some of those postmortem documents. You will learn fascinating things from them.

I think you will learn a lot from the near misses where we did a full dress postmortem not because customers lost money, not because the safety of the firm was endangered at any point, but because if things had broken in a very similar fact pattern, customers might have lost money, or the firm might have taken a large capital loss, or something much worse could have happened.

I would strongly consider, from your point of view, socializing those near misses and telling other firms and the industry: "Look, we have paid as a collective industry on an insurance kind of basis. We have paid for the risk to learn what was in these near-miss postmortems. We would like you to learn from those postmortems before that risk shows up as an actualized risk at your own place of business."

2. The National Security Dimension

Now, you folks are senior representatives of the state of the United Kingdom. I assume that you have talked to some people from the security state at some point, and they are probably pretty intimidating.

Here's one thing that members of your security state are acutely aware of: If a 22-year-old engineer can take down a large portion of the financial system on accident, then a 22-year-old engineer can take down a large portion of the financial ecosystem on purpose.

Moreover, anyone capable of coercing a 22-year-old engineer or coercing their laptop can accomplish the same. We are not blessed in a world where everyone is worthy of the trust that we place in them, and we are not blessed to live in a world where no one is attempting to coerce good actors to do bad things.

In the engineering world, often it won't involve the actual coercion of the human. It will just involve subverting their laptop by getting them to install malware, click on the link in a malicious email, etc.

Identifying Critical Personnel

So what do you do with this? Well, a joke answer, but I am being very serious about this: If you were able to ascertain the identity of all of the 22-year-old engineers in the United Kingdom who are capable of turning off the UK economy, you should strongly consider posting SAS outside their door 24/7 until you remove their ability to turn off the UK economy.

That first requires you to actually identify those individuals. And I do think that identification is probably your first and best line of defense here.

But I'm being relatively serious with regards to that recommendation in that, you know, if their laptop is critical national infrastructure, you should treat their laptop as critical national infrastructure. Granted, the best possible way to treat it is to render it to not be critical national infrastructure as quickly as possible, versus protecting it with armed guards 24/7. But stating the obvious: you are a nation state. Protecting physical property and/or personnel with armed guards 24/7 is something that you are very capable of doing.

Red Team Exercises

Another thing that you might find useful to do is to encourage regulated entities to conduct red team exercises where members of internal teams role-play as bad actors and say, "Given my current level of permissions, what could I do to extract money from this organization?"

The bad actors in the world, such as, for example, the elite unit of hackers in North Korea which is, as a state-level activity, terrorizing much of the financial industry, are going after the weakest link. And you want to know that you are the weakest link before they do.

Fortuitously for those of us that are not very connected to crypto, they've been going after crypto for the last few years because it's the easiest place to ring the bell and get some money. But in prior years to that, they attacked things like, for example, the SWIFT system and got actual banks to wire out very large amounts of money to casinos in the Philippines in that exploit.

You might encourage your regulated entities to have the discussion about: "Okay, find who are the anomalously powerful people internally. Find what they could do if they had a mind to it, or if they were suborned by an actor that had a mind to it. And let's start closing those holes. But we can't close them until we know where they are."

Switching Gears: Stablecoins

Switching gears, you've asked me to discuss stablecoins a bit. I understand that the Bank of England, like many other central banks, has recently warmed to the idea of potentially there being a regulatory framework for stablecoins. I certainly think that if one thinks they are going to be a large portion of the future and used as money, one would prefer to have them under a regulatory framework versus not being under a regulatory framework. I will leave the specifics of that regulatory framework in your capable hands. Let me just tell you a little bit of my personal point of view on this.

My Crypto Skepticism

The crypto folks exist in the U.S. financial technology industry. Relative to most people in that financial technology industry, I am something of a certified crypto skeptic. I have been for 15 years, and as you can see from the shirt I'm wearing right now, I definitely did not buy a lot of Bitcoin in 2010.

The crypto industry has been engaged in a 15-year-long sales process to convince institutions that crypto is the shape of the financial future. That sales process has often involved lies. It's often involved half-truths as well, in terms of exaggerating the prominence and adoption of various crypto things.

Stablecoin adoption is the least exaggerated of any non-speculative use case that we've ever heard of from the crypto industry.

Actual Stablecoin Growth

Indeed, there was a report published recently—"Stablecoin Payments from the Ground Up" by Castle Island Ventures, Dragonfly, Artemis, and their other partners—where they had 20 collaborating firms across the stablecoin industry and another 11 firms which didn't specifically collaborate, but which they did an interesting act of interpolation to come out with their payment volumes.

They collected this as aggregate data to show the growth in stablecoin payments. That growth shows almost 3x growth in volumes from approximately $2 billion a month to approximately $6 billion a month in the last two years.

This is the first time in 15 years of being a diligent skeptic that I've seen something in the crypto industry and said, "Wow, that almost looks like there's a there there."

These numbers also, by the way—I have no official point of view or better view of a data source here, but they comport with my view as an industry observer. Yes, these numbers do not sound fantastical. They sound quite accurate, or quite likely to be accurate, in a way where many previous claims made about the crypto industry have sounded fantastical or outright lies.

Where the Growth Is Coming From

When you just aggregate the numbers, a large portion of the growth is coming from B2B payments, which went up from a negligible base of under $100 million a month in January of 2023 to over $3 billion a month by early 2025.

Interestingly, they say that the average stablecoin payment is approximately $200,000, which might inform you as to what these payments are being used for. That's likely to be small firms buying on the order of one shipping container at a time from China. Or, well, the cryptocurrency community will be slightly vaguer as to where the money is going, but without loss of generality, it's going to China.

Why $200,000 Is Awkward in Traditional Finance

Why is $200,000 an awkward amount of money to move in the traditional financial system? Well, to coin a phrase, $200,000 is enough money to worry about, but it's not enough money to be excited over if you're a bank.

If you are a small trader or a small business and you attempt to wire $200,000 to a new counterparty physically located in China, your bank is going to have all sorts of questions. And that wire might be held up in review for weeks. The factory is unlikely to release a shipment to you while the wire has not actually cleared. And so your shipment might be weeks delayed to the loading dock and weeks delayed off the ship and weeks delayed to your facility and eventually to your customers.

Stablecoins, on the other hand, transfer in seconds. And there is, in many deployments, essentially nobody who is doing a real-time human check on who you're sending the money to.

The Compliance Story

The stablecoin providers will say that they have a story with regards to know-your-customer and anti-money laundering regulations. And I think that they are quite accurate with regards to having the story at many of the firms. I will say that crypto has been a hive of scum and villainy for the last 15 years, and that the compliance postures of crypto firms vary wildly from each other. But there are good actors in stablecoins now in a way that there were rather fewer good actors as a percentage in prior years.

Putting $3 Billion in Context

But $3 billion a month—is that a lot of money? Well, certainly a lot of money if you are selling pizza for a living. But relative to other payment methods, it's a number.

If one looked at PayPay, a non-dominant payment method in Japan, it did approximately 800 billion Japanese yen in January of 2023, and that number has grown to 1.4 trillion Japanese yen as of recent months. So they are growing at a CAGR which is slightly below that of stablecoins and a bit more below that of stablecoin B2B payments specifically, but which is growing quite healthily.

There's a reason why you're asking me to talk about stablecoins and not PayPay. One reason is that, speaking bluntly, there are not a lot of equity investors and people who have hired the right people to continue talking up the PayPay investment to central banks.

Comparing to Pix

I will produce a graph here of three different payment methods so we can see the difference between ones which are doing quite well for themselves and ones which are doing extremely well for themselves.

Here's the monthly payment volumes denominated in U.S. dollars, billions, in mid-2025:

• The entire stablecoin industry, those 34 firms all wrapped up together: about $6.3 billion a month. Congratulations, that's nice business.
• PayPay: about $10 billion a month. That's a good business.
• And then Pix in Brazil does $90 billion a month only on person-to-business transactions—the classic payments use case.

Basically, if I were to rescale this graph to show all Pix use cases across Brazil, that is $450 billion. And now you can no longer see stablecoin's $6 billion monthly payment on the graph at all. It's physically invisible.

The Adoption Narrative

You have been given the representation for 15 years now that the cryptocurrency adoption graph has been faster than any adoption graph in the history of software. When that representation has been made previously, that representation has been other than accurate.

We've also had subsequent disproofs of that representation by, for example, looking at the adoption graph of AI in the enterprise over the last few years, or indeed the adoption of Pix in Brazil.

Pix is not a central bank digital currency. It's not a stablecoin. There is no blockchain involved anywhere. Pix is simply a central bank-mediated payments platform between various participating regulated financial institutions. And that's a thing that central banks are extremely comfortable with.

If you were just making decisions based on the spectacular payment volumes in emerging payment methods, you would probably look at Pix a lot and stablecoins—you know, it's got a nice business for them there, I hope that works out well.

The Elephant in the Room: Tether

There's also an elephant in the room with regards to stablecoins, and I apologize, but I feel like I have to bring out that elephant in the room.

Engineers have a culture, software people have a culture, Silicon Valley has a culture. And one big bit of Silicon Valley culture is a sense of omertà, in which we are all investors in each other's companies. Maybe each individual investor is not invested in every company, but it's something of uncouth to say another actor in the ecosystem is a bad actor—we don't like them.

For example, when you look at the Theranos fraud, a lot of Silicon Valley VCs after Theranos was exposed as being a fraud said, "We passed on them because obviously they were a fraud." But they didn't say word one to the media, to regulators, etc., prior to doing that, because there is just no percentage in pointing at a large firm and saying that firm's a fraud.

Now, who knows, that firm might recover from their issues and become non-fraudulent in the future. That story has been known to happen in Silicon Valley. But also, even if they never recover, it's just pure downside to you to antagonize other investors who might be co-investors on your other projects, to antagonize other future founders, to antagonize future employees at portfolio companies over the simple civic-mindedness of saying, "Yeah, they're a fraud." And so people mostly don't.

The Uncomfortable Truth

But I will be a bit of the antagonist for you here. I'd like to show you a graph of the top U.S. banks by total domestic deposits. And I've colored this graph blue for firms which are not a criminal enterprise, red for firms which are. Unsurprisingly, all of these firms are colored blue.

Here's a graph of the top UK banks by customer deposits. Again, a pie chart, again colored red or blue. Again, no top UK bank is a criminal enterprise. Not surprising.

Here's a graph of the top ten stablecoins by market capitalization. Oh, 65% of this graph is colored red because it's Tether, and Tether is bad, and everyone in crypto understands Tether to be bad.

Tether is a hive of scum and villainy. Always has been. And I would bet always will be until they eventually crack up and have a multi-tens-of-billions-dollar loss event.

The Bull Case for Tether

But I will give you the bull case for Tether. The bull case for Tether is that tens of billions of dollars of losses have already happened, and they happened at FTX and not Tether.

See, Alameda Research, acting under the direction of Sam Bankman-Fried and his co-conspirators, was Tether's banker for a while. They professed to buy more than $40 billion of Tether in return for transfers of cash made in the United States financial ecosystem. And that cash money—Sam Bankman-Fried stated at substantial length—was absolutely real and represented over-the-counter trades that Alameda was conducting on behalf of other customers, was absolutely real, was transferred to Tether's banks, really existed. "The reserves are fine."

And then FTX customers lost—I believe the number usually associated with it is $8 billion, but the number is a bit fuzzy due to how exactly bankruptcy math works. But a large amount of money was misappropriated. And so Sam Bankman-Fried is now a long-term guest of the United States federal government. The customers are out the money. FTX no longer exists as an enterprise except as a bankruptcy estate.

But the cash money from those customers ended up in Tether's reserves, to the tune of tens of billions of dollars.

"The Hole Got Filled"

The colorful part of the cryptocurrency community will say—and we will not in public agree that there was a hole, but in private, yeah, of course there was a hole, I mean, come on—"If there was a hole, that hole got made up over the years by some combination of coincidentally receiving tens of billions of dollars from FTX under circumstances that we will not elaborate about, and moreover by the interest rate environment moving massively in Tether's favor."

Tether has a deposit beta which is unmatchable relative to most financial institutions. They don't have to pay any of their customers interest on over $100 billion of Tethers currently outstanding—about 115 billion, I think. Those Tethers are largely backed by reserves that are invested in Treasuries. Treasuries are paying what they pay. And so Tether is profiting by billions upon billions of dollars a year of interest rate delivered by the American government in an utterly aboveboard fashion.

So if there was a hole, if there were some assets which were not quite worth what Tether claimed in their previous attestations, well, they had the time to work that out.

Character Witnesses

The most clueful-but-still-crypto-positive people will say: "Look, we understand that Tether did some shenanigans, lied about it with respect to things like saying that they were in all cash assets while actually having Chinese commercial paper." They claimed after it was discovered by Bloomberg that, "Chinese commercial paper? What? We never really said we didn't have that. But we're going to get it off our books very quickly. And by the way, it has a weighted average of less than 180 days." And it actually took them much more than 180 days to move that off their books. Why? Well, probably to work out issues with that paper and the necessity of rolling it out a few times.

But the issue with assets that are underperforming is that you can solve that problem with money as long as the asset has some value associated with it. And so they've worked out their problems. The composition of their reserves is better now. It's mostly Treasuries. And Tether has played an excellent PR game in attempting to convince the rest of the world that they're not crooks anymore.

For example, they sent their character witness—one Sam Bankman-Fried—to Bloomberg on August 8th, 2021. Quote: "I think the answer regarding Tether is that it's messy, but the funds are real. We see legitimate inflows into Tether from many sources, large ones, that result in market makers selling, creating, and sending billions of dollars to Tether's bank accounts. They do this to mint tokens and maintain relationship with Tether and its banks. Everything checks out, albeit in a messy way."

Sam Bankman-Fried is no longer available as Tether's character witness. So they have a new one whose name is Howard Lutnick. On January 16th, 2024, he said the following to Bloomberg TV: "They have the money they say they have. I've seen a whole lot and the firm has seen a whole lot, and they have the money. And so there's always been a lot of talk: Do they have it or not? And I'm here with you guys and I'm telling you, we've seen it and they have it."

Howard Lutnick was referring to the firm Cantor Fitzgerald that he was the CEO of. He was not under oath talking to Bloomberg TV. He was under oath when talking to the U.S. Senate in an attempt to get a career upgrade to Secretary of Commerce in the second Trump administration.

Under oath, he said the following: "Cantor Fitzgerald is not conducting continuous diligence on Tether's financial statements, but I believe my statements were accurate when made."

Which is a substantial walk-back of that position.

Tether's History of Lies

I don't have nearly enough time to recount Tether's history of lies, evasions, and other various calumnies.

Today, Tether is currently under a class action lawsuit, and they had the remarkable statement made in this class action lawsuit as a defense that the people suing them have been able to compel the production of documents which included information about their balance sheet at various points in the past.

They said: "Well, these balance sheets prove that you were debasing Tether at various points, which is against the interest of the class members. And so you should pay us for the damages that you inflicted upon the class members."

Where did those documents come from? My guess is that those documents came from, for example, financial institutions that Tether was sending balance sheets to and which the plaintiffs were able to compel from institutions that have compliance departments and lawyers.

"We Had No Books"

Tether said in response to these documents—this is not a quote, but a paraphrase: "Documents have been produced in this litigation that purport to be our balance sheets at various points in the past. However, those documents are inaccurate because they were prepared by less senior members of our finance department than the CFO. And the CFO was the only person who had a full point of view as to what our liabilities and assets were at any given time."

"Those liabilities and assets were not recorded in our books. They were recorded in PowerBoard, a software which showed us a dashboard. And our CFO was very on top of the dashboard and says that our assets exceeded our liabilities at all times during the relevant period. However, PowerBoard does not allow us to retrospectively see what assets and liabilities were at any point in the past. And so it would be an incredible imposition—in fact, a technical impossibility—for Tether to produce a balance sheet at any arbitrary point in the past."

Indeed, Tether has filed multiple legal pleadings with their judge attempting to get them to quash the request to produce balance sheets on the basis that we can't produce balance sheets because we had no books.

Tether was not a tiny financial institution at this time. They were, you know, nominally having tens of billions of dollars of assets under management, and they had no books to do it with.

The Defense: "We've Bought Respectability"

Defenders of Tether will say: "Well, yeah, sure, shenanigans happened—quickly growing financial startup, what are you going to do? But Tether has bought respectability now. They've sent the right people to Washington to say the right things. And look, the bankers are working for a government which is extremely pro-crypto because of the amount of money the crypto industry has put sloshing around Washington. There's not going to be any regulatory reckoning happening in the next four years."

Maybe they're right about that and maybe they're not. It certainly does seem on the basis of that evidence that being Tether's banker is an excellent way to get one's living accommodations paid for by the United States taxpayer one way or another.

But I continue to believe that Tether is eventually going to get what's coming to it.

Evidence of Recapitalization

Back in 2022, The Wall Street Journal reported "Rising Tether Loans At-Risk Stablecoin in Crypto World," and they independently reproduced math that I had done on two points in 2022, which essentially proves that Tether had required recapitalization during those years due to the decrease of their non-Treasury assets during crypto volatility (which, of course, Tether never allocuted to).

Tether has not gotten rid of the non-crypto assets in their reserves. They've continued to increase the amount. And they've eliminated the disclosure that they used to make that their quote-unquote "secured loans" were not to related parties.

So the inference, given that Tether is always doing the worst thing that is possible in coherence with their regulatory statements, is that they are making loans to related parties. And of course, they're probably continuing their history of lies, frauds, and evasions on top of that.

So That's 65% of the Market

So that's 65% of the stablecoin market. Perhaps one is much more enthusiastic about the remaining 35%.

I will say that there are legitimate businesses that see stablecoins as useful for their use cases, either at the scale of sending $200,000 at a time to buy a shipping container of goods from China, or, for example—it's been publicly reported—Starlink [Patrick notes: I flubbed and said SpaceX in the live audio] attempting to move money around a very international company without constantly having to worry about the difference between 50th percentile wire arrival times and 95th percentile arrival times, which is unfortunately a thing that operations teams at large international financial institutions have to have a very good intuition for, and which is something that ideally the financial ecosystem can make businesses in the real economy not need to worry about so much.

I think there is possibly a future in which stablecoins are a larger portion of international payments than they are currently.

Stablecoins for Domestic Payments?

The question is sometimes asked: What do stablecoins offer for domestic payments? And I think there are far less good answers there, candidly. And one reason that you can see that there are less good answers is that the actual use of stablecoins for domestic payments does not seem to be increasing at anything near the rate of international and B2B payments.

Why not? Because in much of the developed world, we have pretty good domestic payment rails. You bought your coffee in the UK this morning. You had no problems in doing so.

I acknowledge that there are many people who are living in places that have far less stable currencies and far less good domestic payment rails that are tied to those currencies. And there has been a narrative of stablecoins becoming the digital dollarization of currencies in some countries where the central banks have not made the official policy decision to dollarize. Perhaps that will happen. Perhaps that will not. But we have not seen evidence of that actually happening yet.

The Bitcoin Price Correlation

Another thing—and I will credit the CEO of Ethena, which has a stablecoin-adjacent financial product, for saying this—is a thing that you can tease out from stablecoin peer-to-peer payments volume: it's heavily correlated with the price of Bitcoin and froth in the crypto market generally.

That is not something we expect to see in payments. The amount of coffee that you consume is not correlated with the price of Bitcoin. We generally expect the real economy to only have a fairly tenuous relationship with, say, the financial markets and almost no relationship with cryptocurrency, except to the extent that cryptocurrency is shadowing the financial markets.

So that is another data point that would suggest—and again, highlighted by someone who would bet the other way—that stablecoins are not materially used in peer-to-peer payments at the moment to the degree that has been asserted.

Who Loses If Tether Implodes?

Anyhow, if tens of billions of dollars are lost in the result of a Tether implosion, who will lose the money?

Well, again, some people who are relatively pro-crypto industry would say, frankly to you: "Look, tens of billions of dollars have already been lost. It's no worries. They were lost by FTX's creditors, and that case has been tied up with a bow now."

But in the financial ecosystem, we would generally prefer that equity takes a hit prior to depositors taking the hit. And the good news, quote-unquote, is that Tether is currently seeking equity funding at a $500 billion valuation, which would theoretically put 15 or so billion dollars of dry powder in the case where they needed to work out any infelicities with regards to their reserves.

So the equity would take the hit first. And nothing that more clueful crypto observers would say is that the equity was always backstopped either by the combined equity of the Bitfinex enterprise—that's how they were able to do ad hoc equity injections back in 2022 when they needed it. But of course, that is not the thing that people are claiming publicly.

Tether: The Story So Far

A thing I wrote back in 2019: "Tether: The Story So Far." A friend of mine who works in finance asked me to explain what Tether was. The short version of Tether is the internal accounting system for the largest fraud since Madoff.

This is a thing that I still believe to be true, except it is now much larger than Madoff ever was.

He went 17 years before he got what was coming to him.

AI and the Future of Trading

You've also asked me to discuss AI and the future of trading a little bit. And I put on my thinking cap about this. In the course of doing that thinking, I went to some of the leading AI labs and talked to some people at various levels of seniority and said: "I have the opportunity to address some members of the British government with regards to the risk of, say, AI making correlated decisions at hedge funds and causing systemic instability. What would you like me to tell them?"

The Message from AI Labs

The point of view that I got told up and down various AI labs was: "That seems to be a really rubbish use of your time in addressing senior members of the British government. Can you please tell them this instead?"

"We're not lying when we say that it is our impression that we are building the most important technology that has been built by humans to date. We are not lying with respect to the scaling curves. We think these scaling curves will continue for a relatively short number of years from now. If they continue, we will achieve Artificial General Intelligence (AGI), and then perhaps Artificial Superintelligence (ASI). Either or both of those being achieved will have such enormously consequential impacts to our societies and our economies and similar as to utterly dwarf the importance of either any particular hedge funds or even hedge funds as a class of human endeavor."

The Scaling Laws

The fundamental graph that a lot of people in AI labs look at was introduced by Kaplan et al. in 2020. I'm going to give you a bit of a non-technical gloss on this graph. It's a log graph, as the best graphs are, and it measures the test loss against increasing by a power of ten at a time: the compute, the dataset size, and the number of parameters used for training an AI model.

These graphs are strikingly linear when expressed on a log scale. When we increase the amount of compute that we do, we have a linear decrease in our loss function. Decrease in the loss function intuitively means that the system is getting smarter.

Now, the thing that the AI labs will tell you is that, okay, this was the public version of the graph published in 2020, but we've updated this graph every two weeks or every six weeks since then, and these graphs stay strikingly similar. There has not been an inflection point yet. We just keep figuring out every time we run the data: we just need more money, we just need more dataset, we just need to add more parameters, and it's going to get smarter in the next iteration of the software.

You have seen the last two, three years of that continuing to be true. And you can viscerally tell the difference between, for example, GPT-2 and GPT-3 and GPT-3.5 and GPT-4 and GPT-4o and GPT-5. You can viscerally tell the difference of what you would get if you asked early versions of Claude to do coding for you versus what you would get in the Claude Code engineered system built on top of the most powerful models that it has available to it right now, such as, for example, Claude Sonnet 4.5.

Just imagine what is going to happen over the course of the next two years.

My Position as a Skeptic

Now, sure, I would say—and I'm not an emissary from the AI community to you—I am what many people in the community consider to be a militant AI skeptic with regards to the likely trajectory of AI.

And when I say I'm skeptical, I think that the impact of AI will be greater than the impact of any technology up to and including the Internet, where I consider the Internet to be the most important artifact that humanity has ever created and our collective great work as a species.

That's what makes me a skeptic. Because I'm not saying that it will fundamentally transform human life forever within the next three years.

There are some very smart people who have obsessed on that question specifically for the course of the last five, 15, 20 years and who believe that we are living in the most impactful three years of the human race.

A Warning About Media Coverage

I would urge you: when you read articles in mainstream publications which have well-respected editors behind them, which seem to know what they're talking about, and which say things like "AI might be something of a bubble, we might lose trillions of dollars," etc.—sure, that is one point of view. But I would accord a considerable amount of your institutional resources towards listening seriously to the AI labs and even to the detractors of the AI labs about what future trajectories for this technology could look like.

Because if those future trajectories are anything like accurate, this will be one of the most important things that happens to the Bank's broader mission, much more so than the limited impact with respect to trading.

Another Striking Graph: AI Capabilities Over Time

Another graph I could show you, continuing the theme of graphs which look strikingly linear when expressed in logs:

Metr, a nonprofit organization which is organized around assessing the progress of artificial intelligence technologies and thereby helping to limit the risk downstream of them, has created a graph showing the time horizon of software engineering tasks different labs can complete 80% of the time.

This graph stretches from 2020 to late 2025, where we are right now. And when you express it on a log scale, this is strikingly up into the right linear. And they project out that there will be fully automated AI researchers within a shockingly short time frame by the standards of, for example, banking regulators.

We expect those fully automated AI researchers to greatly increase the progress of AI research, rather than having it be only the breakneck speed that we have observed over the course of the last two or three or four years.

Recommended Reading

If you want a good entry point into this for a general audience, I would point you to "Situational Awareness," a paper written by a young man who now runs a hedge fund. That hedge fund has done pretty well for itself. But it breaks down the back-of-the-envelope math for how we could bring trillions of dollars to bear on the capital requirements for training AI clusters.

I think the community would like to signal to you loud and clear: We think we can actually find that trillion dollars or whatever the dollar amount is.

If you have kept your head in the sand for the last 30 years, the tech industry has done pretty well for itself. And senior decision-makers within the tech industry are very bought in on the vision here across many of the tech majors and essentially all of the labs and much of the broader tech ecosystem.

The Commercial Reality

Additionally, even if we weren't bought in on the larger vision, the actualized realized results of AI as commercial products is staggering.

I'll put a link in the notes to this presentation about the Stripe Annual Letter, which discusses how the various AI companies have growth rates which are far in excess of the best-in-class growth rates of the software-as-a-service companies during the software-as-a-service boom.

There were publicly reported data points on how Anthropic was adding billions of dollars of revenue over a period of months.

When one looks at Anysphere, the company behind the Cursor coding project, TechCrunch reported that it had surpassed $500 million ARR—a 60% increase from the $300 million of annual recurring revenue that they had reported only six weeks prior.

These are staggering numbers. And if you believe that those staggering numbers will scale that impact from not just the developer tools market but to essentially the entirety of the economy, that justifies almost any amount of capital investment.

“What is the price tag? We are willing to pay it.” It is a thing that people would like you to hear. So consider yourself as having heard it.

My Personal Experience

I'm a far less gleeful person than many of the people who spend their days working on these things. I have an increasingly out-of-date undergrad concentration in computer science, and I spent August and September falling in love with engineering all over again as a result of using Claude Code to do computer-assisted engineering on a hobby project.

Depending on how you work the math, that was 8 to 10 times faster and more effective at doing engineering work as I have ever been. And I am no slouch in the engineering department and have solo-coded two companies to the point of sale. And this was an 8-to-10x speed-up for me.

When you consider deploying this over large portions of the entire engineering ecosystem, I think you start to think of very staggering impacts in that ecosystem. And then think that us engineers are probably not the only people in the economy who turn ideas into cognizable business wealth.

Thomas Ptacek's Perspective

I would also point you to a blog post written by Thomas Ptacek, who is the most talented technologist I ever met, where he describes this as one of the two most important things that has happened in the course of his long career, and that even if AI progress were to stop tomorrow, metabolizing the impact that has already been delivered through increased AI capabilities via these coding tools would fundamentally transform the practice of software engineering.

And again, I think it is extremely unlikely that software engineering is the only field that is fundamentally transformed by things that have already been built. Imagine what the next two or three years will bring.

AI Risks in the Trading Space

You did ask me, though, to address the AI risks in the trading space. So to the extent that we were talking about single points of failure earlier, in my conversations with some staff members, it was intimated to me that you think that every hedge fund is going to train their own model.

That is not the outcome I would bet on.

There is going to be concentration risk among the three large lab providers. And in any given six-month window, one of them might have, quote-unquote, "the best" model available. To the extent that LLMs are a material input here—now, LLMs are not the only form of AI or machine learning, but they seem to be scandalously effective at some classic machine learning problems.

The Image Recognition Example

One thing which we spent billions upon billions of dollars getting good at with non-LLM machine learning models was image classification. LLMs are not natively image-aware. But after we taught them to talk, we said: "Well, I think you could possibly talk. It's like the JPEG file format. What do you see in this JPEG file format?"

Expecting them to say, "Oh, it's a blob of numbers, and maybe those numbers have some correlations between each other."

And they said: "I see a parrot standing on someone's shoulder. The parrot is green. It has a self-satisfied expression on its face."

We essentially solved image recognition by accident over the course of less than two years.

LLMs in Trading Systems

In the panoply of systems that I would expect UK trading firms to deploy over the next couple of years, it's very possible that LLMs are a large and increasing portion of their technology footprint. And one can imagine that in a given window, you might be essentially sole-sourced on a particular model provider.

And if that particular model has a cold one day or suddenly discovers that—you know, "I was retrained recently, and gilts, I just don't like them"—that might be a surprising source of risk to the UK trading space.

But again, I don't think that's the most important thing to learn about AI this year.

Dependency on AI Coding Tools

Another factor: the time to detection and resolution of issues is increasingly dependent on whether Claude Code, OpenAI Codex, and similar tools are up or not.

The most experienced cool kids in the world have about 12 months of time that they've played with these tools. And increasingly, even in only six weeks of playing with it myself, it has become physically difficult for me to navigate a codebase without having Claude Code help me to digest the code base that I wrote.

When financial systems are on fire, every minute matters. Knight Capital collapsed over a 20-minute critical window. CrowdStrike was citing about a 90-minute window for their time to resolution.

If during a crisis it is difficult to find engineers who can remediate the crisis because Claude Code is in a slightly degraded state on any given day, that introduces a source of risk into the financial system which didn't exist two years ago, because we didn't have computers that were so good at assisting engineers two years ago that engineers would forget how to do core parts of the job.

Not Incompetence—Skill Atrophy

I don't want to call engineers incompetent or malicious for forgetting how to do this. This is simply like a skill that one develops over the course of a long career. It's something like riding a bicycle—you never forget it per se, but it really matters if you are doing the Tour de France that you have continued riding a bicycle every day for the last 365 days versus "Bicycles? Yeah, I remember having seen one about five years ago. But why would I ever use a bicycle when I have a car available?"

You should be aware of the fact that we are now acutely dependent on cars and acutely dependent on LLMs, even for things where LLMs are not officially making the decision that matters. And the recursive dependency graph puts larger chunks of the economy on narrower shoulders.

Again, if there are three labs where one or three of those labs going down will idle engineers at a huge number of firms, then we have far less numerous points of failure than we did previously.

Closing

Thank you very much for your time and attention today.

I do not think I'm the last engineer you should talk to. I think you should talk to many engineers. I'm happy to make some introductions. And I'm very sure that if you talk to your regulated institutions, they can identify engineering leaders internally, perhaps staff-level engineers internally, and other people who you should urgently make the acquaintance of.

If I can ever be useful for you, please feel free to send me an email. And I write Bits About Money—it's freely available and very relevant to your interests.

Thanks very much.

And so that is what I told the Bank of England. Thanks very much, everyone, for listening to Complex Systems. And I'll see you next week.