Two banks can't agree on fourth-grade math | Alex Elliott
Patrick McKenzie (@patio11) is joined by Alex Elliott (@alexpotato), who has spent his career in SRE and operations roles at financial firms ranging from ten-person fintechs to global investment banks, to discuss what the back office actually does all day. They cover the life cycle of a trade: execution, clearing, settlement, and the constant reconciliation between counterparties who don't quite agree on what happened. Alex recounts the Friday afternoon when twenty million shares went out to the street instead of five, and Patrick explains what happens between 4:15 and 4:30 when one bank owes another $36 million and the person who owns that process is on vacation.
Presenting Sponsors: Mercury, MongoDB & Granola
Complex Systems is presented by Mercury—radically better banking for founders. Mercury's new feature Command brings an LLM directly into your banking interface, so checking balances, finding invoices, or sending a wire is as easy as asking. Apply online in minutes at https://mercury.com/.
If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS
What's the point of building faster with AI if your database can't keep up? MongoDB's native data model mirrors the language LLMs already speak. Ship at the speed of AI while staying ACID compliant at Fortune 500 scale. Start building at https://mongodb.com/ai.
Timestamps:
(00:00) Intro
(01:48) The life cycle of a trade
(04:37) Why you don't know who you traded with
(06:01) Hiding a large block trade
(10:04) Information walls inside financial firms
(12:49) What the middle office does: managing positions
(15:41) Calendars, time zones, and other sources of gibbering madness
(19:37) War story: twenty million shares out the door
(22:45) Sponsors: Mercury | Granola
(25:50) Ops as an iterated game
(28:53) Busting trades and who has the authority
(33:38) Sponsor: MongoDB
(34:29) Inside the Knight Capital collapse
(35:49) Agreements, obligations, and who you have to call
(36:57) Reconciliation, FTP files, and the humans behind them
(39:16) Deadlines, cutoffs, and the $36 million phone call
(44:26) How two banks fail to agree on fourth-grade math
(48:54) When the golden record goes down
(51:55) Finding the one person who can sell the position
(54:09) How much business do we do with Fidelity?
(56:38) Prime brokerage and extending credit
(01:01:03) Smart flow, dumb flow, and payment for order flow
(01:02:45) Wrap
Transcript
This transcript will be annotated in Patrick’s usual style by the end of this week — be sure to check back.
Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does.
Patrick: Hi-de-ho, everybody. My name is Patrick McKenzie, better known as patio11 on the internet, and I'm here with Alex Elliott, a Twitter buddy of mine who has worked in a variety of operations roles as a financial technologist across a number of financial firms.
Alex: Hi, Patrick. Nice to meet you in person, I guess, or over the podcast. Yeah, I'm Alex Elliott, alexpotato on Twitter. And yeah, as you mentioned, I've worked primarily in SRE/DevOps at a bunch of different financial firms ranging from small fintech startups with like ten to twenty people, all the way up to giant investment banks with like hundreds of thousands of people as well.
Patrick: So I think if people have read about finance for a while, they might have come across this concept of the front office, which is the high status, highly visible customer-facing function, and then the back office, which is often caricatured as, "Oh, those are the geeks moving numbers around," and perhaps paper emanates at some point. And we need them, but we don't love them.
But I love back office minutia to an almost unhealthy degree, and I think it's sort of one of the unsung heroes of the ecosystem that literally keeps the world spinning. And so wanted to talk to you about what the back office actually looks like at some varieties of financial firms. Like, what do you do all day? And then maybe talk through some war stories that show, you know, if this stopped spinning, it would be very bad in a hurry.
So pick a genre of firm from your vast selection and talk about what the back office does all day.
The life cycle of a trade
Alex: Sure, yeah. At broad strokes they're very similar, so I'll do a little bit of the life cycle of a trade in sort of a very fast overview.
So somebody will generally say, "Hey, I am selling, let's say, a million shares of Microsoft." We'll call that party A. Party B will say, "Hey, I wanna buy those shares of Microsoft." And generally party A has a price, party B hits that quote, and then basically an execution happens, and now both parties have this transaction between the two of them that they need to agree upon that it occurred and then make sure that the money and the securities or assets all moved on their own.
Patrick: And importantly for this, many people have this assumption that, oh, that's what a stock exchange is for, right? The stock exchange must take care of all this for you. And while that rounds to true in crypto — that the exchange is the same one that is doing the custody and the clearing and settlement of things — that largely isn't true in the traditional markets. There's a separation of functions.
Alex: That is correct. And we could go down a long rabbit hole in the different types of flow as well. So for example, if you are prop trading, you might be going to an exchange and someone else is going to an exchange. It's the two parties. In traditional investment banking trading, there could be layers, e.g. there's a trader, then there is a sales trader, then there is a customer, which is usually like a pension fund. CalPERS is one of the kind of giant ones, which is a California pension and retirement fund, I believe. But it's true of many different large customers. It could be other hedge funds, sometimes other banks.
So yeah, at the end of the day party A sold something and party B bought something, and you have to determine, okay, do we agree on this? And one of the biggest parts of back office is what is called clearing. And effectively, the exchanges — let's say you execute on the exchange, 'cause there are other places you could execute — the exchange will report back to both parties, "Hey, we have this transaction. Now you guys need to go off and communicate with each other that you agreed on this transaction."
And that can happen a bunch of different ways. There are direct connections between different firms. So firm A and firm B might know about each other and have essentially a bilateral connection where they send each other information about the trades, and they want to make sure that everything cleared. Clearing here would mean I know that I bought a million, you know that you sold a million, and the price is correct. That can happen relatively quickly. And then there's also a settlement portion, which is where things actually exchange, in the sense that I give you the money, you give me the securities, and everybody's happy.
Why you don't know who you traded with
Patrick: And one of the joys of finance is there's a whole lot of complexity smuggled into the words, "I give you the money." But to just zoom in on one bit there for non-specialists: at the point where you've bought a million dollars of a particular — you know, Microsoft in your example, or a million shares of Microsoft, which is rather substantially more than a million dollars — and you did it on a particular stock exchange as a venue, you likely don't know who you bought it from, right?
Alex: That is correct. So in the example I was giving before, if I'm an investment bank, when I send out my quote, that will generally have what's called my MPID, which is a high-level identifier for the firm. E.g. Goldman famously had, I think it was GSCO, but then there was GSCO through I think nine, ten, eleven. Sometimes they don't make a lot of sense. E.g. Citigroup's was SBSH, 'cause that was Salomon Brothers Shearson, which became part of the Citigroup conglomerate.
But if you are going through another firm, you don't always necessarily know who the other party is. Some exchanges and some asset classes you also don't know. It'll just be like you execute against somebody, and then later on you might find out, oh, that was with this counterparty, et cetera. Or you are going through, say, a broker or a prime broker, and then it's even more obfuscated because there's three or four hops between you and the actual exchange when things actually get executed.
Hiding a large block trade
Patrick: And this is actually important for some people — that this not quite anonymity as you're trading, but something close to anonymity as you propose trades that you are willing to do. Because some traders have a private state that they believe about the world that they wanna transact with the rest of the market, and they know transacting with the rest of the market is, one, visible to other people, at least in anonymized fashion via the tape.
Like, someone can see that a buy or sell happened, but they don't wanna tip their hand as to, like, okay, if I'm selling a million shares of Microsoft, it's quite likely that that is not the only Microsoft I will be selling that day. And so that might suggest to the rest of the market, "Hey, the price of Microsoft is probably going down unless someone really, really wants Microsoft right now."
And this is one of the sort of core problems in trading: doing large block trades in ways such that you get the execution you want at a price that is acceptable without informing the rest of the market that you are doing a large block trade, because that will have what is typically called price impact. The rest of the market will understand the totality of the trade before you put it on and start quoting less favorable numbers to you.
Alex: Yes. And you bring up an excellent point, and I can talk about some different ways that people do it. So let's say you are a firm, and you have some alpha or signal, or you have some state on where you think things are gonna go.
One thing you could do — let's say you wanna go through a broker. You could tell the broker, "Hey, here's this giant order. Work it for me." Now, the broker knows that you think this is the case, and so there are rules to prevent them taking advantage of that, AKA front-running. But if you just put out this giant block, as you mentioned, other people will see it.
So some of the techniques you can do is you can chop up the order over the day, and there's different ways you can do that in terms of what's called smart routing, where you give it to AKA an algorithm, and it'll make lots of little pieces, and it'll send it different places. You can send it to different venues yourself, so you might have five or six different brokers, and you can say, "I'm gonna give ten percent to this broker, twenty percent to this broker."
You might even go to the investment bank and you could say something like, "Hey, I want you to execute this million for me, so it won't show up as me executing it. It'll show up as you." And this gets more into what's called Reg NMS and how orders are routed and fills are sent to clients, et cetera. But the bank can then chop it up themselves. They can give you one big fill, and then they can AKA work the order over the day. So yeah, there's a lot of different kinds of flows through the system to get you the position that you want. Some firms do it over multiple days, et cetera.
Patrick: And for each of these, there are people with generally opposed incentives, where if you are aware that there are technologies to chop an order up over the course of a working day, then that suggests that at eleven oh two in the morning, you might be able to say, "Wait a minute."
You know, if the naive way to do this — and believe me, people have tried the naive way to do this — if the naive way to do this is just sending in an order every five minutes, and we've seen dozens of orders at this point that are all for the same quantity and the same symbol at the same price, then I naturally assume that I will see more dozens of orders through the rest of the day and can start positioning myself against those orders before they have actually been made.
Alex: Yes, that's correct. Without going into too many of the alpha details, firms look at all kinds of data on just the tape. E.g. they look at just fills coming in. They look at the position of quotes in the order book as it changes over time. They look at alternative data from different data sets. I've never seen this data set used where I've been, but there is this discussion around, like, oh, satellite photos of Walmart parking lots is a famous one that people say, "Oh, that's an indicator of volume," et cetera, right? So yeah, they look at all kinds of different stuff.
Information walls inside financial firms
Patrick: As we've discussed on previous issues of this podcast, one of the reasons why traders are so tight with regards to information flow — down to not telling their friends what they've been reading recently — is to avoid cluing off other people on, quote-unquote, "the street" on what their firms might be very intellectually interested in, and therefore suggesting, like, maybe if you dug a little bit deeper into the tape on a particular security or particular set of securities, you might find something that is quite interesting.
Alex: So one point I wanna kinda highlight also is different firms have kind of different opinions on how they structure the information sharing within a firm. So in certain cases, there are literal walls between different parts of the business because either legal or regulatory or what have you, this arm can't know what this arm is doing. And there's a lot of work that goes into making sure that doesn't happen.
Then there are other firms where it's not a legal or regulatory issue, but they still want to keep things separate because they have different teams, and they want to avoid sort of leakage of data for different reasons. And one of my favorite stories — this is at a firm that no longer exists, so I can tell the story — it was so quiet that they wanted to pump in pink noise because they didn't want anybody to inadvertently hear somebody from something else that might have been disclosed that nobody wanted to know about.
Whereas at other places, everybody can see the same source code, and everybody's on the same non-compete strategy, et cetera. So yeah, it's interesting how they kind of play out in different ways.
Patrick: Silicon Valley companies have historically had — with exceptions, Apple being a huge one — they've historically had a huge amount of internal transparency on things. And so if the company is having a great year, the fact that the company is having a great year is quite public internally, and the reason it is having a great year is quite public internally.
But given that finance, like tech, has a constant turnover of people moving between jobs for the usual set of reasons, you don't necessarily want everybody in the company to know that the eight-person group that is killing it right now are, I don't know, mortgage derivative traders. Because even if those eight don't leave, that could cause the thing that is causing them to be so lucrative to leave the building to a competitor, or suggest to another competitor, "Wait, we don't have the group analogous to that. Maybe we should, if it is printing like billions of dollars a year of excess profit."
And indeed, there was some litigation about this recently, with the apparent size of the information that you need to disclose to get a company to be very annoyed about the fact of the disclosure and sue you was, "Hey, have you heard of Indian options? They are really lucrative." And I'll point to some Matt Levine columns for people who want more on that detail.
What the middle office does: managing positions
Patrick: Since we're here about the back office and people are thinking of, you know, brokerages that are doing millions upon millions of trades every day, I think they probably assume that it's like one person like yourself and then a million computers. And in fact, most firms have more than one back office professional working for them. Why?
Alex: Sure. So I'll talk a little bit about the middle office as well here. So one of the things that you want to do correctly from a middle/back office perspective is manage your positions. And that might sound pretty simple. Most people have some kind of retail trading account, and they think, "Oh, I look at my account every day. I have ten or twelve stocks in there, and I buy them maybe every week or every day if you're doing day trading."
Firms at large scales, e.g. banks or larger hedge funds, are trading globally in potentially hundreds of different markets or venues. The products are different in many different ways. Even things that you might think are the same can be different. So an example might be, let's take, like, UBS was a Swiss bank. They have a US entity, they have a Swiss entity, they might have an Asian entity. You might also be dealing with different entities in the same country, e.g. there is an investment banking arm, there is a retail trading arm, you might be dealing with both.
So I'm giving a taste of like three or four of the hundreds of dimensions that can exist, and that's why you hear people talk sometimes about security master or reference data, et cetera.
Patrick: And if I can give one more on it. You know, you might think it's fairly straightforward to know how much Microsoft I own at any given time, which is not quite true. But when you're an investment bank, for example, working with many sophisticated clients, there are very many ways to own Microsoft in the world, and there are very many ways to owe Microsoft in the world.
And so for example, you might have a wealth management division that — because relatively wealthy retail investors like Microsoft as a brand name, you might have sold them some income products which have reference to the price of Microsoft securities. And so you are synthetically exposed to Microsoft, which matters when you are synthetically exposed to the tune of like hundreds of millions of dollars.
And so you need to know, okay, if the Microsoft share price goes up by 10% because, I don't know, Bill Gates comes back to work or something, what does that do to essentially my exposure right now? Like, I'm probably long in some ways and short in other ways, and minimally, am I net long or net short is a great idea of something to know. But to what degree, and to what degree does that change even if I do nothing to change it? The joys of option math here are always fun.
Calendars, time zones, and other sources of gibbering madness
Alex: That's an excellent point. And I actually worked — one system I worked on was a swaps database where essentially you go to another party and you say, "Hey, you will own the asset," and there's some agreement in terms of where future interest rates or revenue will come from. That's its own rabbit hole. But the database that managed the relationship of the prices, or the rates that would get paid by both parties, was not just a time database, it was like a time-of-time database. So someone might come in and say, "Hey, remember the deal was from January to August, and now we're in November." Oh, we have to go back and update that and then rerun everything again to make sure that the prices look good and are correct.
Trying to think of another example. Even just exchanging calendars. So I think people just assume, oh, every exchange in the US is largely the same. That is generally true, but trading hours can be different at US exchanges. Different countries have different holiday calendars. Some countries are more religious, some are less religious, so religious events can impact different calendars.
I worked on one system where you could pick the calendar of the thing that you're trying to do, and it was in alphabetical order, and the first one on the list was the Beirut Stock Exchange, and I always wondered, "What is going on at the Beirut Stock Exchange that we have to have this calendar?" I never found out, but there are a lot of different knobs that can be pulled and turned in that sense.
Patrick: I lived in Japan for 20 years and will probably be back there in the future, and while Japan uses the Western calendar in many, many places for a variety of reasons, there are many computer systems and, you know, APIs talking to each other which expect you to express historical dates, the current date, or even potentially future dates in the Imperial calendar, which uses the reign name of the current emperor plus number of years since that has been incepted.
And so when there is a new emperor announced, you get a bit of time in Japan for everyone to do a crash update to all the computer systems to be able to express the right range of dates in terms of the new reign name. And, you know, boundary conditions aren't fun to be able to program for.
Alex: Even time is a good example. I've worked at some firms where daylight savings, not a big deal. Like, all the systems just kind of click over. Everything works correctly. Other firms, it was a disaster every — 'cause, oh, this started, we didn't account for this. This had to start before this other thing.
Patrick: It was always a disaster for me when I was running a company that was quite calendar-oriented. And there's a document that was written by a very senior engineer who worked on Google Calendar who was trying to explain all the intricacies at the appropriate level of documentation for Google Calendar, and it was clearly driving him to gibbering madness.
Like, if you have a meeting that is in two time zones at once, and then the time of the meeting changes, it is beyond what humans can — like, it is unsolvable in the general case how that meeting should change in both time zones. Like, it cannot be done. You have to make assumptions about geopolitics, among other things. And so calendars drive people to gibbering madness all the time, and yet we still have to be able to make some sense of this every day.
I think one of the reasons why more of this is manual than you would expect is that historically there's been some amount of slack for something unexpected came up. You know, a counterparty of ours had a bad day, et cetera, et cetera, et cetera. And now people who have a great deal of expertise on what should happen on a given day need to, in a very short amount of time, come up with something to adjust for a hiccup. Do you have any fun war stories there? I've got a couple I can relate myself.
War story: twenty million shares out the door
Alex: Yes. So there's one — it was a Friday afternoon, and there was a particular desk, and they're trading a particular product. Relatively volatile, big product. And it's four o'clock, and we think we're done for the day because it's Friday at four PM. The trades are done. It might have even actually been after four fifteen, because some things happened at four fifteen.
And one of the traders runs over and just, like, is yelling at us: "Product name, product name, product name." And we're like, "Whoa, whoa, whoa, whoa. What's going on? What's going on?" And she just kind of runs off, and we're like, "What is happening?"
And so we go in and we look at the order flow and execution flow, and effectively what the client had said was — going back to the other part of the conversation — "I'm gonna send you twenty million shares of this product. I want you to execute five of these, in quotes, manually," which means it'll just be one big execution that the firm would take the position on, "and then the other fifteen you can just send out to the street, and whatever price you get at the street is fine with me." That was the plan.
The system was designed to have a single trader working a single order from the customer. What the head trader had decided to do was say, "Hey, junior trader, you do the street orders, I will do the manual fill." So he has the window open and he's working on it, and he hits execute assuming that five million will execute, and then the junior trader will basically send another 15 million to the street. Because she had already started sending orders out, the manual order window kicked off an error saying, "Hey, when you opened the window there was 20, now there's 12, rejected," and she just assumed that it would just all go out. And so 20 million shares went out to the street rather than at the price we agreed on with the customer.
So now the market is closed, and we have this giant position potentially that could move over the weekend, and we end up having to talk to the desk, talk to risk, come in on the weekend, put in a futures order to hedge essentially, not all of it, but a large portion of it. And that also became a technology error, and one of my other jobs at that particular firm was a lawyer for the defense in defending technology from errors claimed by the traders, which is its own conversation if you want to get into as well.
But yeah, it tends to be a lot goes wrong very quickly, and if you can solve it before the cutoff date — of which there are many, and some of them are in the morning, some of them in the evening, some of them are late at night, or they're overnight if you're dealing with Asia — and if you can fix it, great. If you can't, then the cleanup process can take longer, you have to call people.
One of my other favorite examples, this is at a different firm. We did not get our positions out to the broker-dealer — or sorry, the prime broker — and the prime broker was a large bank, and we were talking to someone there, and they said, "Oh, sorry, we have to wait for Budapest to run the end-of-day job." And I remember sending a meme to the head of clearing of James Bond smoking, being like, "We have to wait for Budapest dollar," kind of thing.So yeah, it's some interesting times.
Ops as an iterated game
Patrick: One of the things I'd like to highlight here is a thing that ops does all day, every day is talk to ops at other institutions, and this is an iterated game. Like, it always feels like a crisis in the moment. It sometimes is a crisis in the moment, but mostly you expect to be there tomorrow too, talking to the same people tomorrow.
And there's some amount of sort of usage of relationships to say, "Okay, in the case where we've made a commitment to a client and we didn't do the thing that we said we would, potentially there's a way to call the client back and say, 'Whoops. Okay, so on one level, you might have like a contractually enforceable right to blah, blah, blah. On the other level, we have a relationship that we both mutually value. Can we go halfsies on this? You know, you absorb half the risk, we absorb half the risk, yada, yada.'" And that obviously is not something that you want to do all that often, but it's an expedient resolution in a lot of ways.
This sometimes surprises people who are not professionally involved in this that, you know, in a culture which is often derided as being very zero-sum and merciless, et cetera, et cetera — that like the culture of the trading floor for the longest time was that there is a formal resolution process for determining, like, if two people disagree on reality as to whether a trade happened, did that trade actually happen? And if you engage in that formal resolution process, you've done something wrong.
Like, the first thing to do is to say, "Okay, I thought I sold at seven. You thought you bought at eight. Price is seven fifty," and we shake on it. And people think, "Well, isn't that abusable?" Et cetera, et cetera. But these are high-trust environments. You are not going to burn your professional reputation over one transaction on one day, typically, typically. And so the equilibrium kind of holds.
And then as we sometimes see in Silicon Valley, the exception case is like when an iterated game suddenly becomes like a one-shot game, as, you know, dramatized in Margin Call or similar, where it's like, okay, this is an industry-ending apocalypse. Okay, that's an overstatement for what 2008 did. But it was certainly a firm-ending mini apocalypse for at least some firms, at the degree of, like, we are willing to do something which is acutely against the interests of the people that we are talking to on the phone goes far up versus like the typical day.
Alex: There are many things to say about that. One quick sidebar: at the end of Margin Call, you see multiple trading floors, and they'll say like Deutsche, Citi, et cetera. Those are actually all the same trading floor, and it's the trading floor I worked on. And if you watch the movie, there's a large trophy in one of the scenes, which I think was either the flag football or dodgeball trophy. So yeah, I have a soft spot in my heart for Margin Call.
Busting trades and who has the authority
Alex: So I'll give an example. In general, if you are trading and you what's called fat finger an order — so you buy maybe ten thousand when you meant to buy a thousand and you move the market — you can speak to the exchange, and the exchange will say, "Okay, we are willing to... if you wanna file what's called a clearly erroneous, we will bust that trade," AKA we will pretend like that trade never happened. That's sort of the best case.
A slightly less good case is sort of what you were describing, where they will say, "Okay, well, we're not gonna bust all thousand shares. We'll only bust three hundred or four hundred."
And kind of a very famous example — 'cause I ended up going to work there — was Knight Trading. I was at a firm that Knight Trading did business with, and they called our back office team and were like, "Hey, will you bust a trade with us?" And some of them, they were willing to bust, and some of them they weren't. And famously, some of the banks did not bust any trades, and so they were able to shrink their error a little bit but not a huge amount. But your point around the many cycles of the game — you want to have your customers come back to you, and if you consistently make errors and try to make the customer eat it, then it's gonna be a problem.
Patrick: It's sometimes unclear even with formal rules, et cetera, as to who has authority on things. And in the moment a crisis happens, like who has the authority to countermand an order?
There's a famous example from Japan where there was a particular security which — the correct price for the security is approximately six hundred thousand Japanese yen, which is a few thousand dollars. And for various reasons, like, the anchor point people have for the price of a share of stock is usually, oh, it's probably somewhere in the tens of dollars, hundreds of dollars these days. But this particular one, it's about $6,000. And a broker intends to sell one share at $6,000. Instead, they sell 600,000 shares at one yen, which is a fraction of a cent.
And this order gets to the stock exchange, and people at the stock exchange — and the regulator was furious about this in the aftermath — people at the stock exchange looked at the order and were like, "Oh, they obviously swapped these two numbers. But I personally don't think I can countermand that bank, and, well, I'll just let it go." And so everybody let it go, and the firm ended up taking a, I believe, several hundred million dollar loss on this because obviously, you know, the price of this thing falls to near zero in a way that is unexpected and incorrect. And indeed, there were some — I believe it was publicly reported, I'll see if I can substantiate this — but an American firm that is well known for having sharp elbows was reported as having successfully bought a lot of the stock at, you know, a price not too dissimilar from one yen, and then had a great day for itself.
But the markets kind of, like, have to keep turning. The world has to keep spinning despite one firm having a really bad day. And this is a relatively successful property of them that I think people don't appreciate enough. Like, when Knight Capital blew up one day, there was a — it's a long story for people who haven't heard it, but a firm, due to technical issues, suddenly became acutely institutionally unaware of what its positions were and started firing orders off willy-nilly and essentially spending money it did not have.
And it probably surprises people that you can spend money that you don't have, but that is indeed a key function of financial markets. And they spent much more money than they would ever possibly have. It's really bad news if you work there. It's bad news if you were equity partners in Knight Capital. But everybody else was basically fine by the end of, plus or minus, the next day. Like, "Okay, they blew up. We all have risk functions, so we were only exposed to this to the extent that we expected to be exposed. Some of us got windfalls because they executed trades at prices that were very bad for them, and all right, we're fine."
And, you know, socially speaking, if in two thousand eight a large investment bank that has a million lines of business goes under, you're in a really rough way because some of those lines of business are extremely socially beneficial and difficult to substitute for — and that's where too big to fail comes from. But if a trading firm goes under, like, Knight Capital is a pile of capital and some smart people and computer programs, and money, smart people, and computer programs can be reconstituted under a new name and a new LLC. We're fine. The markets are structured by dueling robots, and if one dies, eh, we'll replace it.
Inside the Knight Capital collapse
Alex: So I actually worked at a different firm and had resigned on the Monday before Knight Capital blew up because I was actually going to Knight Capital. So I was also kinda interested to live through that as like, "Oh my God, what is going on here?"
And I think, if people have — there is an, I believe it's an SEC report, where they go through all the steps. That was a very accurate accounting of the things that went wrong. And sort of two things. One you already highlighted and one I'll add. One of the items in that review was who actually had the power to say, "No, we are going to stop trading," versus, "Oh no, we need to stay up and trading." And that is because some trading firms actually have agreements with their customers that they have to be up a certain amount and/or they are the exclusive venue for that customer. So that in a sense worked in Knight's favor in the aftermath, that some people could really only trade with them or primarily trade with them. So there was an incentive for them to get injections of funds to basically come back and keep going.
So yeah, on one level it's very simple. It's I put up stuff to buy or sell, people buy or sell it from me. But then all the different agreements, and especially for the OTC stuff, like sometimes there isn't a central exchange, so it is all basically agreements back and forth. Yeah, it's a giant web of interconnectedness which can move around in interesting ways.
Agreements, obligations, and who you have to call
Patrick: I like that you brought up this concept of having agreements because, you know, there are piles and piles of spreadsheets and computer code created for this. But there are also piles and piles of either handshakes between people, recorded handshakes between people, or very long, exhaustively drafted legal documents between very well-capitalized firms.
And what have we agreed to, and are we in compliance with those agreements today? And if we are not in compliance, you know, what is the resolution path, at what point do we need to tell someone and call them and say, "Hey, between eleven fifty-eight and one thirty-two today, there was a period in which the thing that we had contractually committed to do, we did not do. And, you know, how do we work through this?" Are all things that basically falls on operations and the back office — to like be aware of this constellation of things that governs our behavior and also just like do the mechanics of making sure that it happens every day.
Reconciliation, FTP files, and the humans behind them
Alex: Yeah, so you hit on the part where you mentioned, "Oh, how do we make sure that between the two companies we kind of agree that this is what had happened?" So I think for the casual listener, much like people balance their checkbook, firms balance their checkbooks against the counterparty's checkbooks. And I can talk a little bit about how that happens in practice from both a technology perspective, then also some of the systems that are built to help kind of reconcile that.
So at one firm I owned — it was creatively called, I think, external data, and was basically the system that would FTP to counterparties and say, "Hey, give me the file that has all the trades that you think we did with you, and then we will reconcile that against what we have internally." And people's first thought might be like, "It's FTPing CSV files around. How hard could it be? There's not that many issues with it."
There are many, many issues. One is just even checking what file is there. So for example, some firms will drop one file for every business unit in the counterparty. So whether you have 1 or 15, you get one file, and then somehow in there you have to kind of figure out, okay, this is for this department, this is for that department.
Some firms will create the file, and as soon as you download it, they delete it. And so if you lose that file or it gets corrupted, you have to call somebody in Budapest, and they have to remake the file. Some people never delete the file, and so you have to check every single file in the FTP server to make sure, like, hey, is this a file that we've already downloaded?
Some firms — one of my favorite examples is one of the banks. So if you traded one asset class, you got an actual XLSX Excel spreadsheet file with one sheet, which was just the trades you did. But as soon as you went to two asset classes, they would add an extra sheet at the beginning that was a summary for each of the additional asset classes. So even if you had a script that was saying, "Oh, for the bank, let me pull down this file," you had to know, wait, is there a sheet? Is there not a sheet?
And there were teams of people — to your earlier question — whose only jobs were to basically every day go through and be like, "Did it come through correctly? Did it not come through correctly? Why not? Was it our issue? Was it their issue?" Et cetera.
Deadlines, cutoffs, and the $36 million phone call
Patrick: Yeah. And a very important sort of recurring pattern that we have here is that there's a process that is known to fire every day. There is a time where it is like, we usually expect it around X, and then there might be a little bit of, we get worried if we don't see it by Y, and then after Z it is capital-L Late. And those have different levels of consequences associated with them.
In credit card processing, which I will always have a soft spot in my heart for, there's a similar thing where you might do net settlement with a bank, for example. The bank has many customers that have credit cards. They go out into the world and spend on those credit cards every day. There is some amount of them also having merchants with credit card processing accounts who are having payments coming in every day. And so somebody else in the chain might think, "Okay, with respect to a particular somewhat small bank, you owe us $36 million on net today. Do you agree with that number? And communicate that by," I'm going to make up a time, "4:15 Eastern Standard Time every day. And you can get it as late as 4:30, but after that it is capital-L Late."
And that is not a particularly reasonable time for that, but I'm fudging things. It will probably be right around the cutoff for Fedwire at the end of the day, because the cutoff for Fedwire is the easiest way to transfer like $36 million to another financial institution.
And, you know, let's say at a relatively small bank there is an employee who owns this process — we'll call her Martha — and Martha's on vacation. That doesn't absolve the bank of the, like, acute necessity of making sure that file comes through every day. But it might have happened once or twice in capitalism that Martha was indeed on vacation and that the bank had underperformed its responsibilities for having a backup in place.
And then what happens? So sometime between 4:15 and 4:30, a series of increasingly urgent phone calls starts happening, which starts somewhere in the bowels of operations and then goes as far up as it needs to to say, like, "Hey, just checking. We really think that you guys owe us thirty-six million dollars. One, do you agree on thirty-six million dollars? Two, is the wire going to happen?" Because if the answer is ultimately, "No, the wire isn't happening today," then someone is out thirty-six million dollars.
And you don't really have the option of, okay, tell all the sandwich shops and gas stations and power companies and similar that they're not getting their money on time. Like, someone is going to take minimally — for a point of time, they're going to take thirty-six million dollars of risk. And that someone needs to be told that as early as possible.
If this happens on, like, an ordinary Tuesday, well, everyone has had a bad Monday before, it happens. If it happens during a time of, like, market stress or widespread panic about the economy or something — you know, in the case where, like, a bank, "Whoopsie, we owed another financial institution thirty-six million dollars on Monday," but Tuesday rolls around, we're still a bank, we're still good for it. In the case where that happens on a Friday and there is, like, a great deal of worry about how many banks the United States will have on Monday — where that number doesn't ordinarily change that much over the weekend — the temperature level in this conversation goes way the heck up.
Alex: Yeah, there are kinda many threads here, and I'll kinda zip through some of them. One, we got a new head of clearing in a past firm I worked at, and someone asked him — and I think he had come from a large bank that did prime brokerage — someone asked him, "What is your nightmare scenario now that you're on this side?"
And he goes, "We trade through a broker. At the end of the day, the broker sends us," very similar to how I was describing before, "we get a list of the trades, and we disagree with the broker on what we think our total number of trades are, AKA what our current position is. So we might say, 'Hey, we are long security X to the tune of a million shares. You think we're long seven hundred thousand. We should talk about this.'"
We talk about it. It turns out at the end of the day, at the end of the whole process, that we are correct and the prime broker is wrong. The prime broker, as part of our agreement, might say, "Well, I'm not gonna let you trade out of this because we have to figure out what the actual position is." To your point, suddenly product X either — if we're long, drops a lot, and it might be so much, either if like a bankruptcy or something happens to the CEO, or some giant lawsuit comes out, et cetera — that could effectively kill our firm. And it happens. This is a thing that has happened more than once.
It's sort of like on the other side, the prime broker will do this to different sides' parties, and it could just be an error on their side. It could be a technical error, procedural error, fat finger error, et cetera. So it can get scary and very high stress from the procedural side.
How two banks fail to agree on fourth-grade math
Patrick: Let's ask a question on behalf of the civilians in the audience who are just, like, gawking in amazement right now. How can two banks fail to agree on fourth-grade math — even fourth-grade math over a large number of transactions?
Alex: Sure. Many, many ways. So one is just straight-up technical errors. So I guess we'll move into kind of the war story part.
I was at one firm where there was an algorithmic trading system. So people would put in some math, and they'd say, "If these numbers are above this value or below this amount, send these types of orders." There was a bug in the code that flipped the symbol on the executions. So if it was an execution for Microsoft, it got flipped to Apple. Like, I'm using explicit examples, but at the time this was happening, we had no idea what the key was of which ones had gotten flipped where. So you might send position reports back out to the other bank, and you thought you were long a million, but you were actually short five hundred thousand because everything got flipped.
And at the time, I think the workaround was we basically pulled all the executions, got a unique list of symbols, went to Yahoo Finance, figured out the day's trading range, and essentially had to do a join on the fly of, "Okay, this trade is for forty-seven dollars, but Apple didn't trade below ninety today, so this one's probably wrong."
There are a lot of hops in the system. I think most people think like, "Oh, those trades come in, and they immediately get booked." There could be, you know — there's the exchange, then there's exchange gateway between the bank, or the firm, and the exchange, then there's the crossing engine, then there's the trading system, then there's the sales trading system, then there's the booking system, then there's the clearing system. Like, any one of these things can break it in any particular way.
And one thing I also wanna highlight: these don't all necessarily get resolved, like, the day of, the next day, or even potentially weeks later. One firm I worked at, they had a dedicated system for basically tracking trade breaks where the two parties didn't agree. And I think the longest trade break in there was four or five years 'cause a firm went out of business, or the product's not traded anymore, or there's a lawsuit, and nobody can trade out of it. And so you're carrying that position/risk for years, potentially. So this can happen in milliseconds or it can happen in years, respectively. So it's a pretty big time range.
Patrick: Yeah. The, quote-unquote, "Why don't the accountants just catch it?" is something that sometimes gets brought up to people. And sometimes the accountants do catch it, but accounting typically happens on a much more long-term basis than the day-to-day work of operations.
And so a later audit might — as a, like, I don't know, with respect to various low frequency pathways of what can go wrong in a credit card transaction, a financial institution might discover, okay, there's a nation, let's call it Mexico, and due to a curious implementation decision, 0.01% of all transactions which were transacted in Mexico failed to get money from counterparties where we expected to get money from counterparties, and we discovered that six months later.
Now multiply transaction volume for the nation of Mexico times 0.1% times six months, suddenly becomes like a pretty big number. And then your resolution options at that point can be somewhat limited because, you know, okay, making up a hypothetical number, do you want to call 108 million people in Mexico and ask them to please send a check in for an average of $20? Probably not that viable. I'm filing the serial numbers off of real incidents that have happened for obvious reasons.
Okay, we have the case of technical errors. I also think that a lot of finance runs on trust, particularly in certain relationships, and there are reasons to say, "Okay, I think this particular firm is — you know, this is not a fly-by-night startup. These folks have an excellent reputation. They are not particularly thinly capitalized. I'm going to just trust that their numbers are right for maybe twelve hours more than I would if they were one-thousandth of the size that they are." And they make mistakes at some rate. And so now my understanding of reality is twelve hours behind, and twelve hours can be a very long time some days.
When the golden record goes down
Alex: Yeah, that's definitely even true internally at firms. So I was at one firm where there was a centralized position management server that was the official golden record of what are all the positions in the firm. And that had an issue, and what ended up happening — so I was the incident commander for a large portion of that. It was overnight. It took like twenty-seven hours. People operating on very little sleep.
And then the next day, it basically came to light that people knew potentially that that position management server could go down, so they would keep their own internal positions as a double check basically on the system. So when they would fire up in the morning, they would reconcile between each other.
And if you hear me say the word reconcile a lot, most of modern finance is effectively always reconciling between all the different nodes in the network all the time. Because you let one — to your point earlier, you let some of these trades, e.g. like a currency exchange trade, might seem small or just one fill, but it could be potentially hundreds of millions of dollars. So you let one through, and it's a gigantic deal also.
Patrick: So what's the downside, if anything, if people are maintaining their own books and they match, like, the system-wide view of risk most of the time, and then system-wide view of risk goes down one day? Dot to dot, what happens next?
Alex: Yeah, that's a great question. So even by — I don't wanna say by design, I guess by historical artifact — different firms have very different position management systems. So for example, I believe it was Goldman was very famous during the 2008 crisis for they had one central monolithic database of all of their positions. So if somebody wanted to say, "Hey, how are we exposed to XYZ?" it was a couple keystrokes to essentially say, "Oh, well, we are long on the one side, maybe on the equities, but we're short on the options, and those net out, so we're flat."
Other firms that were conglomerates or were essentially multiple firms that had been acquired over time had many disparate systems. So their cash position management might be different than their options, their equities might be different than their options, it's different than their treasuries, et cetera. So it could be very difficult to get a centralized picture.
In the world where you do have a centralized thing and then you have sort of satellites, when you come back up, you don't wanna just say, "Hey, let's go with whatever one is up," or, "That's the main one." Again, you have to go back and you reconcile, and in a perfect world, you had thought ahead about this as part of your outage management planning and disaster recovery, where there's a checklist or a book that's like, "Hey, if the big position management system goes down, do these things." There's that, and there's tools to do that. If you're unlucky, there's folks like myself who are writing Perl and Python scripts on the fly to basically make sure that, hey, can we connect everything? Okay, it's in a different format. Okay, we didn't think about that format, et cetera. So yeah, the more organized you are there, the better.
Finding the one person who can sell the position
Patrick: And one of the things that was widely reported in two thousand and eight was we suddenly went from, you know, the — for lack of a better word — peacetime footing to crisis-time footing in a variety of sub-sectors of the financial industry. Because Goldman knew what Goldman's net position was, when things start to hit the fan, they were able to quickly orient around things hitting the fan, close out positions and, you know, redeploy money to, okay, well, it's a really bad time to be in finance right now, maybe we should open some shorts.
Whereas other people were like, "Okay, I'm probably very exposed here. Can you get me an answer by the close of business tomorrow on am I right about that?" Or, you know, even things as simple as, "Okay, I think on net we're very long something, and we need to sell out of that. But I'm not sure who here is long that something."
And it's not, you know, at the scales of an investment bank, it's not E-Trade. I can't simply click like, "Yeah, sell all of that, please." Like, I have to find someone in this office that I can call that has, like, legal title to the thing that my screen says that we have, and can, like, physically affect the sale of it to, you know, decrease the firm's overall exposure to this and free up cash and yada, yada, yada. And even identifying that person is non-trivial when there are 100,000 people who work for you.
Alex: That is definitely — yes, 100%. And like the example I was thinking of when you were discussing this is, sometimes you'll see a photo of — it's effectively the system, the number of computers you would need to be in compliance inside the government for sending secure messaging, and it's like eight desktop computers because there's a Defense Department computer, there's the Secretary of — sorry, there's the State Department computer.
It's very similar at some of these firms where your position management view for equities is on one system, but then there's a completely different system for treasuries, a completely different system for options, et cetera. Sometimes they talk to each other, sometimes they don't.
How much business do we do with Fidelity?
Alex: And then even — I'll give another example. Going back to earlier in the conversation about the different entities, the head of trading one time asked, "Hey, how much business do we do with..." I'll use the real company 'cause everybody does business with them. It's Fidelity. And you would think, "Oh, how hard could it be? Just go on this database, you know, search select star where customer equals Fidelity." And then you realize there are different entities of Fidelity, then there are different companies that have Fidelity in their name, like First Fidelity Bank of Random Town, USA. And these things grow over time organically.
And there, I have a real-world example of a technical version of this where, in a lot of these places, there's a point in time where someone should say, "Hey, it's getting out of control. We should re-architect this to make it simpler." If you don't do that, it gets so big that you just can't do that. Like mainframes at banks is a good example. They've been there forever, nobody knows exactly how they work. It would cost millions and millions of dollars and five years to get off of it, and nobody wants to make that giant decision 'cause it's unclear if it's gonna work, et cetera, et cetera. So yeah, a lot of tech debt effectively.
Patrick: I like the example of how much business do we do with Fidelity, because something that might not be in people's model here is, regardless of how trusted an institution is, there is maybe one institution that is exempt here, which is the United States federal government, because they can print dollars at will. But for everybody else, there is some level of business that you're willing to do with them and no higher. Because if they have, like, the worst day possible and their firm goes down, we can't go down with them.
It's extremely complicated at the level of, like, globally significant financial institutions — like, you know, just to pick two names, not to pick on two firms, but it would take an extensive science project involving thousands of people probably, the better part of a few years, to like completely figure out how many ways that Bank of America and Chase are tied at the hip. But there has to be an answer in some places. And, you know, there has to be an answer for, okay, if you are a credit card processor, how much business do you do with Visa every day? And that's a weird one. But you can imagine, like, vice versa, you know, Visa, like, okay, there are many people on the Visa network and each of them have like some limit that we are comfortably exposed to — even though I'm hand-waving away a lot of complexity here.
Prime brokerage and extending credit
Alex: Yeah, I can use a real example. So I worked at a prime broker, and that comes in — so one of the functions of a prime broker is you might have a lot of small customers, or customers at different... They could be large customers, but they just don't do a lot of flow, particularly in, let's say, like crypto.
So what you do as a prime broker is you go to an exchange and you say, "Hey, exchange, I'm gonna have a session that's tied to me as a prime broker, and then I will be acting as a middleman for a lot of other customers that will flow through me." So A, I get the volume discount of all these customers. I can give them a discount on basically going directly to the exchange. And, you know, you might be dealing with very small, unsophisticated market participants that are just a couple guys in a conference room somewhere in Florida, let's say, and they're like a little day trading shop. Or you might be dealing with different parts of a giant bank or corporation, but there's a US entity and a European entity, et cetera.
And so it can be tricky to sometimes figure out, A, how much volume, and to your point, even who to call. Like, do we call the Euro guy? Do we call the US guy? These are all real questions that pop up, particularly if there's an outage and things are breaking and you need to call somebody — that person might not be at the firm anymore. So yeah, to your point, you just assume it's all accounting, but then you realize there's a lot of fuzziness, and you have to get to the right person and get them on the phone, and they have to know how to handle it, et cetera.
Patrick: And one of the functions of prime brokers is extending credit to people. Lots of trades happen on credit for a variety of reasons. One, it magnifies returns. Another reason is that basically credit functions in many places as sort of like lubricant in the economy, where if we were doing everything cash and carry, if there was no credit extended between firms, there just simply isn't enough money to be at the various places, and that would generally have the effect of, like, raising the cost of capital to various firms, slow things down generally, et cetera, et cetera.
It is broadly underappreciated by people that even if you look at your, like, retail checking account statement, the number that you believe is your balance is, like, not necessarily your balance. That is the roll-up of a number of credit decisions that your bank has made on your behalf. And there are other people that are making credit decisions vis-à-vis your bank, et cetera, et cetera. And so your quote-unquote settled balance is like a probabilistic statement about what we believe about millions of transactions at this exact moment in time. Comma, be that as it may.
At some point, somebody at a prime broker has to say, "You know, a couple of years ago when we got into this relationship, we were happy to have them. And for every day since, we've been happy to have them, and I'm not sure I'm happy to have them tomorrow." Or, like, previously, you know, we had extended this particular client a credit line of up to — make up a number — three billion dollars. And maybe they're still good for two billion of that, but I don't really wanna lose the last billion. Maybe we have to have a communication with them, like, "Okay, well, if you own three billion dollars of assets, maybe own less very quickly."
There's often an or-else involved in those. Do you wanna talk just very briefly on what is the or-else?
Alex: Yeah. Well, sometimes there's just giant events. So for example, in 2008, like, Lehman just disappeared one day, or Bear Stearns just disappeared one day, and you had all of these interactions, like you were mentioning before, about being tied at the hip. You don't even know who to call. Like, you call the number and no one answers, right? They've shut the office down, and now you have these giant positions you have to deal with.
I'm not as familiar with some of the lending, like the pure prime brokerage around, okay, this person is — now we don't think of them as a good customer. But there are definitely amounts of order flow that you'll take from certain people. Sometimes it's not a credit thing. It can be also this is very smart flow or very dumb flow, where we don't wanna take the smart flow. We'd rather take, of course, the dumb flow. Some people might say sharp and not sharp. So it is definitely a thing that relationships can get cut off quickly, and they can get cut off for different reasons, et cetera. So yeah, definitely a thing there.
Smart flow, dumb flow, and payment for order flow
Patrick: The classic example on flow is payment for order flow, where retail orders — people transacting their accounts at Schwab or Robinhood — are almost definitionally not hedge funds. And so it is extremely unlikely that they have material market-moving information. And so they're transacting because daughter is going off to college, or this is the day that my, you know, 401(k) gets the transfer from work, and then I immediately buy, or et cetera, et cetera.
And so if you knew you were just dealing with noise traders, with people who didn't have a point of view on the true value of a security, or who had no better point of view on it than the market does, you would be able to quote them very generous prices. And to the extent your non-informed flow is mixed in with people who are quite informed flow, well, you have to be a little careful about what prices you quote them.
And then if it is Goldman Sachs on line two, you should be very careful about what prices you give to Goldman, because, as one great line of dialogue from The Big Short movie: "This is Goldman Sachs. If you offer us money, we're gonna take it." And it's deployed against their interest there, but broadly a good rule.
Yeah, there are layers upon layers of this. I feel like we could discuss war stories for a very long time, but one thing I wanna just emphasize for people is this is intellectually interesting and keeping the world running. And there are fun challenges every day. And I'm glad that you came on to share some of those fun challenges.
So where can people follow you on the internet if they want to read about stories in the future?
Where to find Alex
Alex: Sure. So I'm very active on Twitter. My user account is alexpotato. I have a whole bunch of threads about all kinds of different topics. A large portion of that is about finance or technology. If folks are interested in what it's like being a technologist in finance, there's a couple there. My website is alexpotato.com. Also have some book recommendations, and my own podcast is there as well. So yeah, this was great.
Patrick: Yeah. Thanks very much for being on the program, Alex. And for the rest of you, thanks very much for listening, and we'll see you next week on Complex Systems.
Thanks for tuning in to this week's episode of Complex Systems. If you have comments, drop me an email or hit me up @patio11 on Twitter. Ratings and reviews are the lifeblood of new podcasts for SEO reasons, and also because they let me know what you like.
