Two banks can't agree on fourth-grade math | Alex Elliott

Patrick McKenzie (@patio11) is joined by Alex Elliott (@alexpotato), who has spent his career in SRE and operations roles at financial firms ranging from ten-person fintechs to global investment banks, to discuss what the back office actually does all day. They cover the life cycle of a trade: execution, clearing, settlement, and the constant reconciliation between counterparties who don't quite agree on what happened. Alex recounts the Friday afternoon when twenty million shares went out to the street instead of five, and Patrick explains what happens between 4:15 and 4:30 when one bank owes another $36 million and the person who owns that process is on vacation.

Presenting Sponsors: Mercury, MongoDB & Granola

Complex Systems is presented by Mercury—radically better banking for founders. Mercury's new feature Command brings an LLM directly into your banking interface, so checking balances, finding invoices, or sending a wire is as easy as asking. Apply online in minutes at https://mercury.com/.

If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS

What's the point of building faster with AI if your database can't keep up? MongoDB's native data model mirrors the language LLMs already speak. Ship at the speed of AI while staying ACID compliant at Fortune 500 scale. Start building at https://mongodb.com/ai.

Timestamps:

(00:00) Intro
(01:48) The life cycle of a trade
(04:37) Why you don't know who you traded with
(06:01) Hiding a large block trade
(10:04) Information walls inside financial firms
(12:49) What the middle office does: managing positions
(15:41) Calendars, time zones, and other sources of gibbering madness
(19:37) War story: twenty million shares out the door
(22:45) Sponsors: Mercury | Granola
(25:50) Ops as an iterated game
(28:53) Busting trades and who has the authority
(33:38) Sponsor: MongoDB
(34:29) Inside the Knight Capital collapse
(35:49) Agreements, obligations, and who you have to call
(36:57) Reconciliation, FTP files, and the humans behind them
(39:16) Deadlines, cutoffs, and the $36 million phone call
(44:26) How two banks fail to agree on fourth-grade math
(48:54) When the golden record goes down
(51:55) Finding the one person who can sell the position
(54:09) How much business do we do with Fidelity?
(56:38) Prime brokerage and extending credit
(01:01:03) Smart flow, dumb flow, and payment for order flow
(01:02:45) Wrap

Transcript

Patrick: Hideho, everybody. My name is Patrick McKenzie, better known as patio11 on the internet, and I'm here with Alex Elliott, a Twitter buddy of mine who has worked in a variety of operations roles as a financial technologist across a number of financial firms.

Alex: Hi, Patrick. Nice to meet you in person, I guess, or over the podcast. Yeah, I'm Alex Elliott, alexpotato on Twitter. And yeah, as you mentioned, I've worked primarily in SRE/DevOps at a bunch of different financial firms ranging from small fintech startups with like ten to twenty people, all the way up to giant investment banks with like hundreds of thousands of people as well. 

[Patrick notes: Site Reliability Engineering is a rebranding, largely instigated by Google (c.f. Site Reliability Engineering: How Google Runs Production Systems, O'Reilly 2016), for what used to be called system administration. Sysadmins were considered blue collar in tech; skilled artisans who maintained configuration files and wired machines together. Google’s intellectual contribution was that there should be real engineering discipline in running global-scale systems, which requires writing config files and wiring machines together, and therefore the people who do it are engineers and deserve pay, status, and authority as such. (SRE as practiced has engineered levers for telling the business No. One of them is, essentially, “Either you’re going to have engineering implement a fairly expansive list of things we need you to do, or, we will abandon what engineering builds for you to the fates. The machines can page you if they have a problem. Please check with other executives on what will happen to your evenings if you think this is an idle threat.”)

I think often about the SRE/sysadmin distinction in discussions of Operations, because Operations has also historically been a cost-center backwater at the largest firms in the world. This is unjust and a root cause for many, many failures we observe. Sadly, nobody has written a snappily titled book to give the back office their day in the sun. I continue to think it will happen, as the most successful companies in the world have large, empowered Ops teams and at least a few of them don’t think “eh back office is a cost center to be done by the cheapest possible person globally at an acceptable level rather than something we want our best people doing.”]

Patrick: So I think if people have read about finance for a while, they might have come across this concept of the front office, which is the high status, highly visible customer-facing function, and then the back office, which is often caricatured as, "Oh, those are the geeks moving numbers around," and perhaps paper emanates at some point. And we need them, but we don't love them.

But I love back office minutia to an almost unhealthy degree, and I think it's sort of one of the unsung heroes of the ecosystem that literally keeps the world spinning. And so wanted to talk to you about what the back office actually looks like at some varieties of financial firms. 

What do you do all day? And then maybe talk through some war stories that show, you know, if this stopped spinning, it would be very bad in a hurry.

So pick a genre of firm from your vast selection and talk about what the back office does all day.

The life cycle of a trade

Alex: Sure, yeah. At broad strokes they're very similar, so I'll do a little bit of the life cycle of a trade in sort of a very fast overview.

So somebody will generally say, "Hey, I am selling, let's say, a million shares of Microsoft." We'll call that party A. Party B will say, "Hey, I wanna buy those shares of Microsoft." And generally party A has a price, party B hits that quote, and then basically an execution happens, and now both parties have this transaction between the two of them that they need to agree upon that it occurred and then make sure that the money and the securities or assets all moved on their own.

Patrick: And importantly for this, many people have this assumption that, oh, that's what a stock exchange is for, right? The stock exchange must take care of all this for you. And while that rounds to true in crypto — that the exchange is the same one that is doing the custody and the clearing and settlement of things — that largely isn't true in the traditional markets. There's a separation of functions.

[Patrick notes: A noted crypto luminary once suggested to Congress that the U.S.’s encoding in law of the separation of functions in the financial industry was unnecessary. Crypto had the better path: put settlement, clearing, custody, and trading all under the same roof. It would reduce costs, speed trading, decrease cost of capital, and minimize errors.

I criticized SBF’s proposal contemporaneously, not because I thought he was going to use the single point of failure to steal the customers’ funds, but because it was an obvious SPOF. Double entry bookkeeping being done by multiple firms adversarially with each other, even though it is immensely costly (as discussed below), is a feature.]

Alex: That is correct. And we could go down a long rabbit hole in the different types of flow as well. So for example, if you are prop trading, you might be going to an exchange and someone else is going to an exchange. It's the two parties. In traditional investment banking trading, there could be layers, e.g. there's a trader, then there is a sales trader, then there is a customer, which is usually like a pension fund. CalPERS, the California state employee retirement fund, is one of the giant ones. But it's true of many different large customers. It could be other hedge funds, sometimes other banks.

[Patrick notes: When CalPERS, which has more than $600 billion in assets, needs to “roll” a bond at maturity, it might need to buy several hundred million dollars of a single issuance. Which you can do, for say a particular maturity of a Treasury. But if you did it the most straightforward way on e.g. a municipal bond, which pension funds own in quantity, that would be very bad for the realized results of public employees’ pensions. Broadly, one reason why pension funds pay the financial industry is to do the sensible thing on execution. (One can make the reasonable argument that they pay way too much for this, but that is another discussion. Broadly, it’s extremely unlikely that anyone will make bonds decisions that outperform the broader bond market after fees, so most pension funds should pay far less to managers for buying the best bonds and just have a computer program buy very boring very low-cost bond funds.)]

So yeah, at the end of the day party A sold something and party B bought something, and you have to determine, okay, do we agree on this? And one of the biggest parts of back office is what is called clearing. And effectively, the exchanges — let's say you execute on the exchange, 'cause there are other places you could execute — the exchange will report back to both parties, "Hey, we have this transaction. Now you guys need to go off and communicate with each other that you agreed on this transaction."

And that can happen a bunch of different ways. There are direct connections between different firms. So firm A and firm B might know about each other and have essentially a bilateral connection where they send each other information about the trades, and they want to make sure that everything cleared. Clearing here would mean I know that I bought a million, you know that you sold a million, and the price is correct. That can happen relatively quickly. And then there's also a settlement portion, which is where things actually exchange, in the sense that I give you the money, you give me the securities, and everybody's happy.

Why you don't know who you traded with

Patrick: And one of the joys of finance is there's a whole lot of complexity smuggled into the words, "I give you the money." But to just zoom in on one bit there for non-specialists: at the point where you've bought a million dollars of a particular — you know, Microsoft in your example, or a million shares of Microsoft, which is rather substantially more than a million dollars — and you did it on a particular stock exchange as a venue, you likely don't know who you bought it from, right?

Alex: That is correct. So in the example I was giving before, if I'm an investment bank, when I send out my quote, that will generally have what's called my MPID, which is a high-level identifier for the firm. E.g. Goldman famously had, I think it was GSCO, but then there was GSCO through I think nine, ten, eleven. Sometimes they don't make a lot of sense. E.g. Citigroup's was SBSH, 'cause that was Salomon Brothers Shearson, which became part of the Citigroup conglomerate.

But if you are going through another firm, you don't always necessarily know who the other party is. Some exchanges and some asset classes you also don't know. It'll just be like you execute against somebody, and then later on you might find out, oh, that was with this counterparty, et cetera. Or you are going through, say, a broker or a prime broker, and then it's even more obfuscated because there's three or four hops between you and the actual exchange when things actually get executed.

Hiding a large block trade

Patrick: And this is actually important for some people — that this not quite anonymity as you trade, but something close to anonymity as you propose trades that you are willing to do. 

[Patrick notes: As we’ve previously covered with Ricki Heicklen, there are many different order types available at various trading venues. A limit order which rests on the book is essentially a (binding!) proposal to engage in a trade at a specified price with anyone who later comes prepared to do that trade.] 

Because some traders have a private state that they believe about the world that they want to transact with the rest of the market, and they know transacting with the rest of the market is, one, visible to other people, at least in anonymized fashion via the tape.

[Patrick notes: A long time ago, executed trades were reported to the market via “ticker tape”, which was originally produced by clerks on the floor of exchanges using telegraphic ticker machines, which printed a running paper tape in brokers' offices nationwide. These days there are real time data feeds available from various providers, but the ones which show trade executions are referred to as “the tape.”]

Like, someone can see that a buy or sell happened, but they don't wanna tip their hand as to, like, okay, if I'm selling a million shares of Microsoft, it's quite likely that that is not the only Microsoft I will be selling that day. And so that might suggest to the rest of the market, "Hey, the price of Microsoft is probably going down unless someone really, really wants Microsoft right now."

And this is one of the sort of core problems in trading: doing large block trades in ways such that you get the execution you want at a price that is acceptable without informing the rest of the market that you are doing a large block trade, because that will have what is typically called price impact. The rest of the market will understand the totality of the trade before you put it on and start quoting less favorable numbers to you.

Alex: Yes. And you bring up an excellent point, and I can talk about some different ways that people do it. So let's say you are a firm, and you have some alpha or signal, or you have some state on where you think things are gonna go.

One thing you could do — let's say you wanna go through a broker. You could tell the broker, "Hey, here's this giant order. Work it for me." Now, the broker knows that you think this is the case, and so there are rules to prevent them taking advantage of that, AKA front-running. But if you just put out this giant block, as you mentioned, other people will see it.

So some of the techniques you can do is you can chop up the order over the day, and there's different ways you can do that in terms of what's called smart routing, where you give it to AKA an algorithm, and it'll make lots of little pieces, and it'll send it different places. You can send it to different venues yourself, so you might have five or six different brokers, and you can say, "I'm gonna give ten percent to this broker, twenty percent to this broker."

You might even go to the investment bank and you could say something like, "Hey, I want you to execute this million for me, so it won't show up as me executing it. It'll show up as you." And this gets more into what's called Reg NMS and how orders are routed and fills are sent to clients, et cetera. But the bank can then chop it up themselves. They can give you one big fill, and then they can AKA work the order over the day. So yeah, there's a lot of different kinds of flows through the system to get you the position that you want. Some firms do it over multiple days, et cetera.

[Patrick notes: Reg NMS, and the related best-execution obligation, mean a broker is obligated to transact orders at the best price for the customer available in the U.S., the National Best Bid / Offer (NBBO). This is a non-trivial problem in computer science, because there are dozens of venues stocks trade at that have independent pricing available, and it updates many times a second. Most brokerages cannot sufficiently invest to solve this math problem, but legally they have to, and so they contract with “wholesalers” to do it for them. (Confusingly, this is sometimes called “internalization”, because it descends from the practice of a brokerage matching customer orders against other customer orders prior to sending it to the market. Brokerages basically don’t internalize retail orders any more; the wholesaler will do it for you, better.)

Wholesalers pay brokerages in some cases, because they are technology providers which also do large amounts of trading, like Citadel Securities. Market making retail order flow is a lucrative business to be in, so much so that you would pay for the privilege of giving the retail customer the best price presently available in America.] 

Patrick: And for each of these, there are people with generally opposed incentives, where if you are aware that there are technologies to chop an order up over the course of a working day, then that suggests that at eleven oh two in the morning, you might be able to say, "Wait a minute."

You know, if the naive way to do this — and believe me, people have tried the naive way to do this — if the naive way to do this is just sending in an order every five minutes, and we've seen dozens of orders at this point that are all for the same quantity and the same symbol at the same price, then I naturally assume that I will see more dozens of orders through the rest of the day and can start positioning myself against those orders before they have actually been made.

Alex: Yes, that's correct. Without going into too many of the alpha details, firms look at all kinds of data on just the tape. E.g. they look at just fills coming in. They look at the position of quotes in the order book as it changes over time. They look at alternative data from different data sets. 

I've never seen this data set used where I've been, but there is this discussion around, like, oh, satellite photos of Walmart parking lots is a famous one that people say, "Oh, that's an indicator of revenue prior to the quarterly report coming in," et cetera, right? So yeah, they look at all kinds of different stuff.

Information walls inside financial firms

Patrick: As we've discussed on previous issues of this podcast, one of the reasons why traders are so tight with regards to information flow — down to not telling their friends what they've been reading recently — is to avoid cluing off other people on "the Street" on what their firms might be very intellectually interested in, and therefore suggesting, like, maybe if you dug a little bit deeper into the tape on a particular security or particular set of securities, you might find something that is quite interesting.

Alex: So one point I wanna kinda highlight also is different firms have kind of different opinions on how they structure the information sharing within a firm. So in certain cases, there are literal walls between different parts of the business because either legal or regulatory or what have you, this arm can't know what this arm is doing. And there's a lot of work that goes into making sure that doesn't happen.

Then there are other firms where it's not a legal or regulatory issue, but they still want to keep things separate because they have different teams, and they want to avoid sort of leakage of data for different reasons. And one of my favorite stories — this is at a firm that no longer exists, so I can tell the story — it was so quiet that they wanted to pump in pink noise because they didn't want anybody to inadvertently hear somebody from something else that might have been disclosed that nobody wanted to know about.

Whereas at other places, everybody can see the same source code, and everybody's on the same non-compete strategy, et cetera. So yeah, it's interesting how they kind of play out in different ways.

Patrick: Silicon Valley companies have historically had — with exceptions, Apple being a huge one — they've historically had a huge amount of internal transparency on things. And so if the company is having a great year, the fact that the company is having a great year is quite public internally, and the reason it is having a great year is quite public internally.

But given that finance, like tech, has a constant turnover of people moving between jobs for the usual set of reasons, you don't necessarily want everybody in the company to know that the eight-person group that is killing it right now are, I don't know, mortgage derivative traders. Because even if those eight don't leave, that could cause the thing that is causing them to be so lucrative to leave the building to a competitor, or suggest to another competitor, "Wait, we don't have the group analogous to that. Maybe we should, if it is printing like billions of dollars a year of excess profit."

And indeed, there was some litigation about this recently, with the apparent size of the information that you need to disclose to get a company to be very annoyed about the fact of the disclosure and sue you was, "Hey, have you heard of Indian options? They are really lucrative." (Covered by Matt Levine on several occasions.)

What the middle office does: managing positions

Patrick: Since we're here about the back office and people are thinking of, you know, brokerages that are doing millions upon millions of trades every day, I think they probably assume that it's like one person like yourself and then a million computers. And in fact, most firms have more than one back office professional working for them. Why?

Alex: Sure. So I'll talk a little bit about the middle office as well here. So one of the things that you want to do correctly from a middle/back office perspective is manage your positions. And that might sound pretty simple. Most people have some kind of retail trading account, and they think, "Oh, I look at my account every day. I have ten or twelve stocks in there, and I buy them maybe every week or every day if you're doing day trading."

Firms at large scales, e.g. banks or larger hedge funds, are trading globally in potentially hundreds of different markets or venues. The products are different in many different ways. Even things that you might think are the same can be different. So an example might be, let's take, like, UBS was a Swiss bank. They have a US entity, they have a Swiss entity, they might have an Asian entity. You might also be dealing with different entities in the same country, e.g. there is an investment banking arm, there is a retail trading arm, you might be dealing with both.

So I'm giving a taste of like three or four of the hundreds of dimensions that can exist, and that's why you hear people talk sometimes about security master or reference data, et cetera.

Patrick: And if I can give one more on it. You know, you might think it's fairly straightforward to know how much Microsoft I own at any given time, which is not quite true. But when you're an investment bank, for example, working with many sophisticated clients, there are very many ways to own Microsoft in the world, and there are very many ways to owe Microsoft in the world.

And so for example, you might have a wealth management division that — because relatively wealthy retail investors like Microsoft as a brand name, you might have sold them some income products which have reference to the price of Microsoft securities. And so you are synthetically exposed to Microsoft, which matters when you are synthetically exposed to the tune of like hundreds of millions of dollars.

[Patrick notes: One of the infinite variety of products here is a structured note. The investment bank might solicit its wealth management customers—who are well off but are in the main closer to dentists than financial professionals—with a pitch like “Like Microsoft but don’t want to lose money if their share price goes down? Then lend us money. We’ll pay you 12% interest a year, much more than you would get from the bond market. The only way you lose money is if Microsoft goes down more than 50%, in which case you start losing money as shown on this chart.” (In the commonly sold variant, 50% is the “knock-in” price: down 49% and you lose nothing, down 51% and you lose 51%.)

This is economically similar to asking the customer to write you a put on Microsoft, and might indeed be motivated specifically by someone having bought a large number of puts from the investment bank.

So the bank has an exposure, and the structured products desk’s job is to take that exposure the bank would like to be out of, package it in a way that some customers would find appealing, and earn a bit of fees along the way. How does it earn fees? Because if someone is capable of pricing this product correctly, they work on a structured products desk, not in dentistry. Which isn’t intrinsically improper: every time you buy a burger you’re paying someone to do the thing you didn’t want to do.]

And so you need to know, okay, if the Microsoft share price goes up by 10% because, I don't know, Bill Gates comes back to work or something, what does that do to essentially my exposure right now? Like, I'm probably long in some ways and short in other ways, and minimally, am I net long or net short is a great idea of something to know. But to what degree, and to what degree does that change even if I do nothing to change it? The joys of option math here are always fun.

[Patrick notes: Non-specialists might think that one’s exposures largely change in response to trades one makes. If you owned 1,000 shares of Google, and you did nothing, you’d assume that tomorrow you would also own 1,000 shares of Google. But there are many more complicated ways to be exposed to Google equity. “The Greeks” (alpha, beta, delta, theta, gamma, and so on) are shorthand which options traders (and Risk, etc etc) use to express how various changes in the state of the world, including time simply advancing, would impact one’s exposures.]  

Calendars, time zones, and other sources of gibbering madness

Alex: That's an excellent point. And I actually worked — one system I worked on was a swaps database where essentially you go to another party and you say, "Hey, you will own the asset," and there's some agreement in terms of where future interest rates or revenue will come from. That's its own rabbit hole. But the database that managed the relationship of the prices, or the rates that would get paid by both parties, was not just a time database, it was like a time-of-time database. So someone might come in and say, "Hey, remember the deal was from January to August, and now we're in November." Oh, we have to go back and update that and then rerun everything again to make sure that the prices look good and are correct.

Trying to think of another example. Even just exchanging calendars. So I think people just assume, oh, every exchange in the US is largely the same. That is generally true, but trading hours can be different at US exchanges. Different countries have different holiday calendars. Some countries are more religious, some are less religious, so religious events can impact different calendars.

I worked on one system where you could pick the calendar of the thing that you're trying to do, and it was in alphabetical order, and the first one on the list was the Beirut Stock Exchange, and I always wondered, "What is going on at the Beirut Stock Exchange that we have to have this calendar?" I never found out, but there are a lot of different knobs that can be pulled and turned in that sense.

Patrick: I lived in Japan for 20 years and will probably be back there in the future, and while Japan uses the Western calendar in many, many places for a variety of reasons, there are many computer systems and, you know, APIs talking to each other which expect you to express historical dates, the current date, or even potentially future dates in the Imperial calendar, which uses the reign name of the current emperor plus number of years since that has been incepted.

And so when there is a new emperor announced, you get a bit of time in Japan for everyone to do a crash update to all the computer systems to be able to express the right range of dates in terms of the new reign name. And, you know, boundary conditions aren't fun to be able to program for.

Alex: Even time is a good example. I've worked at some firms where daylight savings, not a big deal. Like, all the systems just kind of click over. Everything works correctly. Other firms, it was a disaster every — 'cause, oh, this started, we didn't account for this. This had to start before this other thing.

Patrick: It was always a disaster for me when I was running a company that was quite calendar-oriented. And there's a document that was written by a very senior engineer who worked on Google Calendar who was trying to explain all the intricacies at the appropriate level of documentation for Google Calendar, and it was clearly driving him to gibbering madness.

If you have a meeting that is in two time zones at once, and then the time of the meeting changes, it is unsolvable in the general case how that meeting should change in both time zones. It cannot be done. You have to make assumptions about geopolitics, among other things. And so calendars drive people to gibbering madness all the time, and yet we still have to be able to make some sense of this every day.

I think one of the reasons why more of this is manual than you would expect is that historically there's been some amount of slack for something unexpected came up. You know, a counterparty of ours had a bad day, et cetera, et cetera, et cetera. And now people who have a great deal of expertise on what should happen on a given day need to, in a very short amount of time, come up with something to adjust for a hiccup. Do you have any fun war stories there? I've got a couple I can relate myself.

War story: twenty million shares out the door

Alex: Yes. So there's one — it was a Friday afternoon, and there was a particular desk, and they're trading a particular product. Relatively volatile, big product. And it's four o'clock, and we think we're done for the day because it's Friday at four PM. The trades are done. It might have even actually been after four fifteen, because some things happened at four fifteen.

And one of the traders runs over and just, like, is yelling at us: "Product name, product name, product name." And we're like, "Whoa, whoa, whoa, whoa. What's going on? What's going on?" And she just kind of runs off, and we're like, "What is happening?"

And so we go in and we look at the order flow and execution flow, and effectively what the client had said was — going back to the other part of the conversation — "I'm gonna send you twenty million shares of this product. I want you to execute five of these, in quotes, manually," which means it'll just be one big execution that the firm would take the position on, "and then the other fifteen you can just send out to the street, and whatever price you get at the street is fine with me." That was the plan.

The system was designed to have a single trader working a single order from the customer. What the head trader had decided to do was say, "Hey, junior trader, you do the street orders, I will do the manual fill." So he has the window open and he's working on it, and he hits execute assuming that five million will execute, and then the junior trader will basically send another 15 million to the street. Because she had already started sending orders out, the manual order window kicked off an error saying, "Hey, when you opened the window there was 20, now there's 12, rejected," and she just assumed that it would just all go out. And so 20 million shares went out to the street rather than at the price we agreed on with the customer.

So now the market is closed, and we have this giant position potentially that could move over the weekend, and we end up having to talk to the desk, talk to risk, come in on the weekend, put in a futures order to hedge essentially, not all of it, but a large portion of it. And that also became a technology error, and one of my other jobs at that particular firm was a lawyer for the defense in defending technology from errors claimed by the traders, which is its own conversation if you want to get into as well.

But yeah, it tends to be a lot goes wrong very quickly, and if you can solve it before the cutoff date — of which there are many, and some of them are in the morning, some of them in the evening, some of them are late at night, or they're overnight if you're dealing with Asia — and if you can fix it, great. If you can't, then the cleanup process can take longer, you have to call people.

One of my other favorite examples, this is at a different firm. We did not get our positions out to the broker-dealer — or sorry, the prime broker — and the prime broker was a large bank, and we were talking to someone there, and they said, "Oh, sorry, we have to wait for Budapest to run the end-of-day job." And I remember sending a meme to the head of clearing of James Bond smoking, being like, "We have to wait for Budapest dollar," kind of thing.So yeah, it's some interesting times.

Ops as an iterated game

Patrick: One of the things I'd like to highlight here is a thing that ops does all day, every day is talk to ops at other institutions, and this is an iterated game. It may feel like a crisis in the moment. It sometimes is a crisis in the moment, but mostly you expect to be there tomorrow too, talking to the same people tomorrow. 

[Patrick notes: Frequently other people involved in the transaction believe it to be a crisis. For example, if you accidentally wire the wrong person $10 million, you might think “Oh no, wires are not reversible! I just lost $10 million.” Ops is largely not worried about this. Ops is going to call their counterparts, say “Yep, misdirected wire”, their counterparts will say “OK great, sending you our standard hold harmless agreement”, Ops will sign that, and then the money gets sent back despite wires being “irreversible.”

And yes, I know, the Revlon loan repayment, but Ops also knows about the Revlon loan repayment, and it resulted in manuals and agreements getting updated across the industry.]

And there's some amount of sort of usage of relationships to say, "Okay, in the case where we've made a commitment to a client and we didn't do the thing that we said we would, potentially there's a way to call the client back and say, 'Whoops. Okay, so on one level, you might have a contractually enforceable right to blah, blah, blah. On the other level, we have a relationship that we both mutually value. Can we go halfsies on this? You know, you absorb half the risk, we absorb half the risk, yadda, yadda.'" And that obviously is not something that you want to do all that often, but it's an expedient resolution in a lot of ways.

This sometimes surprises people who are not professionally involved in this that, you know, in a culture which is often derided as being very zero-sum and merciless, et cetera, et cetera — that the culture of the trading floor was what it was. There is a formal resolution process for determining, like, if two people disagree on reality as to whether a trade happened, did that trade actually happen? And if you engage in that formal resolution process, you've done something wrong.

The first thing to do is to say, "Okay, I thought I sold at seven. You thought you bought at eight. Price is seven fifty," and we shake on it. And people think, "Well, isn't that abusable?" But these are high-trust environments. You are not going to burn your professional reputation over one transaction on one day, typically, typically. And so the equilibrium kind of holds.

And then as we sometimes see in Silicon Valley, the exception case is like when an iterated game suddenly becomes like a one-shot game, as, you know, dramatized in Margin Call or similar, where it's like, okay, this is an industry-ending apocalypse. Okay, that's an overstatement for what 2008 did. But it was certainly a firm-ending mini-apocalypse for at least some firms, at the degree of, like, we are willing to do something which is acutely against the interests of the people that we are talking to on the phone goes far up versus like the typical day.

Alex: There are many things to say about that. One quick sidebar: at the end of Margin Call, you see multiple trading floors, and they'll say like Deutsche, Citi, et cetera. Those are actually all the same trading floor, and it's the trading floor I worked on. And if you watch the movie, there's a large trophy in one of the scenes, which I think was either the flag football or dodgeball trophy. So yeah, I have a soft spot in my heart for Margin Call.

Busting trades and who has the authority

Alex: So I'll give an example. In general, if you are trading and you what's called fat finger an order — so you buy maybe ten thousand when you meant to buy a thousand and you move the market — you can speak to the exchange, and the exchange will say, "Okay, we are willing to... if you wanna file what's called a clearly erroneous, we will bust that trade," AKA we will pretend like that trade never happened. That's sort of the best case.

A slightly less good case is sort of what you were describing, where they will say, "Okay, well, we're not gonna bust all thousand shares. We'll only bust three hundred or four hundred."

And kind of a very famous example — 'cause I ended up going to work there — was Knight Trading. 

[Patrick notes: Knight Trading, a prop shop (firm which invested its own money), infamously blew up as a result of technical issues. If you’re not familiar with the story, the SEC has a fairly good look at it in the context of an administrative order.]

I was at a firm that Knight Trading did business with, and they called our back office team and were like, "Hey, will you bust a trade with us?" And some of them, they were willing to bust, and some of them they weren't. And famously, some of the banks did not bust any trades, and so they were able to shrink their error a little bit but not a huge amount. But your point around the many cycles of the game — you want to have your customers come back to you, and if you consistently make errors and try to make the customer eat it, then it's gonna be a problem.

Patrick: It's sometimes unclear even with formal rules, et cetera, as to who has authority on things. And in the moment a crisis happens, like who has the authority to countermand an order?

There's a famous example from Japan where there was a particular security which — the correct price for the security is approximately six hundred thousand Japanese yen, which is a few thousand dollars. And for various reasons, like, the anchor point people have for the price of a share of stock is usually, oh, it's probably somewhere in the tens of dollars, hundreds of dollars these days. But this particular one, it's about $6,000. And a broker intends to sell one share at $6,000. Instead, they sell 600,000 shares at one yen, which is a fraction of a cent.

And this order gets to the stock exchange, and people at the stock exchange — and the regulator was furious about this in the aftermath — people at the stock exchange looked at the order and were like, "Oh, they obviously swapped these two numbers. But I personally don't think I can countermand that bank, and, well, I'll just let it go." And so everybody let it go, and the firm ended up taking a, I believe, several hundred million dollar loss on this because obviously, you know, the price of this thing falls to near zero in a way that is unexpected and incorrect. And indeed, there was an American firm that is well known for having sharp elbows was reported as having successfully bought a lot of the stock at, you know, a price not too dissimilar from one yen, and then had a great day for itself.

[Patrick notes: Mizuho Securities was the brokerage and J-Com was the stock. You can read the court’s decision in a subsequent lawsuit against the Tokyo Stock Exchange for a recital of the refusal by TSE supervisors to countermand Mizuho’s order despite it being obviously incorrect (TSE had actual knowledge they were attempting to transact more of the stock than existed).

Morgan Stanley was, rather briefly, 31% owner of J-Com, as was publicly reported. It took the position that someone wanted to sell a Japanese equity at a very attractive price, and at that price Morgan Stanley was a buyer.

A correction on my contemporaneous recall: the stock did not actually transact near one yen.

In keeping with recurring themes in this episode, while Mizuho did suffer about 40 billion yen in losses, these were shared partly out of the legal process (TSE paid about 10 billion) and partly out of social suasion by securities regulators in Japan, who suggested that Tokyo has a vibrant international financial market and a very long memory. Some of the profits were donated to the usual cast of inoffensive charities.]

Patrick continues: But the markets kind of, like, have to keep turning. The world has to keep spinning despite one firm having a really bad day. And this is a relatively successful property of them that I think people don't appreciate enough. Like, when Knight Capital blew up one day, there was a — it's a long story for people who haven't heard it, but a firm, due to technical issues, suddenly became acutely institutionally unaware of what its positions were and started firing orders off willy-nilly and essentially spending money it did not have.

And it probably surprises people that you can spend money that you don't have, but that is indeed a key function of financial markets. And they spent much more money than they would ever possibly have. It's really bad news if you work there. It's bad news if you were equity partners in Knight Capital. But everybody else was basically fine by the end of, plus or minus, the next day. Like, "Okay, they blew up. We all have risk functions, so we were only exposed to this to the extent that we expected to be exposed. Some of us got windfalls because they executed trades at prices that were very bad for them, and all right, we're fine."

And, you know, socially speaking, if in 2008 a large investment bank that has a million lines of business goes under, you're in a really rough way because some of those lines of business are extremely socially beneficial and difficult to substitute for — and that's where “Too Big To Fail” comes from. 

[Patrick notes: TBTF is the much derided notion that it is effectively policy of the United States government that certain financial firms are so systemically important that, should they fail catastrophically, the government will swoop in to rescue the firm. The ethics and politics of TBTF are a long story. The pragmatics are a short one: it is impossible to sensibly unwind a globally significant financial institution in a day, and many tens of millions of people (mostly without accounts at it) would experience severe disruption to their life if it went down. And therefore, we don’t allow them to go down.]

But if a trading firm goes under, like, Knight Capital is a pile of capital and some smart people and computer programs, and money, smart people, and computer programs can be reconstituted under a new name and a new LLC. We're fine. The markets are structured by dueling robots, and if one dies, eh, we'll replace it.

Inside the Knight Capital collapse

Alex: So I actually worked at a different firm and had resigned on the Monday before Knight Capital blew up because I was actually going to Knight Capital. So I was also kinda interested to live through that as like, "Oh my God, what is going on here?"

And I think, if people have — there is an SEC report where they go through all the steps. [Patrick notes: linked above but for your convenience here as well.] That was a very accurate accounting of the things that went wrong. And sort of two things. One you already highlighted and one I'll add. One of the items in that review was who actually had the power to say, "No, we are going to stop trading," versus, "Oh no, we need to stay up and trading." And that is because some trading firms actually have agreements with their customers that they have to be up a certain amount and/or they are the exclusive venue for that customer. So that in a sense worked in Knight's favor in the aftermath, that some people could really only trade with them or primarily trade with them. So there was an incentive for them to get injections of funds to basically come back and keep going.

So yeah, on one level it's very simple. It's I put up stuff to buy or sell, people buy or sell it from me. But then all the different agreements, and especially for the OTC stuff, like sometimes there isn't a central exchange, so it is all basically agreements back and forth. Yeah, it's a giant web of interconnectedness which can move around in interesting ways.

Agreements, obligations, and who you have to call

Patrick: I like that you brought up this concept of having agreements because, you know, there are piles and piles of spreadsheets and computer code created for this. But there are also piles and piles of either handshakes between people, recorded handshakes between people, or very long, exhaustively drafted legal documents between very well-capitalized firms.

And what have we agreed to, and are we in compliance with those agreements today? And if we are not in compliance, you know, what is the resolution path, at what point do we need to tell someone and call them and say, "Hey, between 11:58 AM and 1:32 PM today, there was a period in which the thing that we had contractually committed to do, we did not do. And, you know, how do we work through this?" Are all things that basically falls on operations and the back office — to be aware of this constellation of things that governs our behavior and also just like do the mechanics of making sure that it happens every day.

Reconciliation, FTP files, and the humans behind them

Alex: Yeah, so you hit on the part where you mentioned, "Oh, how do we make sure that between the two companies we kind of agree that this is what had happened?" So I think for the casual listener, much like people balance their checkbook, firms balance their checkbooks against the counterparty's checkbooks. And I can talk a little bit about how that happens in practice from both a technology perspective, then also some of the systems that are built to help kind of reconcile that.

So at one firm I owned — it was creatively called, I think, external data, and was basically the system that would FTP to counterparties and say, "Hey, give me the file that has all the trades that you think we did with you, and then we will reconcile that against what we have internally." And people's first thought might be like, "It's FTPing CSV files around. How hard could it be? There's not that many issues with it."

There are many, many issues. One is just even checking what file is there. So for example, some firms will drop one file for every business unit in the counterparty. So whether you have 1 or 15, you get one file, and then somehow in there you have to kind of figure out, okay, this is for this department, this is for that department.

Some firms will create the file, and as soon as you download it, they delete it. And so if you lose that file or it gets corrupted, you have to call somebody in Budapest, and they have to remake the file. Some people never delete the file, and so you have to check every single file in the FTP server to make sure, like, hey, is this a file that we've already downloaded?

Some firms — one of my favorite examples is one of the banks. So if you traded one asset class, you got an actual XLSX Excel spreadsheet file with one sheet, which was just the trades you did. But as soon as you went to two asset classes, they would add an extra sheet at the beginning that was a summary for each of the additional asset classes. So even if you had a script that was saying, "Oh, for the bank, let me pull down this file," you had to know, wait, is there a sheet? Is there not a sheet?

And there were teams of people — to your earlier question — whose only jobs were to basically every day go through and be like, "Did it come through correctly? Did it not come through correctly? Why not? Was it our issue? Was it their issue?" Et cetera.

Deadlines, cutoffs, and the $36 million phone call

Patrick: Yeah. And a very important sort of recurring pattern that we have here is that there's a process that is known to fire every day. There is a time where it is like, we usually expect it around X, and then there might be a little bit of, we get worried if we don't see it by Y, and then after Z it is capital-L Late. And those have different levels of consequences associated with them.

In credit card processing, which I will always have a soft spot in my heart for, there's a similar thing where you might do net settlement with a bank, for example.

[Patrick notes: I previously worked at and am still an advisor to Stripe, which does not endorse what I say in my personal spaces. Also, apologies to former colleagues: I know this following explanation is handwavy.]

The bank has many customers that have credit cards. They go out into the world and spend on those credit cards every day. There is some amount of them also having merchants with credit card processing accounts who are having payments coming in every day. And so somebody else in the chain might think, "Okay, with respect to a particular somewhat small bank, you owe us $36 million on net today. Do you agree with that number? And communicate that by," I'm going to make up a time, "5:15 Eastern Time every day. And you can get it as late as 5:30, but after that it is capital-L Late." [Patrick notes: I adjusted these from what I said in the episode to be more plausible.]

These times are illustrative, and anchored to the Fedwire daily cutoff. Fedwire is the easiest way to transfer $36 million to another financial institution. [Patrick notes: Many settlement processes in the U.S. do go over ACH.]

And, you know, let's say at a relatively small bank there is an employee who owns this process — we'll call her Martha — and Martha's on vacation. That doesn't absolve the bank of the acute necessity of making sure that file comes through every day. But it might have happened once or twice in capitalism that Martha was indeed on vacation and that the bank had underperformed its responsibilities for having a backup in place.

And then what happens? So sometime between 5:15 and 5:30, a series of increasingly urgent phone calls starts happening, which starts somewhere in the bowels of operations and then goes as far up as it needs to to say, like, "Hey, just checking. We really think that you guys owe us thirty-six million dollars. One, do you agree on thirty-six million dollars? Two, is the wire going to happen?" Because if the answer is ultimately, "No, the wire isn't happening today," then someone is out thirty-six million dollars.

And you don't really have the option of, okay, tell all the sandwich shops and gas stations and power companies and similar that they're not getting their money on time. Like, someone is going to take minimally — for a point of time, they're going to take thirty-six million dollars of risk. And that someone needs to be told that as early as possible.

If this happens on, like, an ordinary Monday, well, everyone has had a bad Monday before, it happens. If it happens during a time of, like, market stress or widespread panic about the economy or something — you know, in the case where, like, a bank, "Whoopsie, we owed another financial institution thirty-six million dollars on Monday," but Tuesday rolls around, we're still a bank, we're still good for it. 

In the case where that happens on a Friday and there is a great deal of worry about how many banks the United States will have on Monday — where that number doesn't ordinarily change that much over the weekend — the temperature level in this conversation goes way the heck up.

Alex: Yeah, there are kinda many threads here, and I'll kinda zip through some of them. One, we got a new head of clearing in a past firm I worked at, and someone asked him — and I think he had come from a large bank that did prime brokerage — someone asked him, "What is your nightmare scenario now that you're on this side?"

And he goes, "We trade through a broker. At the end of the day, the broker sends us," very similar to how I was describing before, "we get a list of the trades, and we disagree with the broker on what we think our total number of trades are, AKA what our current position is. So we might say, 'Hey, we are long security X to the tune of a million shares. You think we're long seven hundred thousand. We should talk about this.'"

We talk about it. It turns out at the end of the day, at the end of the whole process, that we are correct and the prime broker is wrong. The prime broker, as part of our agreement, might say, "Well, I'm not gonna let you trade out of this because we have to figure out what the actual position is." To your point, suddenly product X either — if we're long, drops a lot, and it might be so much, either if like a bankruptcy or something happens to the CEO, or some giant lawsuit comes out, et cetera — that could effectively kill our firm. And it happens. This is a thing that has happened more than once.

It's sort of like on the other side, the prime broker will do this to different sides' parties, and it could just be an error on their side. It could be a technical error, procedural error, fat finger error, et cetera. So it can get scary and very high stress from the procedural side.

How two banks fail to agree on fourth-grade math

Patrick: Let's ask a question on behalf of the civilians in the audience who are just, like, gawking in amazement right now. How can two banks fail to agree on fourth-grade math — even fourth-grade math over a large number of transactions?

Alex: Sure. Many, many ways. So one is just straight-up technical errors. So I guess we'll move into kind of the war story part.

I was at one firm where there was an algorithmic trading system. So people would put in some math, and they'd say, "If these numbers are above this value or below this amount, send these types of orders." There was a bug in the code that flipped the symbol on the executions. So if it was an execution for Microsoft, it got flipped to Apple. Like, I'm using explicit examples, but at the time this was happening, we had no idea what the key was of which ones had gotten flipped where. So you might send position reports back out to the other bank, and you thought you were long a million, but you were actually short five hundred thousand because everything got flipped.

And at the time, I think the workaround was we basically pulled all the executions, got a unique list of symbols, went to Yahoo Finance, figured out the day's trading range, and essentially had to do a join on the fly of, "Okay, this trade is for forty-seven dollars, but Apple didn't trade below ninety today, so this one's probably wrong."

[Patrick notes: Quite a bit of ops is unglamorous work done by smart people. When you describe this series of events to regulators, other smart people will default to saying “The incident team promptly remediated by falling back to a secondary provider of securities prices.” And you’ll say Yahoo Finance verbally at the meeting, and the other side will smile, because it isn’t their first meeting, either.] 

There are a lot of hops in the system. I think most people think like, "Oh, those trades come in, and they immediately get booked." There could be, you know — there's the exchange, then there's a gateway between the firm and the exchange, then there's the crossing engine, then there's the trading system, then there's the sales trading system, then there's the booking system, then there's the clearing system. Like, any one of these things can break it in any particular way.

And one thing I also wanna highlight: these don't all necessarily get resolved, like, the day of, the next day, or even potentially weeks later. One firm I worked at, they had a dedicated system for basically tracking trade breaks where the two parties didn't agree. And I think the longest trade break in there was four or five years 'cause a firm went out of business, or the product's not traded anymore, or there's a lawsuit, and nobody can trade out of it. And so you're carrying that position/risk for years, potentially. So this can happen in milliseconds or it can happen in years, respectively. So it's a pretty big time range.

Patrick: "Why don't the accountants just catch it?" is something that sometimes gets brought up to people. And sometimes the accountants do catch it, but accounting typically happens on a much more long-term basis than the day-to-day work of operations.

A later audit might discover, say, an infelicity with respect to various low frequency pathways of what can go wrong in a credit card transaction. Suppose, due to a curious implementation decision, 0.1% of all transactions which were transacted in Mexico failed to get money from counterparties where we expected to get money from counterparties, and we discovered that six months later.

Now multiply transaction volume for the nation of Mexico times 0.1% times six months, suddenly becomes like a pretty big number. And then your resolution options at that point can be somewhat limited because, you know, okay, making up a hypothetical number, do you want to call 8 million people in Mexico and ask them to please send a check in for an average of $20? Probably not that viable.

I'm filing the serial numbers off of real incidents that have happened for obvious reasons.

Okay, we have the case of technical errors. I also think that a lot of finance runs on trust, particularly in certain relationships, and there are reasons to say, "Okay, I think this particular firm is — you know, this is not a fly-by-night startup. These folks have an excellent reputation. They are not particularly thinly capitalized. I'm going to just trust that their numbers are right for maybe twelve hours more than I would if they were one-thousandth of the size that they are." And they make mistakes at some rate. And so now my understanding of reality is twelve hours behind, and twelve hours can be a very long time some days.

When the golden record goes down

Alex: Yeah, that's definitely even true internally at firms. So I was at one firm where there was a centralized position management server that was the official golden record of what are all the positions in the firm. And that had an issue, and what ended up happening — so I was the incident commander for a large portion of that. It was overnight. It took like twenty-seven hours. People operating on very little sleep.

And then the next day, it basically came to light that people knew potentially that that position management server could go down, so they would keep their own internal positions as a double check basically on the system. So when they would fire up in the morning, they would reconcile between each other.

And if you hear me say the word reconcile a lot, most of modern finance is effectively always reconciling between all the different nodes in the network all the time. Because you let one — to your point earlier, you let some of these trades, e.g. like a currency exchange trade, might seem small or just one fill, but it could be potentially hundreds of millions of dollars. So you let one through, and it's a gigantic deal also.

Patrick: So what's the downside, if anything, if people are maintaining their own books and they match, like, the system-wide view of risk most of the time, and then system-wide view of risk goes down one day? Dot to dot, what happens next?

Alex: Yeah, that's a great question. So even by — I don't wanna say by design, I guess by historical artifact — different firms have very different position management systems. So for example, I believe it was Goldman was very famous during the 2008 crisis for they had one central monolithic database of all of their positions. So if somebody wanted to say, "Hey, how are we exposed to XYZ?" it was a couple keystrokes to essentially say, "Oh, well, we are long on the one side, maybe on the equities, but we're short on the options, and those net out, so we're flat."

Other firms that were conglomerates or were essentially multiple firms that had been acquired over time had many disparate systems. So their cash position management might be different than their options, their equities might be different than their options, it's different than their treasuries, et cetera. So it could be very difficult to get a centralized picture.

In the world where you do have a centralized thing and then you have sort of satellites, when you come back up, you don't wanna just say, "Hey, let's go with whatever one is up," or, "That's the main one." Again, you have to go back and you reconcile, and in a perfect world, you had thought ahead about this as part of your outage management planning and disaster recovery, where there's a checklist or a book that's like, "Hey, if the big position management system goes down, do these things." There's that, and there's tools to do that. If you're unlucky, there's folks like myself who are writing Perl and Python scripts on the fly to basically make sure that, hey, can we connect everything? Okay, it's in a different format. Okay, we didn't think about that format, et cetera. So yeah, the more organized you are there, the better.

Finding the one person who can sell the position

Patrick: And one of the things that was widely reported in 2008 was we suddenly went from, you know, the — for lack of a better word — peacetime footing to crisis-time footing in a variety of sub-sectors of the financial industry. Because Goldman knew what Goldman's net position was, when things start to hit the fan, they were able to quickly orient around the new reality, close out positions and redeploy money. “It's a really bad time to be in finance right now, maybe we should open some shorts.”

Whereas other people were like, "Okay, I'm probably very exposed here. Can you get me an answer by the close of business tomorrow on am I right about that?" Or, you know, even things as simple as, "Okay, I think on net we're very long something, and we need to sell out of that. But I'm not sure who here is long that something."

And it's not, you know, at the scales of an investment bank, it's not E-Trade. I can't simply click like, "Yeah, sell all of that, please." Like, I have to find someone in this office that I can call that has, like, legal title to the thing that my screen says that we have, and can, like, physically effect the sale of it to, you know, decrease the firm's overall exposure to this and free up cash and yada, yada, yada. And even identifying that person is non-trivial when there are 100,000 people who work for you.

[Patrick notes: Legal title is important! You very much cannot sell the customer’s assets which they have in custody with you to satisfy your need for cash in a crisis. Very much not allowed! However! In a crisis, you will be making many risk-based decisions, and one of those will be urgent reevaluation of how much credit risk you’re comfortable with taking. And so you might blow some of your hedge fund clients out of their trades, which will—mechanically—result in a) selling the stock they own and b) shoring up your capital position. But this is entirely different.]  

Alex: That is definitely — yes, 100%. And like the example I was thinking of when you were discussing this is, sometimes you'll see a photo of — it's effectively the system, the number of computers you would need to be in compliance inside the government for sending secure messaging, and it's like eight desktop computers because there's a Defense Department computer, there's the Secretary of — sorry, there's the State Department computer.

It's very similar at some of these firms where your position management view for equities is on one system, but then there's a completely different system for treasuries, a completely different system for options, et cetera. Sometimes they talk to each other, sometimes they don't.

How much business do we do with Fidelity?

Alex: And then even — I'll give another example. Going back to earlier in the conversation about the different entities, the head of trading one time asked, "Hey, how much business do we do with..." I'll use the real company because everybody does business with them. It's Fidelity. And you would think, "Oh, how hard could it be? Just go on this database: select * where customer = 'Fidelity'" And then you realize there are different entities of Fidelity, then there are different companies that have Fidelity in their name, like First Fidelity Bank of Random Town, USA. And these things grow over time organically.

And there, I have a real-world example of a technical version of this where, in a lot of these places, there's a point in time where someone should say, "Hey, it's getting out of control. We should re-architect this to make it simpler." If you don't do that, it gets so big that you just can't do that. Like mainframes at banks is a good example. They've been there forever, nobody knows exactly how they work. It would cost millions and millions of dollars and five years to get off of it, and nobody wants to make that giant decision 'cause it's unclear if it's gonna work, et cetera, et cetera. So yeah, a lot of tech debt effectively.

Patrick: I like the example of how much business do we do with Fidelity, because something that might not be in people's model here is, regardless of how trusted an institution is, there is maybe one institution that is exempt here, which is the United States federal government, because they can print dollars at will. But for everybody else, there is some level of business that you're willing to do with them and no higher. Because if they have, like, the worst day possible and their firm goes down, we can't go down with them.

It's extremely complicated at the level of, like, globally significant financial institutions — like, you know, just to pick two names, not to pick on two firms, but it would take an extensive science project involving thousands of people probably, the better part of a few years, to like completely figure out how many ways that Bank of America and Chase are tied at the hip. 

But there has to be an answer in some places. 

And, you know, there has to be an answer for, okay, if you are a credit card processor, how much business do you do with Visa every day? And that's a weird one. But you can imagine, like, vice versa, you know, Visa, like, okay, there are many people on the Visa network and each of them have like some limit that we are comfortably exposed to — even though I'm hand-waving away a lot of complexity here. [Patrick notes: For one, credit card processors are mostly not transacting with Visa… except where they are. (You’ll largely be transacting with someone else in the chain most of the time but Visa when doing e.g. international settlement, which the credit card networks make a huge amount of money on.)]

Prime brokerage and extending credit

Alex: Yeah, I can use a real example. So I worked at a prime broker, and that comes in — so one of the functions of a prime broker is you might have a lot of small customers, or customers at different... They could be large customers, but they just don't do a lot of flow, particularly in, let's say, like crypto.

So what you do as a prime broker is you go to an exchange and you say, "Hey, exchange, I'm gonna have a session that's tied to me as a prime broker, and then I will be acting as a middleman for a lot of other customers that will flow through me." So A, I get the volume discount of all these customers. I can give them a discount on basically going directly to the exchange. And, you know, you might be dealing with very small, unsophisticated market participants that are just a couple guys in a conference room somewhere in Florida, let's say, and they're like a little day trading shop. Or you might be dealing with different parts of a giant bank or corporation, but there's a US entity and a European entity, et cetera.

And so it can be tricky to sometimes figure out, A, how much volume, and to your point, even who to call. Like, do we call the Euro guy? Do we call the US guy? These are all real questions that pop up, particularly if there's an outage and things are breaking and you need to call somebody — that person might not be at the firm anymore. So yeah, to your point, you just assume it's all accounting, but then you realize there's a lot of fuzziness, and you have to get to the right person and get them on the phone, and they have to know how to handle it, et cetera.

Patrick: And one of the functions of prime brokers is extending credit to people. Lots of trades happen on credit for a variety of reasons. One, it magnifies returns. Another reason is that basically credit functions in many places as sort of like lubricant in the economy, where if we were doing everything cash and carry, if there was no credit extended between firms, there just simply wouldn’t be enough money to be at the various places. That would generally have the effect of raising the cost of capital to various firms, slow things down generally, et cetera, et cetera.

It is broadly underappreciated by people that even if you look at your, like, retail checking account statement, the number that you believe is your balance is not necessarily your balance. That is the roll-up of a number of credit decisions that your bank has made on your behalf. And there are other people that are making credit decisions vis-à-vis your bank, et cetera, et cetera. And so your quote-unquote settled balance is a probabilistic statement about what we believe about millions of transactions at this exact moment in time.

[Patrick notes: “Checking accounts are credit products” is one of the most important facts about checking accounts, and widely disbelieved by many people. This is understandable with respect to the typical checking account holder, and annoying when it is e.g. legislators or bank regulators.

If you want a concrete example of the divergence between the common model and, well, reality, take a look at Regulation CC funds availability timelines, where there is (due to settlement minutiae) substantial ambiguity as to whether a check or other item posted to your account is good. Banks historically made risk-based decisions, underwriting you and your counterparty, to decide when to release funds and how much. Regulation CC is a patch on that historical practice to mandate that one must release on a particular schedule, with a short list of enumerated exceptions, even where there is ambiguity as to whether the funds are good.]

At some point, somebody at a prime broker has to say, "You know, a couple of years ago when we got into this relationship, we were happy to have them. And for every day since, we've been happy to have them. I'm not sure I'm happy to have them tomorrow." Or, like, previously, you know, we had extended this particular client a credit line of up to — make up a number — three billion dollars. And maybe they're still good for two billion of that, but I don't really want to lose the last billion. Maybe we have to have a communication with them, like, "Okay, well, if you own three billion dollars of assets, maybe own less very quickly."

There is often an “or-else” implied in that sort of communication.

Alex: Yeah. Well, sometimes there's just giant events. So for example, in 2008, like, Lehman just disappeared one day, or Bear Stearns just disappeared one day, and you had all of these interactions, like you were mentioning before, about being tied at the hip. You don't even know who to call. Like, you call the number and no one answers, right? They've shut the office down, and now you have these giant positions you have to deal with.

I'm not as familiar with some of the lending, like the pure prime brokerage around, okay, this person is — now we don't think of them as a good customer. But there are definitely amounts of order flow that you'll take from certain people. Sometimes it's not a credit thing. It can be also this is very smart flow or very dumb flow, where we don't wanna take the smart flow. We'd rather take, of course, the dumb flow. Some people might say sharp and not sharp. So it is definitely a thing that relationships can get cut off quickly, and they can get cut off for different reasons, et cetera. So yeah, definitely a thing there.

Smart flow, dumb flow, and payment for order flow

Patrick: The classic example on flow is payment for order flow, where retail orders — people transacting their accounts at Schwab or Robinhood — are almost definitionally not hedge funds.

[Patrick notes: You will very likely be detected by KYC/KYB processes at various brokerages if you are, in fact, running a hedge fund or similar, and be Invited to Choose A More Appropriate Provider of Financial Services.

This fact sometimes surprises non-specialists, including users of long standing who suddenly find themselves running a family office. 

Once upon a time, at a much lower asset level than implied by a family office, I got a friendly phone call from a U.S. bank which observed that I seemed to have many more transactions than they expected and asked why. I said, oh, 40 incoming payments a month because my business has two payment providers and each sends in an ACH every business day. The bank congratulated me on running a legitimate business and then said, compassionately but firmly, that I had thirty days to find an appropriate bank for it. Advocates sometimes call this debanking, and one reason financial experts don’t love that term is that it smuggles in the conclusion that there was something inappropriate happening on that call.] 

And so it is extremely unlikely that they have material market-moving information. And so they're transacting because daughter is going off to college, or this is the day that my, you know, 401(k) gets the transfer from work, and then I immediately buy, or et cetera, et cetera.

And so if you knew you were just dealing with noise traders, with people who didn't have a point of view on the true value of a security, or who had no better point of view on it than the market does, you would be able to quote them very generous prices. And to the extent your non-informed flow is mixed in with people who are quite informed flow, well, you have to be a little careful about what prices you quote them.

And then if it is Goldman Sachs on line two, you should be very careful about what prices you give to Goldman, because, as one great line of dialogue from The Big Short movie: "This is Goldman Sachs. If you offer us money, we're gonna take it." And it's deployed against their interest there, but broadly a good rule.

Yeah, there are layers upon layers of this. I feel like we could discuss war stories for a very long time, but one thing I wanna just emphasize for people is this is intellectually interesting and keeping the world running. And there are fun challenges every day. And I'm glad that you came on to share some of those fun challenges.

So where can people follow you on the internet if they want to read about stories in the future?

Where to find Alex

Alex: Sure. So I'm very active on Twitter: @alexpotato. I have a whole bunch of threads about all kinds of different topics. A large portion of that is about finance or technology. If folks are interested in what it's like being a technologist in finance, there's a couple there. My website is alexpotato.com. Also have some book recommendations, and my own podcast is there as well. So yeah, this was great.

Patrick: Yeah. Thanks very much for being on the program, Alex. And for the rest of you, thanks very much for listening, and we'll see you next week on Complex Systems.