Think like a trader, with Ricki Heicklen

This week, Patrick is joined again by Ricki Heicklen, our very first guest on Complex Systems, to discuss the evolution of her trading education business, Arbor. They dive deep into the pedagogy of trading, exploring how simulated markets teach concepts like adverse selection, team dynamics, and risk management through hands-on experience.
[Patrick notes: As always, there are some after-the-fact observations sprinkled into the transcript, set out in this format.]


Complex Systems now produces occasional video episodes like this one. In addition to this site, you can access them directly on YouTube. My kids inform me that I’m supposed to tell you to like and subscribe.

Timestamps

(00:00) Intro
(00:46) Ricki's journey from trading to teaching
(01:25) The birth of Arbor and first bootcamps
(03:32) Developing a trader's mindset
(05:53) Understanding heuristics in trading
(08:21) Adverse selection in everyday life
(15:40) Insights from teaching trading bootcamps
(21:07) Pedagogical approach: learning by doing
(32:00) Handling mistakes and learning opportunities
(36:17) Unplanned bugs and real-world lessons
(39:47) Learning from Knight Capital's bug
(40:24) Understanding exchange-side bugs
(43:10) Risk limits and strategy separation
(44:41) Importance of UI in trading bots
(46:53) The Madagascar button
(48:20) The big red button in manufacturing
(49:45) Simulated trading and information aggregation
(50:29) Sibling trading game explained
(53:24) Modeling and hidden information
(01:01:15) Trading behavior and market updates
(01:04:38) Real-world applications and lessons
(01:13:58) Surprises and market opportunities
(01:16:24) Pedagogical approaches in trading education
(01:17:08) Market dynamics and counterparty behavior
(01:17:53) Retail vs. institutional order flow
(01:19:23) Simplifying trading concepts for beginners
(01:21:27) Introducing market characters and their roles
(01:31:31) Team dynamics and communication in trading
(01:39:13) The importance of redundancy in trading systems
(01:47:52) Future of trading education and online classes
(01:53:47) Wrap

Transcript

Patrick: Hideho everybody. My name is Patrick McKenzie, better known as Patio11 on the internet, and we're here in a video studio this time. But Ricki was actually the first audio guest of the podcast way back a year and change ago, I suppose, when we started it.

Ricki: Yeah, almost exactly a year ago, Patrick.

Patrick: So brief recap of history for people who weren't here for the first episode. You previously did trading at one of the firms that does that professionally. And I think at the time were mooting running a business to train people. Right? And hadn't actually started.

Ricki's journey from trading to teaching

Ricki: Yeah. So I traded at Jane Street Capital for a few years professionally, and after that I left and entered a period of life that I called my "explore before exploit" phase, where I did a bunch of different things for periods of about three months at a time to figure out what is it that I'm actually good at, what is it that I actually enjoy and that I want to do with my life.

About a year ago, I was on the edge of figuring out the answer to that. I'd been running these trading bootcamps, but in a very casual capacity. I taught them to high schoolers at a program called the Atlas Fellowship, and I was really enjoying teaching them, but had at that point never charged anybody money for them, and was teaching them only on a somewhat casual, ad hoc basis.

The birth of Arbor and first bootcamps

Leading up to what is now called festival season at Light Haven, which is where you and I are actually filming this episode right now, there was going to be a one-week summer camp period and they were looking for programming for people to run. So the person organizing it reached out to me and said, "Hey Ricki, I know you run these trading bootcamps. Would you like to run one at summer camp?" And I said, "Sure, whatever. I don't care. Happy to do that."

And then Patrick, you reached out to me and said, "Hey Ricki, I heard you run these trading bootcamps. Do you want to talk to me about it on a podcast?" And I said, "Sure, whatever. I don't care. Happy to do that." At that point, I was way more consumed by running a giant puzzle hunt that I was running, so I didn't think twice about either of these decisions.

Turns out that podcast episode has been the counterfactual launch point of the company Arbor that I now run, because a whole bunch of people tuned in, listened to this episode about the trading bootcamp that I had just run and said, "When is your next bootcamp? I would like to give you my money."

So I said, "Yeah, August," and ran my first paid trading bootcamp in August. Now at that point I had a complicated exponential pricing scheme to determine how much people should pay for tickets. Word of note to listeners: do not use complicated exponential pricing schemes. [Patrick notes: Endorsed. Just charge more than you think is reasonable, and let people select into the offering or not.] 

Even when you are marketing to quantitative trading bootcamp students, they will correctly be suspicious about whether there's something going on that they don't understand. And in the end, we figured out that charging a flat rate for people, which now we charge about $2K per student, though it's been rising over the past year, makes a lot more sense as a business decision and for accounting purposes.

But our first paid bootcamp was in August. And since then, we've run six paid bootcamps, the most recent one culminating this past week at the following year's "summer camp" at festival season at Light Haven. So when you and I spoke, I was really only getting started thinking about this as a serious thing to structure a curriculum around.

We developed some internal trading games and some ways to create this curriculum, but in the past year, figuring out what are the use cases of the people signing up for this, why is it that they're paying money and taking a few days off of work to learn about trading? What is it that we can uniquely provide to them and what do we want to be providing to them? Have been a really fun set of questions for me and a growing team now to think about.

Developing a trader's mindset

Patrick: I would love to dig in on the trading pedagogy specifically to the extent that you want to share it. But let's start with a bit of the higher level question of if one is not using this as a stepping stone into working in the industry or not using it as stepping stone to running their own fund, what are the metaphors and kind of ways of seeing the world that trading gets you which are less convenient to get to if you simply haven't done it?

Ricki: Yeah, so I think a big part of the value that we provide is helping provide people with this trader mindset, this set of tools for looking at the world that they might not get access to basically anywhere else. I think a lot of people know what it means to look at the world through the framework of a mathematician or an economist or a psychologist, and—

Patrick: I would press X to doubt on that with regards to most definitions of “many.” But yeah, sorry to interrupt.

Ricki: Fair enough. Let me rephrase. I think the tools are publicly available such that if somebody wants to look at the world through the lens of an economist, they can go and read books, or take courses or otherwise engage with material that allows them an on-ramp to do so.

Patrick: Mm-hmm.

Ricki: I think this is true for many different domains, and interestingly, not very true for trading. And this makes sense if you think about the incentives involved. The people who know how to think like a trader, the people who have honed those skills, who have done reps and reps and reps of practicing trading and figuring out what works aren't especially interested in spreading that information to the mass public.

I'm super interested in that because it's so damn fun. I just really enjoy teaching trading, and I've fallen in love with it. And I also think that there are a lot of things that people can benefit from in understanding how to think like a trader that are overwhelmingly not limited to financial markets.

Patrick: Mm-hmm.

Ricki: So a lot of what we're trying to do in Trading Bootcamp is not just give people the skills required to turn around and make profitable trades in their personal accounts, or turn around and get jobs with quantitative trading firms, but instead be able to turn around and make good life choices informed by the ways that traders think about the world, that have a unique additional component to the ways that mathematicians, economists, software developers, etc., already have for thinking about the world.

Patrick: So aside from the most obvious level of thinking much more numerically and hopefully more rigorously, what are the ways that traders think that might be less common in say the [university] majors that usually traders are drawn from?

[Patrick notes: Margin Call, a wonderful film for many reasons, created a character who succinctly explains how many geeks were directed from math/physics into Wall Street: “It’s all just numbers really, just changing what you’re adding up. And, to speak freely, the money here is considerably more attractive.”] 

Understanding heuristics in trading

Ricki: Sure. So one of the main ways that we try to hammer in for people is thinking using heuristics and thinking in a way that allows us to understand approximately what is the correct answer. What is the right direction, what is the right order of magnitude? And it is okay if you haven't yet gotten to the point of what is the exact right decimal point answer to this question. Because often being the first one to get there is actually worth quite a bit and having an answer that is approximately right and directionally correct can be enormously valuable if you are the first person to get that answer.

Understanding what kinds of heuristics will get you to the approximate ballpark answer quickly is a pretty important part of being a trader. It's important whether you see financial numbers flashing in front of you, all of a sudden something has changed in the markets and you need to recalibrate what your fair values, what your expected value of a bunch of different stock prices are, and it's important to figure out: “where is there the most value for me to be focusing my time?”

Should I be spending my time focusing on the trading in front of me, or spending time with my family or working on hiring other people who will be able to do the same activities that I'm doing now or journaling in my journal or getting eight hours of sleep at night? I think the answers to a lot of these kind of meta prioritization questions do often have interplays with ways that traders might think about how to make decisions when you're presented with a vast number of different opportunity sets and you need to make trade-offs between them.

I think about the world in terms of trading in a lot of different ways. My wife sometimes makes fun of the fact that often I will describe what other people would describe as a normal social interaction as a trade. A context in which multiple people have what to gain from it. There's a lot of value on the table. Often these things are positive value.

I think when we talk about trading, we're often talking about zero sum situations, but trading happens and takes place because we have a lot to gain from one another. And anytime that you are making a decision about what to do, that you have something to gain and the other person might also have something to gain when there's maybe some degree of disagreement between you about how much there is to gain, but in fact, that might be downstream of valuing things different amounts.

[Patrick notes: Trading also often happens for essentially structural reasons rather than differences of opinion on the valuation of assets, and much of modern HFT is providing financial services to the economy via quickly trading assets versus by having a bunch of salesmen sit down with customers and draw up contracts. As a legible example of this, ever payday the American middle class is a bid for all major assets, due to purchasing them for their 401ks automatically. Many of those assets are wrapped in ETFs for convenience, and so every payday, HFTs worldwide translate the Usual Retirement Bid into the appropriate buy orders for constituent stocks.

Major credit to Luke Constable for explaining to me how capital flows are an important structural factor in asset prices.]

There might be a profitable or a valuable trade for you to do. And better understanding frameworks that traders use when looking at economics, when looking at financial markets and applying those to other areas of our lives can be pretty helpful.

Patrick: Mm-hmm.

Ricki: Let me give another example of this kind of thing.

Patrick: Sure.

Adverse selection in everyday life

Ricki: One of the main ideas that you and I talked about on the podcast a year ago, and indeed one of the main lessons of our trading bootcamp is that of adverse selection. That conditional on getting to do a trade, conditional on being presented with an opportunity, you should adjust your models of the world in the direction that this trade might be slightly less favorable than you otherwise would've thought.

When driving around the streets in the neighborhood that I live in Washington Heights, New York, it's pretty hard to find a parking spot. Sometimes my wife and I will refer to this as the "tour of the Washington Heights fire hydrants." We'll see a spot. We'll look, oh great, there's an opportunity over. Oh, nope, there's a fire hydrant. Oh, there's another fire hydrant, etc. And even when it looks like there's a spot available, often that'll be because there's alternate side parking rules in effect that day, and we simply didn't read the street signs properly.

Occasionally you'll see a fire hydrant, and you'll go, "Oh, I wish that fire hydrant weren't there." But if that fire hydrant weren't there, in all likelihood a car would be. Because conditional on getting to pull into that parking spot, it's way less likely that it actually has value to you. I think that understanding the ways in which adverse selection creeps into every part of everyday life, that anytime you're at the grocery store and able to choose from a pile of tomatoes, you need to be thinking about, "Hmm. The tomatoes in the front might actually not be as good 'cause other people have looked at them and decided against them. Maybe I'll take one all the way from the back that people haven't already encountered and decided to pass over." Can be a useful set of tools for making decisions that look out for your interests, regardless of whether that opportunity set is one that is strictly economic or one that just has to do with other things that you encounter on a day-to-day basis.

Patrick: I would agree that people largely don't have a great intuition for this, and it shows up everywhere in life. Consider things which are iterated games, such as the set of candidates applying for jobs at this exact moment in time. You, the hiring manager, might after a frustrating week assume everyone on the market for this job is an idiot. 

"I've interviewed five people and they're all terrible." There were other people who were assessed by the market for competence, and they now have jobs. The people who are still on the market are the competence distribution that you're detecting. And yet we don't—shaking my fist about the Starfighter thing again, people can listen to our prior episode. We probably won't talk about it too much, but we still persist in having systems and processes that are not informed by that understanding.

[Patrick notes: Incidentally, this is a major reason why recruiters screen for resume gaps; they’re (perhaps without explicitly modeling this) screening the market for lemons prior to spending expensive interviewer time on the mostly-delemoned candidate pool.] 

Ricki: You see this on the job market for sure. You also see this on the dating market. Basically anytime you're constraining a population to what part of that population is actually still available, or let's say is interested in getting a job or getting a date but hasn't had the opportunity to yet and is still searching for that thing, you're looking at a more specific slice and an adversely selected slice in this case compared to what you otherwise would be.

And I think understanding these selection effects and how they affect what the range of counterparties you might be trading against is what the range of opportunities in terms of trades on the order book or employees that you could potentially hire or other possibilities like that. Restaurants with open reservations available, things along those lines will help you make wise choices that understand what the actual details facing you are.

Patrick: It's often something that I see in the financial industry and not specifically trading related ways, how there's differentiation between products at banks, for example, and using the differentiation from products. People often assume sometimes they go as far as assuming like if I'm allowed into this product that suggests that something must be bad about the product, it's like, well, no, you can kind of play that game at level two too, where if you're allowed into this product that they have constructed something where it can be mutually advantageous for two people. Anyhow. Yeah we're getting a little far away from the central example of training by my digressions into everything.

Ricki: Well, actually I will say one more thing on that because I think that a lot of people will react to me talking about adverse selection and say. "Okay, Ricki, if the environment is so adverse. If everyone's out to get you, why does anyone ever actually trade?" And I think better understanding what conditions in an environment make adverse selection more prevalent or less prevalent can be especially useful here to figure out when should I be trading, when should I not be trading?

Patrick: Mm-hmm.

Ricki: If you're driving around Washington Heights looking for parking spots and there are no cars around because it's a time of day that nobody's parking, then it's way more likely that a spot that you pull up to will be fine if you have a less competitive environment. Likewise, if you see somebody pull out of a spot and you have a story for why that spot is available, even better if you can ask them "why are you pulling out of that spot" and find out whether it's because alternate side parking rules are about to go into effect and therefore it's illegal to park there, versus, "Oh, I just have an appointment that I need to go and meet somewhere across town."

You can have a good story for why I'm getting to do this trade. I'm getting to execute on this opportunity that you otherwise might not know, and that can be a helpful motivator for feeling comfortable with the trade you're making. There are examples of positive selection all over times where you and a counterparty might want to trade because instead of both wanting and competing for the same thing, like money, you actually want opposite things in some sense, and therefore are willing to do a trade with one another.

Your desire to trade with each other if you have anti-correlated preferences can actually be a piece of evidence that you should be happier trading as a result. And I think that figuring out how to identify those situations is a big part of what we're trying to teach in this bootcamp in terms of teaching trader mindset.

Patrick: I would love an example of a time where people have anti-correlated preferences, although I can think of a couple, but they seem less central. Maybe you have a better example off the top of your head.

Ricki: Sure. Sometimes when our students are trying to decide which of multiple different groups to sort themselves into when we're splitting them into electives some of them will have the thought, "Well, I know that I want to go to elective A on auction pricing, or I know that I want to go to elective B on how to get a job at a quant trading firm."

And sometimes they'll say, "Hmm, I don't know which one I want. But I know that I just kind of trust the masses. If other people know which one's better, I want to go with that one. So I'm going to ask somebody else and follow along what they want." And then often we'll end up with an elective that's somewhat overcrowded, but sometimes people will say, "I know that the thing that I want is more instructor attention, more quality time with that instructor. So whatever other people are choosing is actually the one that I want to anti choose. I'd rather list myself as indifferent or list myself as, you know, I'll take whatever's left over whatever slots are still available. So that I'm in a less crowded class." And in this case, other people wanting an option is a piece of evidence for you that you don't want that. Even though it might be a positive signal about the quality of that item. If the thing that you actually care about is being in the less crowded group, that can be a sign of positive selection toward the thing that you actually want. Which is to be in the less crowded space.

Patrick: That makes sense. And there are some times where this comes up in things as mundane scheduling. When I'm recording podcasts at the venue that I usually use, which is not here (we use it for audio only episodes), I explicitly tell them my favorite podcast times are times which are bad for almost every other user of this venue. So, please book me towards or reserve availability towards things that someone who is working the typical kind of business they serve doesn't want. Because that will mean it's less likely that there's jostling over this. Maybe you can give a price break, etc., etc.

Insights from teaching trading bootcamps

Perfect. So, you've had about a year now of teaching trading to people who are coming into trading for the first time. Before we dig into and maybe retread a little bit ground on what specifically we do to do that, what are the interesting things that you've learned about the class of people who maybe have less experience with trading hitting that interface for the first time.

Ricki: Yeah, it's been really interesting. So like I said before we spoke a year ago, I'd only ever taught this to people who were, for the most part in high school or in college pretty young, and encountering these topics for sure, for the first time since we started running these bootcamps a year ago.

In part because a lot of people are taking them interested in potentially making a career transition or figuring out what jobs might be available on the market. A lot of our demographic are made up of people in their twenties and thirties. Sometimes we'll even have somebody a little bit later on in their career considering making a pivot. Sometimes we'll have a precocious high schooler or two who are a little bit bored with their usual material and want to try something out. 

And almost across the board, none of them have ever actually done any quantitative trading themselves. Many of them have done some amount of day trading stocks or thinking about option strategies, putting on positions. A lot of them are enmeshed in some community online of people who've thought about these things. But despite that, very few them...

Patrick: Hopefully not /r/WallStreetBets, but… 

[Patrick notes: As someone who is extremely positively disposed towards message boards, I think it’s difficult to read WSB and understand how much of it is a public art performance, and how much of it is people valorizing conduct which some are actually engaging in, and which is destructive to their interests.

There is a nexus between WSB and Robinhood in convincing much of a generation to use options trades as a source of random numbers for gambling. I think we should regret that.] 

Ricki: You know, we get more than one or two from WallStreetBets types. But for the most part our best marketing happens via word of mouth. So some people are coming and taking the trading bootcamp 'cause they've heard that it's really fun. This is actually a pretty core contingent of ours and maybe that's because we don't do broad enough marketing on our own and end up relying on word of mouth from past alums of ours. But it is a really nice selection effect 'cause it means that the people who are coming are earnest and excited and not just trying to get something profitable out of it, but instead trying to have a good time.

Patrick: Mm-hmm.

Ricki: One of our former students once described us as "Disneyland for nerds" and this successfully convinced all of his housemates and a whole bunch of his friends to come and take the trading bootcamp. And it's become a nice little ecosystem among our alumni, the people who are there for the Nerd Disneyland experience.

Another subset of people are coming to take the trading bootcamp because they really want to get a job in quantitative trading and think this might be their avenue there. And I think this is an okay reason to take the bootcamp. I used to strongly caution against this and say, "We are not offering you a job. We are not giving you a job. We are not optimizing around you getting a job and therefore you shouldn't come if you just want a job." Despite us not especially optimizing around this, it does seem like it has actually helped a few people get jobs. In part, I think through just exposure to other people in the area, awareness of the state of the field.

In part it's hard to really know for sure because a lot of the people signing up to take this bootcamp are already going to be pretty well positioned to get jobs, but a number of people have gone from us to a quant trading internship or a full-time job offer, and for their sake, I'm happy that they're getting an opportunity to do that.

But the value that I think we provide much more strongly than that is helping people figure out the answers to the questions. "What is quant trading? Do I like it? And am I any good at it?" In order to decide, "Do I want to invest a whole bunch of months or years of my life in trying to do quantitative trading, trying to prepare for interviews and internships, trying to practice whatever skills it is I'd need, and put out feelers in the broader quantitative trading world, etc.," so that they can make decisions that are looking out for their interests in environments of varying opportunity sets, which might be trying to go into quant trading, or might be saying, "Huh, I've tried this and it's not for me." Or, "I think I just realistically won't succeed at this. I'm going to go and do something else with my life." And to us, that's also a great outcome.

Patrick: I think as you were saying, there's structural reasons why trading is so difficult to access. And it's behind some very forbidding permission gates, both sort of the obvious ones. One needs capital to do it. The type of quantitative trading you do often relies on access to data systems, etc., etc., which would be forbiddingly difficult for people to build on their own. But you also very literally have to beg for a job in the field. Well “beg” is a strong word—do the usual set of social processes to get a job in the field to get an experience of what the field is like.

And you can contrast this to other walks through life. There is some amount of gatekeeping in programming. There is some amount of, you are much more likely to get certain tier of jobs that will expose you to certain tier of problems if you've been through the right feeder schools and the right feeder institutions. But you can also download all the software for free on the Internet and just play with it.

And there is a sort of secondary information market these days of people on YouTube who are showing people from literally downloading your first client to making your first program to dot, dot, dot to give people months or years of experience on whether they would like the actual job before they get into the actual job.

Which—shaking my fist—it's a shame that more professions don't have to put a little more thought into what they would do for feeding people into the profession on a 10 plus year time scale. They should fund more anime among other reasons, but that's neither here nor there.

So, if we can talk a little bit more about the actual pedagogy. I know what some of the lessons are, but of course, spoiling the lesson or giving people differential access to the lessons prior to the lesson happening seemed negative, both for pedagogical purposes and all sorts of other purposes. But speaking generally, what's one lesson you would like to teach people and what's the non-obvious thing about how to teach that lesson?

Ricki: Yeah. It might be useful if I back up for a moment.

Patrick: Sure.

Insights from teaching trading bootcamps

Ricki: And explain a little about our overall structure of our bootcamp. So when people walk through the door, the very first thing we do is have them start trading. Absolute number one, immediately just throw people in without explaining the rules, without explaining too much logic about what is all the syntax going to look like? We say, "Hey, we're going to go around, play this trading game," and we get them started and correct as they go. And that's because a very very big part of our pedagogical philosophy is that the best way to learn how to trade is to trade and figuring it out through making mistakes and being corrected as you go and figuring out what happens when you actually try something and see what changes in the market as a result of that. And then can course correct if that was the thing you didn't want, is going to ingrain those lessons in you so much more strongly than something that's more lecture based or something like reading out of a textbook.

In order to cause that to take place, not just in our first hour long class, but over the course of the entire four day bootcamp, a thing that we'll do is throw everybody into a virtual economy together. We have a pretend currency that we use called clips. And everybody starts out the week with 2000 clips and is told your goal this week is to optimize for maximizing the number of clips that you have by the very end. So we put people through a series of trading games, trading scenarios, or simulations. In which they go in trading either independently or together with a partner or team and trade in a bunch of different scenarios that are intended to increasingly come to approximate real live financial markets using their play currency, using clips, and being able to see how their currency balance goes up and down over the course of the four day period.

Patrick: Mm-hmm.

Ricki: By the very end of the week, people have accumulated some number of clips in their balance as a result of both trading that they've done out loud with one another and trading they've done on our electronic exchange and the final night. Our after party is also an auction called "provide your own liquidity" in which people can auction off bespoke goods and services that Trading Bootcamp students can spend clips on.

So each of our modules over the course of the week of Trading Bootcamp is a few hour long session in which there are some parameters for which people are engaged in a variety of trading exercises. With markets that they can interact with much the way they would real financial markets, but not having to worry about putting their own money directly on the line. With the principle being that if the best way to learn how to trade is to trade, but in the real world, in order to actually trade, you are risking a whole bunch of your own money and potentially going to bankrupt yourself if you don't understand what's going on. It's best to give people a play version of this thing in which they can mess around and experiment and see what happens.

One of the most common conversations I will have with somebody when they're in the process of doing some manual trading, or let's say writing a bot that's going to execute a certain strategy is "How do you know that your bot is going to succeed at what you're trying to accomplish?"

And they'll start explaining to me all the logic in their code or what it is there that makes sure of it. And I say, "Okay, let's do it. If your bot can successfully perform an arbitrage," i.e. a strategy in which one can buy and sell in multiple markets in order to at least in theory, take on a risk-free positive dollar value position.

Patrick: Mm-hmm.

Ricki: Or a flat position that still results in them making money in those markets. I say, "Great, let's set up the markets so that one could perform an arbitrage and see if your bot actually does this thing. See if that actually happens." And they're like—

Patrick: I have a prior on how much success there is with it, but won't spoil the story.

Ricki: I think that occasionally this works. More often than not, the thing that happens is their bot fails to perform an arbitrage and they have no idea why. It's totally unclear to them why the arbitrage didn't go through. So the thing that we'll do from that is we'll unwind a step back and say, "Okay, what is behavior that you feel confident your bot would successfully do? Under what circumstances do you think it would trade?" We write down some predictions about what the bot that they wrote, whether by themselves or with the help of AI models, which are now a pretty standardized part of our curriculum as well.

Patrick: Mm-hmm.

Ricki: "What is it that you think will happen if the order books? If the markets that we are using for our simulated trading games look a certain way."

Patrick: Mm-hmm.

Ricki: They'll write down their predictions, then I'll actually throw some orders into those markets to cause the markets to look that way.

Patrick: Mm-hmm.

Ricki: And then they'll notice ways that the behavior that they predicted did or did not occur. Then we'll all try to come up with what are our best hypotheses for why this did happen or why this didn't happen. "Okay. What's the next step? Well, if your hypothesis is correct. Then what would happen under conditions A, B, C, now let's go and test those conditions." And by actually seeing the ways that the markets move or don't move as a result of the behaviors that they're taking, they learn in a way more ingrained, feel it in your bones way. Whether or not the trades that they're hoping will happen will happen and what it is they need to change in order to bring that about.

Patrick: Mm-hmm.

Ricki: A shape of student that we very often get is someone who is quite talented at software engineering or at mathematics. And quite good at reasoning from first principles and way less comfortable with just try things and see what happens as a result. And be okay with making a few mistakes or experimenting with something before the point that you've gotten the exact right answer.

Patrick: Mm-hmm.

Ricki: Here's a pretty interesting selection effect we see among the students taking our class. There is sometimes a negative correlation. Between students who are very strong software engineers or very strong mathematicians and students who have what I like to call the trader mindset. These certain heuristics for "I can get something approximately right. I can get to an answer quickly and execute on that. I have a sense of what's actually going to happen in this market as they result of my behavior." A certain kind of scrappy willingness to try things and figure things out organically and have intuitions for ways things are going to go.

I don't think these traits are negatively correlated in the population as a whole.

Patrick: Mm-hmm.

Ricki: I think these things are probably pretty positively correlated and track pretty closely other general traits like intelligence and critical reasoning. But among our students, I think we are selecting for students who have some interest in trading.

Patrick: Mm-hmm.

Ricki: And we see a phenomenon that's sometimes referred to as Berkson's paradox going on here, where by restricting to that subpopulation, you're going to see people who are either really strong at math, software engineering, whatever it may be, or have this other scrappy, tradery characteristic that isn't as well defined and established in society in general that is driving them toward taking this class.

One of the things I feel most excited about in this bootcamp is the way in which we can capture people with that kind of interest and help them figure out, "What do I want to do with that? I have trader brain. I have something that allows me to get a good sense for what are the selection effects of my environment? What's going to happen as a result of my behavior? Why would I, who might be pushing these other buttons? And who might my counterparties be? And how should I think about modeling a world in which they exist? And then I need to react to what they're doing? And if I'm reacting, what level of the game are each of us playing on, etc."

And helping them figure out, "How do I want to use that skill? What do I want to do with that talent? Do I care about quantitative trading? Maybe yes, maybe no. If I care about other things, what good is this trader brain mentality that I have, and where should I apply it?" I'm actually really not that invested in there being more quantitative traders in this world.

Patrick: Mm-hmm.

Ricki: I think a lot of people look at me and say, "Ricki, you run a quantitative trading business. You invest all of your time in thinking about how to teach quant trading. You sure must care a lot about keeping US equities markets moderately more efficient." [Patrick notes: Flagging Ricki’s dryly sarcastic wit for people reading along.]

Patrick: Mm-hmm.

Ricki: I think for the most part, I really love teaching and I really love helping people figure things out. And I really do have this trader brain thing and recognize that it has some value, but I'm not at all convinced that the thing to do with that value is add another quantitative trader to the efficient markets in the ways that they currently exist and make them moderately more efficient.

Patrick: Mm-hmm.

Ricki: And helping people identify this talent and figure out "What are the things I want to do with that? What kinds of strategic and mathematical and scrappy, heuristic based reasoning skills that I have, do I want to apply to different parts of my life?" is something that excites me more than almost nothing else.

Patrick: Yeah, I think there are a lot of them. 

The LLMs someday might be able to tell us what the various unlabeled vectors are in human space. But whatever this trader brain or intuitive feeling is, there there are some other things like that. As a random example, for guests of this podcast, people who have done professional gambling before are massively overrepresented. Some of that selection effect, some of that is that we fish in the community, and so we get bites from selected fish. 

But some of it is like, if you're drawn to that particular weird thing, you'll be doing other high variance, weird things in the rest of your life. And eventually, you might end up talking here about one of them.

And I do think I could give a pretty good apologia on how society should put more resources than are obvious into getting US equity prices and ETFs to be priced accurately. It's very important for many people's retirement. Financial innovation has decimated the cost of people saving for their retirement. The EFT wrapper, which essentially requires HFT to function well, caused the decline from paying 1.6% of assets every year for mutual funds to like 0.06 percent for ETFs.

[Patrick notes: this might not sound like a lot, but running that forward 30 years implies approximately 50% more assets available to the retiree. ]

But I'm very much getting into the weeds.

Ricki: I love getting Patrick talking about this stuff. Yeah.

Patrick: I am exactly the kind of guy I am. What can I do?

Ricki: Amen.

Patrick: So, I do love that you mentioned that you throw people into the deep end straightaway and get—I feel we do so much in our society to tell people like, you must not make the mistakes. You start at a hundred points and can only go down from there. And you're going to be ostracized in class, get a wrong answer in front of the teacher, etc., etc., etc. And love giving people permission to be imperfect, particularly in liminal spaces like a game or a bootcamp or similar, where this obviously isn't on your permanent record. You aren't burning a scarce resource here other than your own time and attention. Just try things and see what they work.

One of the most impactful experiences in my life and a pedagogy I see very, very infrequently was when I got into my first Japanese class. And the first two words of the class written on the board were "no English." And you can go a surprisingly far way on learning a language without any basis in the language, and no other language. This has been demonstrated by, well, every child ever. I will try to avoid a digression into comparative Japanese pedagogy.

[Patrick notes: I swear by Japanese the Spoken Language, but will note that the young’uns I meet these days with the best spoken Japanese overwhelmingly got it by shadowing target speakers on YouTube. (Shadowing is an exercise that translators/interpreters are told to do to firm up fluency: play a source speaker and attempt to reproduce exactly their output at the same speed, cadence, intonation, and similar as they do. Do this much more than you think is reasonable. This is by a substantial margin more effective than most forms of drills, which are more effective than most unstructured conversations or studying vocabulary via memorization.] 

So people have a goal of like, earn all the clips, but obviously that isn't the real goal. I would love if you could spend a few words on the tension between that the goal that is earn all the clips and the other goals that bring people to the Trader Bootcamp. Does that generate positive sum traits for people?

Handling mistakes and learning opportunities

Ricki: I'm happy to, before I go there, I actually want to piggyback a little on what you were saying about making mistakes. Because I think a pretty big part of our bootcamp is trying to get people comfortable with recognizing that they make mistakes all the time. The people they are interacting with, whether they be their partners on their teams or their counterparties on other teams in our trading games, also make mistakes all the time. That this is not only okay, but a thing to be closely modeled and understood about people and to figure out what protections you want for yourself and what opportunities will be available to you in light of existing in an environment where a lot of people make mistakes.

Patrick: The oh man—story of my life. So many people run around with a heuristic from the markets that is effectively the strongest possible form of the efficient markets hypothesis, where every company that has smart people and lots of resources must be investing all those smart people and resources at the correct margins at all times. And therefore, anything bad I see in the world is planned for some reason. And also there are never mistakes in the plans.

This is not a great model to attempt to understand the world. And you'll miss many opportunities for positive sum trades. If you have that model, sorry, I have to rant on that one.

Ricki: Yeah, I think that that is, it's definitely true that this model breaks down, it particularly breaks down in environments where the conditions of those environments are brand new. Or are very suddenly shifting or reacting to something that's happening in an environment that hasn't had a chance to adjust into an equilibrium yet. These are often the points where there's the most opportunity to be had. In the very first trading scenarios that we throw our students into at the very beginning are both the points where each student is least equipped to do good trading 'cause they don't know what they're looking at yet. And also the point where the trading is best because competition hasn't yet eaten away most of the positive value that they'd be able to get by trading against, let's say, our bots or our automated market makers in the environments that we're throwing them into.

One of the things that I try to emphasize to students about mistakes is when you are making a mistake, you are getting a great piece of data. About ways that the human system can be buggy, and that is a piece of data that is useful not only in predicting your own future behavior, but also in predicting the behavior of both your teammates and your counterparties. And you want to hold onto that and take away a really strong lesson from it.

Being loud about your mistakes, and this was something I learned from my former employer at Jane Street, pretty strongly is an extremely valuable thing, and it requires letting go of some ego.

Patrick: Mm-hmm.

Ricki: This is way easier said than done. I think everybody would co-sign the claim. "Oh yes. Being clear about your mistakes so you can learn from them is a good thing." And it's much harder when you're actually making those mistakes and under pressure and have eyes on you to do so.

Patrick: Mm-hmm.

Ricki: But one of my favorite things is when I make a mistake in front of students. When I'm up in front of the class and I make anything from an arithmetic error, which let me tell you, happens all the time to some kind of fundamental reasoning error. This happened last week when I was talking about logarithmic returns versus linear returns versus other types of things like that. And a student called me out and I got to say, "Great, of course I was doing this deliberately so that you guys could learn the kinds of mistakes that people make all the time that you interact with people. They're going to be making mistakes of various different flavors, and knowing which kinds you make are going to be really, really helpful at knowing both what to defend against in your own case and what opportunities might emerge as a result of that."

Patrick: The bugs are carefully planned pedagogical content, which is both a joke but also serious.

We had a couple bugs in Starfighter back when it was still a thing. And one, sometimes people thought: carefully planned pedagogical content! Including with regards to the bugs that were not in fact planned. They were just me, a fairly experienced programmer, introducing bugs accidentally. (As has never happened before to any other programmer, I am sure.)

But it is a good reminder that we live in an imperfect world where people do imperfect things. And a model that includes people doing imperfect things, particularly in the way which are like attractive basins for those imperfect things, will outperform a model which says everyone is always a hundred percent efficient Homo economicus 100% of the time.

Unplanned bugs and real-world lessons

But interestingly, do you have any planned bugs in the simulations before we go into the some unplanned bugs?

Ricki: I sure wish that we were at a point where having to write in planned bugs was a relevant pedagogical tool. Fortunately, we have so many unplanned bugs that this hasn't proven necessary.

Patrick: Mm-hmm.

Ricki: Let me tell you a couple examples of the kinds of things we see as unplanned bugs. So, back in November, we had a system in which we'd set up a number of different markets, and these markets included individual stocks as well as ETFs that were aggregates of those different stocks combined. And the way in which our internal bots would reason about those markets and decide what to do is they would, let's say, flip a coin. And as a result of that coin flip, decide whether they were going to buy or sell a certain number of shares. So we had one bot, let's call it Alice. Alice is one of our standard hedge fund bots that decided to buy a number of shares. And the thing that Alice did was looked at the market, looked at the price, figured out what price she'd have to buy up to in order to buy 50 shares and bought those 50 shares.

Patrick: Mm-hmm.

Ricki: And then her code said, "The thing you want to do is buy 50 shares." So she looked at the market and figured out what price she needed to buy up to, to buy 50 shares, and she submitted another order to buy 50 shares.

Patrick: Mm-hmm.

Ricki: And then she did it again and again and again, and at no point checked her position in order to see "had I actually already performed that action, and bought those shares" and ended up in an infinite loop in which she kept buying and buying and buying because no condition ever told her that she had successfully satisfied this.

A few really interesting things happened. One was the person who had written this specific piece of code on our team was busy talking to a team about their trading behavior. And nobody else on our team knew how to operate that code. And as a result, instead of as soon as we realized this bug was happening, because price graphs were looking insane and because positions were in all sorts of places they shouldn't be, it took three different people to track down the relevant person and get him to come back to the computer and shut down the code. 'Cause nobody else knew how to do so.

Patrick: Mm-hmm.

Ricki: This was a major source of risk that we, the central exchange that ended up distributing 75,000 clips far more than we usually give out to students did not realize we were taking on. But had we paid any attention to the fact that only one person really understood our code base and had written this up, we would've realized was a major source of risk that we were encountering.

Patrick: Mm-hmm.

Ricki: Another source of risk we had was, it was way harder for us to see these problems because we didn't always have all of the relevant graphs and charts up in front of us on the screen that would've showed us, "Oh my God, all of a sudden there was a huge price spike in one of these charts." And therefore, something is going wrong, that something might be a student team doing something wrong, but it also might be us doing something wrong. And we should immediately check our systems and see what everybody's positions are, see what behaviors happened in the past few minutes, etc. Because we didn't have a good monitoring dashboard. We didn't have immediate access into this problem, and it took us way longer to notice it and intervene and resulted in us losing a lot more clips into the ecosystem than we otherwise would've.

Patrick: I love that the amount that you're paying for these learnings is so much less than so many postmortems have said about paid for these learnings in real financial markets. This is the story of Knight Capital. This is part of the story of FTX. [Patrick notes: The “poorly labeled fiat account” is not the sum of the badness at FTX, but is part of why management accomplished a controlled flight into terrain versus successfully keeping their plates sipping..] 

If you squint at it, part of the story of Mount Gox, if you squint at it, and I can probably come up with another five examples with more time to think. [Patrick notes: The best discussion of the Mt. Gox failure is Wizsec’s investigation, which I took notes of.]

But yeah, graphs: have them up. They're helpful.

Learning from Knight Capital's bug

Ricki: A hundred percent. I think it was great because immediately following that we got to tell everybody the story of Knight Capital. Talk about the times that bugs in systems and the lack of other systems surrounding those. To be able to suddenly shut everything down or step up and deal with this problem somehow in a kind of holistic way to identify the problem and to step in and fix the problem ended up biting us in the butt much in the way that it has bit countless financial firms in the butt to the tune of a lot more than 75,000 clips, our pretend currency.

Understanding exchange-side bugs

This was just a fantastic learning opportunity. It was also a great learning opportunity for the students for whom I see time and again, the problem of it not occurring to them that the bug might be happening on the exchange side.

Patrick: Mm-hmm.

Ricki: They think, "What is the story for what's going on here?" Well, probably they're just confused about the fundamental rules of the trading scenario we're engaged in. Or maybe what this means is that somebody's die roll or somebody's secret information about the stock or whatever is at the extreme end. And not somebody's system is broken. Occasionally they'll think maybe another team's system is broken, or maybe our system is broken, but it's very, very rare that they will generate the hypothesis, "The exchange made a mistake." Or "The rules that we presented to them are rules that we are not succeeding at, abiding by about what behavior we're going to have either through error or through malice."

While in fact, quite often we have some sort of bug in our system and we're having fewer and fewer of these bugs over time, but they definitely still crop up and. Helping them understand that sometimes when you're looking at weird data, it's because things are breaking in parts of the system that weren't within your model of where things in the system might break is a pretty useful lesson for them to learn. 

That absolutely happens in real financial systems all the time.

Patrick: Yep. And indeed there is a well—you don't want to be having this discussion when things are crashing around you because they might be crashing faster than your ability to process information on the discussion. 

But one of the things in real financial systems is, for example, there's a rule book that's by the exchange. And in the case of things which are well out of the distribution, the interest of orderly markets, blah, blah, blah, there is a procedure for what to do.

 And then people are sometimes making guesses on, okay: contingent on trading right now, am I really trading? Or am I going to be in a trade that is broken by the exchange post facto, which for obvious reasons, exchanges don't like doing, but will do. And that's why there's a rule book.

And so we have to successfully anticipate the operation of essentially the semi-judicial procedure that will be following the rule book.

For example, —if you're offering me Google stock for $10 a share, you're mistaken. If I'm given that opportunity, and attempt to execute, very possibly I will not actually get the share. Should I want to buy Google stock at $10 a share? What about the potential opportunity to have it at $10 a share? You have to reason your way through that.

And again, it is easier for you to reason your way through that when you've reasoned your way through it before, rather than encountering the once in a decade bug and everybody trying to quickly reason through their first once in a decade bug, or finding someone who's been around for two decades to tell them, "Oh yeah, this is just like '93 again." [Patrick notes: That was 3 decades ago? Crikey, I feel old.] 

Ricki: Yep. Exactly. I think that the other big takeaway that we got from running this experiment and finding out that we had this bug was it's pretty useful to have risk limits. Even on systems that you think, "Oh, well, I've written them in ways that should cause them to never lose more than 500 clips. So why should this be a problem?"

Well, if you can make sure that they don't lose more than 500 clips, or at least make sure as much as possible by, let's say, giving the account that is running that strategy no more than let's say a thousand clips or something like that, you are going to be able to place much safer constraints on how much money you lose to this system than if you just treat everything like an admin account. Same thing for separating out your different accounts from each other, both for purposes of safety checks, like preventing these outcomes. And also for purposes of getting a better sense of which of my strategies are doing how well can I actually separate these out into different accounts and pay attention to the strategy that's doing market making and the strategy that's coin flipping and crossing spreads—i.e. trading with either the best offer, the lowest offer, or the best bid, the highest bid, depending on which side of the coin comes up, or other strategies like those.

We'll want to be able to, both for ourselves, be able to see how well these different strategies are doing and also demonstrate to the students, "Hey, look, this is what Mark, the market maker as we call him, is doing in the markets at each point in time, this is how much money he's made or lost in each of these markets. Why do we think that is what's going on in this market that makes it the case that Mark, the market maker is making more money? What's going on in this other market that's making him lose money there?" etc.

And then this is one of the principles that we'll try to teach our students too when they're building bots and having those bots trade in these markets is, "Hey, it looks like you're trying to implement six different strategies here. If you have them all trading out of the same account, it's going to be pretty hard for you to answer questions like, how much money did I make in the past five minutes? And do I want to continue executing this trade or not? And is something broken?" I think having a UI set up and having a infrastructure set up under the hood that allows you to easily answer those kinds of questions is going to make your interface with the bot way better.

A lot of students, especially the software engineering savvy types, come in thinking, "I just need to build a really good bot. I need to focus on getting good latency and making sure that I understand that it's executing the arbitrage, etc.," and don't focus nearly enough on UI and on how you can have good feedback loops between you, the human monitoring things and the bot that is executing these trading strategies. And as a result, end up writing something that is 90% of the way to a good answer, but never know whether they've gotten there or not. And in some cases end up losing tons of clips in our play money economy.

Patrick: Mm-hmm.

Ricki: And would benefit a lot from something that is a much simpler stupider strategy. A way, way, way less advanced algorithm, but allows them to know, "Am I actually making money?” and iterate on that?" 

This is true especially in environments where a lot of their fellow students, a lot of their counterparties, are making mistakes of type one of "90% of the way to a really good advanced, sophisticated arbitrage bot. But maybe now it's doing insane things that are setting up great opportunities for other people."

Patrick: It's interesting to see my code before and after where working at Stripe, which is not in trading, but is in the financial industry.

I wasn't an engineer at Stripe, but the notion of a for loop that can lose money on every iteration of the for loop is very clarifying for the mind. My code got much more defensive after e.g. reading incident reports for a few years. And so there was something I wrote last year, a trivial bit of code could have been five lines long was 200 lines long because it was 195 lines of "if you detect X die, if you detect Y die," blah, blah, blah, blah, blah.

[Patrick notes: Here’s just the command line parsing part, for flavor. 

]

And then, a feature that goes into all my systems these days that I affectionately call the Madagascar button—make it very easy to Shut Down Everything. It's called the Madagascar button because of a video game called Plague where you're trying to kill the entire human population and your greatest enemy is the President of Madagascar, who notoriously in the game community is capable of shutting down your access to Madagascar very quickly, and thus, humanity survives another day.

If you want to survive another day, make sure the system has a Madagascar button in it.

Ricki: Yeah, I think people are often pretty scared of using this button. They're scared because of the sunk costs. Of, "Well we put so much work into setting up this system, if we just press the button and turn it off, that's no better than if we hadn't written something in the first place." People are scared because there are often liability reasons in real financial markets to having your systems go down if other people are expecting you to be providing liquidity, whether those be people to the tune of your counterparties who are looking at markets and hoping they'll be liquid and only coming back if they are, or people like regulatory bodies.

Patrick: Mm-hmm.

Ricki: But in fact, the cost of not shutting down your systems at times that they might be doing risky things. Is such a greater cost—is an existential cost that you might be taking on. And having a culture that is okay with taking that short term hit in order to not be in a state of deep existential risk is a pretty important thing to learn when you are trying to gain this set of trader heuristics. And I think a thing that applies in the real world too. There are a lot of situations where having the ability—

Patrick: So manufacturing in Japan they called the big red button andon cords. But you give anyone historically in well-managed Japanese companies following these set of manufacturing practices, you give literally everyone in the factory the authority to hit a big red button to stop the line if they detect a quality defect or that the process is out of control.

[Patrick notes: Famously, a certain large automobile manufacturer had a QC process which would involve measuring parts as they came off the line then making a mark on a blackboard. It taught workers who were sweeping floors enough Stats 102 to understand what series of marks on a histogram meant rejecting the null hypothesis that the system was under control, and made it very clear to cleaning staff that if they were to notice that, they were to scrub the line. In the finest tradition of Japanese management, this is a statement to one’s cleaning staff, but it is (importantly) a larger statement to the rest of the organization.]

Patrick continues: And then of course, you want to very quickly debug the thing that is causing the process to be out of control. But you would much prefer to be doing the debugging when people are realizing they're in an emergency situation and the process of out of control versus, "Hmm. That car seems like it doesn't have a door on it. That's a little weird. Are we doing cars without doors today? Maybe if I wait until lunch to ask Taro." etc.

Ricki: Absolutely.

Patrick: So sorry. You give people the intuition and allow them to learn for the low, low price of 250 clips that some ability to instrument their software and see what the actual operation of it and make that more comprehensible to themself does. And so, put more effort into these sort of things before you start trading for real money. I think that's one of the advantages that people who are doing this at firms have, and that there is institutional memory of these things and often a risk layer, a controls layer built up systems that have successfully anticipated some of these problems.

But yeah. So what are other fun situations that people get to experience in this simulation of trading that would otherwise be very expensive educations later.

I wanted to get a little bit more concrete on how we explore trading in the simulation, and particularly how trading helps to enable one of the primary functions of markets generally, which is to aggregate information about things that people might have privately or where information is hidden or not well understood and quickly broadcast it out to the market and how multiple layers of traders reacting to other traders reacting to hidden information causes that to happen.

So do you want to tell me about how many siblings you have?

Ricki: Yeah, so this is the very first market we have people trade on, like I mentioned earlier on, when people walk in through the door before we do almost anything else. No connection to the wifi, no "here's what the schedule is like for the week." We have them immediately start trading. And the very first contract that we have people trade on is how many siblings do we, the people in this room right now have. Combined. So we're trading on the sum of the number of siblings of the people in the room, which is usually about a dozen people. It's going to include me, a co-instructor, and about 10 students sitting around in a circle, playing a game of "bid or trade" where each person needs to put a new order in the market, either a better bid, a higher bid, or a better offer, a lower offer or trade with an existing order in the market.

Patrick: Mm-hmm.

Ricki: And the contract itself is going to settle to the number of siblings that we have summed total, which means that Patrick, if you and I are playing this game together with 10 other people and I have two siblings and you have three siblings and everybody else is an only child and has no siblings, then that contract will settle to a value of five. 

The sum of our siblings will be five, and anybody who bought this contract for a price of four clips, clips being our pretend currency, will end up making a profit of one clip. And anybody who bought this contract for a price of 20 clips, if it ends up being five, will end up having lost 15 clips on that trade. Just as if you were to buy an apple in the market for $20 and then sell it back to market the following day for $5, you'd be losing $15 on that set of trades.

Okay. So one of the exciting things about this is each person in the circle typically has a pretty good model of how many siblings an average person in the world has.

Patrick: Mm-hmm.

Ricki: And they have some hidden information, which is how many siblings they have. So usually at the end of trading, once things are all settled up, we'll ask people, "How did you form your models? What did you do?"

And people will walk us through their reasoning.

You'll usually hear something like, "Well, I know that the average household size is something like 2.3." Where do you know that from? "Oh, I just have that cached in my head as a number from when I was young." Okay. Sounds great. Therefore, the actual number of siblings, each average per each person has on average is 1.3. 'cause you have to subtract yourself. 

Okay. Sounds good. Therefore, if there's 12 people in this room, I need to multiply that by 1.3 and that's going to take me to like, I don't know, 15 or 16 ish.

Patrick: Mm-hmm.

Ricki: And it's pretty important here that I emphasize the thing you need is not a hyper precise model that gets the exact multiplication right. It's approximately what are we trading around as a starting point. When you get thrown into this from which you can refine your model in a bunch of ways, one of those first points of refinement is someone will say, "Well, how many siblings do I have? That's a piece of information I have access to. Now I'm going to adjust things by a bit. 'cause I probably don't have exactly 1.3 siblings. Let's say I have four siblings. That means I'm going to be adding another 2.7 siblings to whatever number it is that I produced by multiplying 1.3 times the number of people in this room."

So now we have a bunch of people doing trades with one another based on models that are reasonable approximations of the world as a whole, and also have some hidden information about their family size.

Patrick: Mm-hmm.

Ricki: And as trades start happening between people, we'll notice that some people tend to consistently be on the same side of the order book. So if Brent starts buying contracts and is now long three shares because he's purchased this contract three times. We think it's more likely that Brent has a higher number of siblings than the average number of siblings that someone in the room has.

Patrick: Mm-hmm.

Ricki: Likewise, somebody who keeps selling this contract is more likely to have fewer siblings. 'cause if they're selling, then they think that the actual value is less than the prices that they're trading at, and therefore they think that it's more likely compared to other people, their fair value, their expected value for the number of siblings is lower. One of the inputs to that could be they just have fewer siblings themselves.

Patrick: Mm-hmm.

Ricki: There are other ways that people might have hidden information that aren't just how many siblings they have. One year ago, you and I recorded a podcast in which I made the now questionable choice of revealing how many siblings I have.

Patrick: Mm-hmm.

Ricki: Some of your listeners then signed up to take my trading bootcamp and using that alpha, using their information that they had gained, made some profitable trades as a result of it. This year, I'm going to leave that number a mystery, and people who want to do the research can go back through the books and find it.

Patrick: Mm-hmm.

Ricki: But for people in that position. They took on certain types of positions in the market as a result of their hidden information and other people got to find out, "Huh? Other people had even more information than I had. 'cause I had this cached 2.3 number from which I subtracted one and then did some multiplication, etc. And I had my own number of siblings, but other people had their number of siblings and Ricki's number of siblings. And maybe they discussed this with a few other people in class before class started. 'cause they knew what the first class of this trading bootcamp was going to be. So they also knew other people's numbers of siblings, etc., etc." What does it mean about the environment I'm in that other people are willing to trade with me? How should I be updating on my models of the world in light of their willingness to do this trade?

Patrick: I would love calling out here that it is the case that people have unequal access to information in the world. And many people's immediate sort of instinctive reaction here is, "This is unfair," etc., etc.

[Patrick notes: As Matt Levine has noted many times, many people’s moral intuition is that insider trading is banned because it is unfair. It is actually criminalized, at least in the U.S., because it is misappropriation of value from one’s employer. The financial markets are places in which people with grossly unequal resources and knowledge trade. The schoolyard sensibility of fair, where everyone has an equal chance to win the game, is not something that sophisticated professionals expect from the markets, and if they were to obtain that outcome that fact would be alarming.] 

Patrick continues: And, the aesthetic appreciation for fairness does not lead one to make either net beneficial trades for oneself or utility maximizing trades generally with regards to these things. You should learn early and often that if you have been invited into this room you do not have the same information as everybody else in the room. You will virtually never be in a situation where everyone has access to the same things. And also sometimes it seems trivial—it sounds unlikely that market moving information is one Google search away and yet people don't organize themselves to do the one Google search. But there are so many times in life where there are lots of real dollars where market moving information is one LLM invocation or Google search away and people don't organize themselves to do the Google search.

[Patrick notes: People have such a strong prior for the Efficient Markets Hypothesis that they assume this has basically never happened, but in fact we see the maximally surprising version of it happen every time, when the US equities markets fail to ascertain material information for an extended period before coming to sudden consensus about it. Byrne Hobart has the distinction of being the likely but-for cause for the tech industry finding out one of their key banks was insolvent. That required reading a quarterly report (or the FT commentary from someone who had). You would naturally assume that many people on Wall Street had long since done that, and yet.]

Ricki: That's true. Yeah. And in this class I often ban Google, though not always before the point that somebody has successfully looked something up. And of course there are these students who've, whether because of hearing a podcast or talking to me casually over lunch, let's say they happen to know that I'm from a religious Jewish family that then have happens to have more siblings or something like that. They'll have some kind of alpha in this market, whether or not I try to specifically intervene to enforce some sort of aesthetic standard of fairness or something like that.

But I think that there's also this kind of interesting game of there being multiple layers of "what you think other people think." In light of their behavior, what their models of you might be factoring in and ways that you should be. Correctly modeling how many layers deep they're going. I think often we'll see people who don't model any layers to other people. They'll come in and say, "Okay, I think the average number of siblings is 1.3, and then I multiply that by the number of other people and take my number of siblings and then make zero more updates beyond that in figuring out what is my fair value, what do I think they'll like? My best guess about the expected sum of our number of siblings is over the course of the whole game."

And you also see people in the opposite position of, "Well, I know that I don't know the answer, and other people are making wise choices. So whatever the market says is probably just the correct answer. Who am I to assert my own beliefs into the market? I must update that everyone else has also already incorporated each other's information." As a result, that person doesn't end up inserting their own information into the market. Because they are, instead of overly confident in the model they started with, they're overly timid about inserting their own views and taking out some stake in this market.

A thing that my wife often hammers over the head when teaching this class with me is, "Trading means I think you're wrong." [Patrick notes: Materially untrue, I think. It is true to a first approximation, but falls apart under any scrutiny in the real world.

For example, some traders are hedging an exposure, someone might be selling a call not because they’re bearing on the stock but because they’re buying a call spread (bullish on the stock but mechanically they will sell one contract for every one they buy).

Some trades happen for structural reasons, as discussed in another note, and do not necessarily represent any instant POV of market prices. (In fact, the capability to be ambivalent about prices is one thing the trader is purchasing from the market!)] 

When you are trading against someone, you are stating that you have opposite views of the world. If I'm selling at a price of 20 and you are buying for a price of 20, it's 'cause you think there are more than 20 siblings summed up among the people in this room. And I think there are fewer and those worldviews are incompatible.

Patrick: Mm-hmm.

Ricki: We are doing a trade because we disagree about the value of this contract. In real life, when you're trading for a t-shirt, you disagree about the value of the T-shirt relative to dollars. In a way that could be totally fine because I want the dollars more and you want the t-shirt more and we're happy to do this trade 'cause we'll both get what we want.

Patrick: Mm-hmm.

Ricki: But given that this contract, the number of siblings pays out to be a number of clips, and the thing you're using right now to trade for it is a number of clips. In all likelihood, this disagreement about the state of the world, about what your beliefs are. Is happening in a way where one person is going to be happy and one person's going to be sad. This is much closer to being just truly a zero sum trade than a lot of the other kinds of trades we might experience in day-to-day life. So it's going to come from a disagreement in models where somebody at the end of the day is more right than the other person, perhaps because they have a more sophisticated model with more input points, perhaps because they got lucky and somebody with a ton of siblings happened to be sitting in the room even though they didn't know about it.

But that if the market is behaving as it "should," and everybody is doing the correct amount of updating their models of the world in light of other people's trading behavior, eventually will reach a point where it's trading around what the actual fair value is in the market that we run. We have a way of making sure that it ends up trading around the true value. Which is by revealing that hidden information into the market.

Patrick: Mm-hmm.

Ricki: Halfway through trading, we have people start getting up to the board and writing down the number of siblings that they have.

Patrick: Mm-hmm.

Ricki: So we have a growing ledger of, "my name's Alice, and this is how many siblings I have, and my name's Bob, and this is how many siblings I have," etc., etc.

Patrick: Mm-hmm.

Ricki: Such that before trading even closes, everyone has written up how many siblings they have. And the market is by now actually adjusted to be centered on number of siblings. That is the true sum of the number of siblings.

Patrick: Mm-hmm.

Ricki: And we get to see as more and more information gets revealed, the market move closer and closer to that fair value, sometimes moving faster than others, depending on how big a surprise new information was. And we'll get to see whether the person who was buying and buying and buying indeed had a lot of siblings, or if they were buying and buying and buying and then write down a zero and they have zero siblings. Is this because they were just confused? Is this because their hidden information wasn't their number of siblings, but somebody else's number of siblings? How do we reason about that?

So we'll alternate between first taking turns trading, then having trading in this open outcry fashion. [Patrick notes: “Open outcry” is a term of art in trading; it means that anyone is allowed to place a trade at any time by shouting it out, and trades “cross” essentially automatically if that should happen based on the current state of the book. This was how essentially all stock and commodities trading used to happen at places like e.g. the Chicago Merchantile Exchange, before we had computers which made this process much more rational. (It took a long time for some markets.) The process is a form of ordered chaos.]

And then having open outcry trading while people's numbers of siblings are getting added to the board and taking pauses for a moment and discussing what just happened, what trade did someone do, and what do we think it means about their state of beliefs? I'll very often ask somebody to explain what somebody else believes about the world as a result of their trading behavior. What can you infer about what's going on in their head? Where at the beginning, this is literally just about exercising the muscle of "what do different words mean? What do different directions mean?" If somebody put out a bid for a price of 20. Tell me, what does that person believe? "Oh, it means they believe that there are more than 20 siblings." Okay, now somebody else put out an offer at a price of 22. What does that mean? "It means they think it's less than 22." Doesn't mean they also think it's more than 20, is their choice to not sell to the person with the 20 bid. Also evidence about their beliefs. Why or why not? And we'll do a lot of Socratic reasoning of that form as the class progresses.

Patrick: I would think on a first order consequential level, after you get up to the board and give away your hidden information, you should trade less. Does that actually happen in real life or no?

Ricki: Yeah. So often that will be a thing that we discuss through example, where somebody will get up to the board and right before writing down their number might do something like adding an order to the order book, and then they'll write down their number and then sometimes someone will trade with them and we'll pause and we'll say.

"Hey Joe, what caused you to want to add an order to the book right before writing down your number?" And Joe will say, "Oh, well it's because I knew my number was really big, so I wanted to add a bid to the order book."

Patrick: Mm-hmm.

Ricki: And then we'll get to go through these steps of, "Okay, Joe, so you knew that your number was, let's say you had 12 siblings. So you added a bid to the order book 'cause you thought it was higher. What happens in worlds where once you write down your 12 people see that number and then they still choose to sell to you? What does that mean about their state of beliefs?"

"Well, maybe it means that they were already modeling you as having a lot of siblings."

Patrick: Mm-hmm.

Ricki: "Such that when you wrote down your number, I guess 12 is kind of comically large. Yeah. But let's say, let's say Joe had written down a four, maybe they had thought Joe has. Seven siblings and then Joe writes down a four. Their willingness to sell to you is actually a sign that wherever their model was at, it had incorporated information about you such that now you're giving even more information. And causing people to want to trade with you. You should maybe regret having those orders out there."

Patrick: Mm-hmm.

Ricki: "We talk about in what circumstances do you want to immediately say, I am out, I would like to clear all of my orders from the book and no longer have them posted because I would like nobody else to trade with me." That this is especially important when there's brand new information and information that moves the market a lot, and you don't necessarily think you're going to be the first person to do all of the necessary arithmetic to get you to the right answer, but instead cases where. You want to be extra cautious, get rid of your orders from the book and then figure out, "Okay, this brand new information came out. It was a pretty big update. What direction should that move the market? And approximately by how much? And in light of that and where things were a second ago and where they are now, what do I want to do?" Do I want to go in that direction that all of these steps can happen before you have summed up every single line item of looking at all of the numbers that came before. And in fact, if you wait around until the point that you've done all the addition perfectly, you're likely to have missed the opportunity to do those trades.

Patrick: Can I highlight a learning here that people encounter 30 minutes into their trading career which nonetheless, evades professional financial journalists, regulators, and members of Congress. The, I'm out—pushing the Madagascar button and canceling your orders that are on the book is actually quite common in market making.

And various professional journalists and regulators and members of Congress believe that things like 97% order cancel rate means that the liquidity we're providing isn't real is a reasonable model for the world. This is straight up incorrect.

[Patrick notes: I try not to be political, but to prove I am not debating a strawman: Hillary Clinton proposed a tax on canceled orders as part of her presidential platform. Michael Lewis discusses what he calls “phantom liquidity” frequently in Flash Boys, which is incorrectly shelved in the non-fiction section of your local bookstore. The Senate Committee on Banking, Housing, and Urban Affairs had several senators demonstrate this misconception in a hearing. And so on.] 

Patrick continues: One way to learn this playing a toy game where obviously you're not trying to manipulate the market and no one is being particularly invidious. Another way is: just try to make a market maker for the first time in your life and do the most naive thing possible, you're going to end up with a 90%+ cancel rate. (Says the guy who wrote a naive market maker once in his life.)

So you learn real things that have real applicability almost immediately. And apparently subpar ways to get that knowledge are getting elected to Congress and being on the committee for regulating the people that do this full time.

Ricki: Sounds like maybe there's a demographic I could market this class to some.

Patrick: Yeah, they might be a little bit busy, but who knows? They should wonder: Conditional on you marketing this, is there an adverse selection effect to them? Sorry, I'm joking.

Ricki: I think that one especially interesting lesson that people sometimes get toward the end of this class is that when we go around and talk about what models informed us, everyone has this cached number of, let's say it's 2.3, 2.1, something like that. How many kids there are in an average household.

Patrick: Mm-hmm.

Ricki: And since we're counting how many siblings you have, you subtract one from that and get to 1.3 or 1.1 or whatever it is. At that point, I'll typically poll the room and I'll say, "Okay, who in this room is from a household with five children in it?" And Joe, who has four siblings, will raise his hand. "Okay. Who's from a household with four children? What about three children? What about two? What about one? Okay. Raise your hand if you're from a household with zero children." Nobody will raise their hand.

And at this point. I think people will sometimes make the connection that the average household size incorporates households that have zero children in them.

Patrick: Mm-hmm.

Ricki: But the average number of siblings that a person might have is not going to be a number that incorporates households of zero size.

Patrick: Mm-hmm.

Ricki: Because those people are not going to be represented in the room of people you're trading with on the sum of number of siblings. If you took all the people in the world and divided them up into different rooms, a family that has five children in it has those five children represented in five different rooms, whereas a family with one child has that child represented only once.

Patrick: Mm-hmm.

Ricki: This point will cause students to realize there was something wrong with my model. That caused me to go from 2.3 to 1.3 by just subtracting one for a person in a family. But wrong in what direction and buy how much.

Ricki: Nobody I've met so far has been able to instantaneously get the math right. And immediately figure out how to convert from their cached 2.3 number to a number of siblings. And the point I'm trying to make at this point in the lecture is not that you need to know how to do that math in order to do good trades.

Patrick: Mm-hmm.

Ricki: It's instead, you need to be able to think what direction should this affect my model?

Patrick: Mm-hmm.

Ricki: Does this mean that the average person in this room will have more than 1.3 siblings or fewer? And the answer here is…

Patrick: More.

Ricki: And by about how much, how important is this factor? Is this something I can throw away or is this something that is a major contributor? And how do I approximate about how much in order to reason about it. Because from there you can either say, "I'm just going to ignore this fact. It doesn't matter that much," or "It kind of matters. 'cause it means that if I think half the people are from one child households and half the people are from two child households, and those two child households will exist twice as often. That actually is going to be a slightly bigger factor."

Patrick: Mm-hmm.

Ricki: And it'll help you realize "What mistakes are other people making." Other people are going to underestimate how many siblings people have based on using that cached. 2.3 number from which they subtract one. And if they're going to be making that mistake. Directionally, even if I don't have much special information, if I've noticed this phenomenon I think that people are going to under model the number of siblings such that on the margin I would be happy buying from the best offer in this market. Even if I don't know much more, and even if the number of siblings I have is one or two, a typical number of siblings for someone to have,

Patrick: You're helping people understand the speed premium here, where your trades are happening throughout this entire discussion, presumably. Right. Or minus some very small periods. And you can go down to your notebook and start plotting out scenarios of like, "Okay, I can't do the calculus in my head, but let me sketch a hypothetical curve for the US population and see how poorly lamb calibrated by." Or you could say, "Alright, I goofed. Perhaps many of us goofed and we all underestimated.” Somebody is going to get to the, We Are All Underestimating realization first, and maybe the only thing that you need to know is a lot of us were underestimating. 

So the price is wrong in that direction and get in front of four other people making increasingly good realizations that I underestimated by 0.3 versus 1.2 versus etc., etc. But all those trades will go in the up direction rather than the down direction. And maybe the second person to be like, "Oh, wait, he answered that so fast." Even in a room that is heavily selected for math ability, there's no way he has anything more than "we all goofed" or "many of us goofed. Maybe I don't necessarily need to update that much on. He's placing a trade that has one bit of useful information in it. But is not that he has successfully picked out the number maybe." I'm also willing to make the trade that we goofed and maybe we goofed a little bit more than people think is accurate.

I think my prior with a minute to think of it is it probably matters less than people think it does, but haven't done the math on that yet. And I'm interested in whether actually doing the math helps you or not in that circumstance versus simply trading the market in front of you.

Ricki: Yeah. I think that there are a lot of ways in which understanding who the other people are in the room, who you're trading against, and how sophisticated your models are relative to theirs is a way more important thing to answer quickly than getting your models to be slightly more sophisticated.

Patrick: Mm-hmm.

Ricki: 'cause if your models are better than theirs, even only by some small margin, or maybe especially if only by some small margin, trading quickly on them is pretty valuable.

Patrick: Mm-hmm.

Ricki: If your models are worse. Holding on until you've understood things better. But by default, not trading is probably going to be a better idea. Maybe the thing you want to do with your extra resources is do some more research and find out, whether it be a math problem or asking other people or something else. And as a result of that, where do you want to focus your improvements in your model, or which kinds of trades you want to do, how much you want to do them, etc., can be pretty valuable.

Takeaways from correctly modeling the pool of people around you. This is a pretty big part of Trader Brain. Figuring out "What are the important things that I'm noticing, and approximately how important are they? What direction do they push me in? And am I in a pool of people who also understand this thing? And if so, how much have their models already incorporated it? Are they getting there now? Are they going to get there in a minute from now? What kinds of trades do I want to do because of that?" And finally. "If I'm only now realizing that there's this interesting mathematical effect of translating from average household size to number of siblings, I am so sure there are other effects that I'm not realizing that come into play here. Whether they be the fact that the demographics of people in this room are not randomly selected from US households or from worldwide households. Whether it be because the population rates, like at the birth point of me learning the 2.3 cached number were different from the birth ages of the set of people in this room right now based not just on country of origin, but also age and also socioeconomic status, stuff like that. Whether it be, the fact that I'm even being presented with this market is itself an update about the world because Ricki might not have a normal number of siblings 'cause she's choosing to give me a market on number of siblings. Maybe there's some kind of gotcha. Maybe there's some kind of extra piece of information here, whether it be her number of siblings or whether it be features to this market themselves."

Patrick: I'm very, when you are faced, I'm very interested whether anyone does, "I have just been surprised. I'm not exactly sure how I can work through this surprise, but contingent on me being surprised in this lesson once, I think there might be another surprise coming. I want to find out that and have a strategy ready to trade that versus trying to figure out what this first surprise means for me."

Ricki: Yep. And I think there's an interesting thing here with the number of siblings where it's a lot easier to be surprised to the upside than surprised to the downside.

Patrick: Mm-hmm.

Ricki: I think sometimes people will say, "Oh, well I think the average number of siblings is actually higher. 'cause there could be things, there could be people in the class with a lot of siblings. And not people with a small number." And I'll say, "Look, the fact that there are households with a lot of siblings, that's already going to be incorporated into your cached average number."

Patrick: Mm-hmm.

Ricki: "That number already knows that you don't have households with negative size, but you do have a few households with large size." But the point that surprises can happen in one direction much more heavily than they can happen in the other. Is a pretty good point and the kind of point that allows you to think about "if I'm in an environment that is more likely than random to have surprises in it and those surprises have a skew as to what direction they can go in. Then maybe I should be adjusting my expected value up."

Patrick: Mm-hmm.

Ricki: I think in general it's a pretty important thing to know about not just financial markets, but all opportunities that these set of opportunities you're presented with are not given to you from a random distribution of all possible questions or markets or opportunities that the world could face you with.

Patrick: Mm-hmm.

Ricki: Somebody else has either through deliberate choice on their part or through all sorts of selection effects, whether they be evolution or capitalism or something else. Decided to, for some strong definition of decided there, give you those opportunities because of some way in which your interaction with them. Having some kind of benefit. In my case, that benefit is pedagogical.

Patrick: Mm-hmm.

Ricki: I am not personally trying to pick off my students. I don't trade in this market. If I traded in this market, I would be at a huge advantage. 'cause I've seen now a tremendous data set of people in their first intro to trading class and how many siblings they typically have and what kind of mistakes they typically make. But I am trying to teach certain lessons and therefore choosing markets based on which markets will most effectively teach those lessons. And that's going to be true whether I'm trying to teach classes on adverse selection and nail it into people that they should be scared to trade or teach classes on all the different wonderful sources of expectancy that do exist in markets. And inserting more liquidity in the form of asking one of my co-instructors to flip a coin and trade randomly with one side of the market at the point where what the market needs is more liquidity. And I want to incentivize students to enter the markets more and not less.

Patrick: Mm-hmm.

Ricki: A lot of the pedagogical choices I make are reactive to what that set of students specifically needs. If I have a set of students who are especially excited about engaging in a bunch of degenerate coin flip based trades with one another about Aping for size, as I've recently learned from them about other strategies of that form, I'm a lot more likely to show them ways in which markets can be unfriendly to them and ways in which they should be more thoughtful and diligent and conscientious of what they do. If they're too timid and too unwilling to approach the markets, I'm more likely to give them markets that do have more opportunities in them that they can approach. And this is going to be a lot more stark from me as a pedagogue who's trying to design classes in ways that will cause them to learn those lessons. But things like this will even happen in markets themselves. If somebody is a really sophisticated counterparty, they're going to get less presented with opportunities to trade and take money from others. 'cause others will be more scared of them and less likely to want to trade with them. If somebody is willing to trade in a way that gives away some amount of expected value to others in order to just attain a certain position and go home, other counterparties will be more excited to want to trade with them. And likewise.

Patrick: Yeah. And happens in the real market as well. And there's as we've said a lot of the time, there's the level one realization, level two realization, and then as many layers down the rabbit hole as one wants to go. The classic behavior in options market was true statement that continues to be true. Retail does not have an advantage against institutional order flow and options or inequities for that matter. And therefore there are companies that would buy the order flow of brokers that were selling options to retail because you could hedge, etc., etc. That you could offer retail better pricing than you would offer institutional. And then WallStreetBets happened along. And you might think the existence of WallStreetBets does not cause retail orders to get much more efficient relative to true market prices. But at WallStreetBets did do is that certain brokers started to have much more skewed books on certain options than one would've expected. And then intuitions or models that the internalizers and other professional market participants had for the range of outcomes that could happen in options suddenly stopped working in that. They were getting their own version of adversely selected where people were aping in zero DTE, yada, yada yada.

[Patrick notes: DTE means “days to expiry” (of, particularly, an option). An option which has a while to go to expiry will tend to have a lot of its value be the time factor (called “delta”), and it will be less sensitive to changes in the underlying as a percent of its price than an option with lower delta. A zero DTE option anywhere near the money will notoriously bounce around quite a bit. They are a product for gamblers and people who make money enabling gamblers. (Robinhood is extremely aware of this.)]

Patrick continues: And in the cases where they were descriptively wrong about how long people could be irrational about GameStop before they went insolvent, unfortunate things happened to them as a result of that.

So, we've talked a lot about the one hypothetical market and then as being. The, well, less hypothetical, the first one, alpha dropped on the podcast. But as time goes to infinity, people are getting more concepts added to their toolkits and the design of the markets is starting to more closely approximate markets that exist in real life, right?

Ricki: Yeah, that's right. So the thing that we've learned most starkly over the past year is however simple we think we've built it, we should probably make it even simpler. At least to start with, in order to cause people to really learn trading concepts in their bones, you want to start with an extraordinarily simple market, like one stock, or one contract that they're trading on.

Patrick: Mm-hmm.

Ricki: With extremely well-defined settlement criteria and a well-defined set of behaviors during a period of time. And. Make things even easier. Every single order should have the name of the person on that order. You should have a trade log that shows who traded when you should have people's positions be easily visible, etc. Because at the beginning, trading is super confusing. And however much in our heads as the instructors designing this curriculum material, a certain lesson might be clear. It's not going to be clear yet to the people who are encountering it for the first time.

Patrick: And this is why we don't use real world instruments like say Bank of America stock, etc., etc., because there are people whose jobs for their entire career is just subsuming themselves in the business that is without loss of generality Bank of America. And rather than spending the entire four days on correcting people's model of how a bank makes money you want to spend the entire four days on correcting their understanding of how people's mental model of the world informs their trading actions.

Ricki: That's exactly right. I think a lot of people come in thinking that they're going to get a really good lesson in how to analyze these specific stocks returns, and how to understand which items in the market to buy. And I'm always like, "Look buddy, I'm not here giving you investment advice. I don't have much alpha over the rest of the market in terms of which stocks you should buy and hold in your portfolio. I'm here to teach you something pretty fundamental about how traders think and we are going to be doing kind of the most basic things, getting reps in of what it is like to trade those the same way you would going to the gym and lifting weights or just practicing the same skill again and again, and making it slightly harder each time."

So we'll build up first starting with an extremely simple market and then adding one new character to that market. Maybe that character will be Mark, the market maker who has orders out on the order book at all times that are spaced the same amount, that have the exact same size as each other. That follow a very simple formula for what the center of that market is. Such that if you buy a bunch from Mark market maker, he's going to come in with higher bids. And once you've traded with some of his offers, he's not going to replace those offers such that the center of the market will move around per the way in which your trading behavior with him pushes his center of his market.

And then we will add in Bob, the coin flipper, a retail trader who bobs back and forth between the bid and the offer and trades with each one. And then etc., etc., etc.

And we'll start to establish to people what are the roles of these different characters in the market? How are each of them making or losing money? On what time horizons are they thinking? One of them might be a broker dealer who's executing an order of a certain size on behalf of a client of theirs. What is it that they're going to try to do when they're executing that order? Are they going to do it all at once in one fell swoop at some point in this trading scenario? Are they going to do it incrementally? Can we identify their orders by looking at what times they send those orders? Can we identify them by looking at the size of those orders, by other things about their trading behavior? How can we go about figuring out who's who in this economy? Who's who in this financial ecosystem, and what is it that we want to do in light of the sophistication and goals of the different counterparties that we're trading against?

[Patrick notes: The Stockfighter simulation also had a population of bots on each level running very different pre-programmed strategies, and it was interesting seeing how players became aware of the truth of that, both before and after they had successfully deanonymized trades (a security engineering subchallenge in our final level). 

Patrick: Mm-hmm.

Ricki: And. Like I said, we just keep learning that we want to make this simpler. It used to be that we would start out trading with all of their identities hidden all of the information about who they are and their size obscured all the different characters trading at the same time, and try to teach lessons like, "Well, you can identify that this person must be the one who's executing as a hedge fund because they wouldn't have done this if not for this, that, and the other, etc." Way too confusing. Instead, start out showing all of their names, all of their information. Start out with fewer characters and as you add them, show their names and their positions and their information. Have a UI that looks a lot like what you would ideally want your student UI to end up looking like. That shows all of the positions and moves of each of the different characters involved, including their moves and what they're doing over time, and then. You can demonstrate what are the traits of each of these different characters and what are the behaviors that they're exhibiting so that once you start anonymizing some of that information, once you start moving it closer to looking like what the US equities exchange might look like, you'll have the ability to identify certain distinguishing features that would be really hard for people to just produce on their own if thrown immediately into that environment.

Patrick: This is the thing we did at Starfighter as well. The game I made a number of years ago, and one of the things we did for these characters is both for pedagogic reasons, start by being very explicit on "this character is in the market." And, maybe amping up the characteristics of them far more than you would see in real life. So, we had two Titanic hedge funds doing battle in one scenario. And one of 'em was always buying ridiculous size at ridiculously far from the market. And they all the other was always selling ridiculous size at ridiculous far from the market. Both to make it really clear to the player that the fluctuations they're seeing are coming from actors A and B and nobody else or only so much as they bounce off of A and B. And two to suggest to people just the fact of you knowing the maximally legible version of this that there is a price and sensitive person who is buying in tremendous size, etc., etc., is not enough to immediately orient all of your thinking around that one fact. But you have to use what else you see to say, "Okay. If I wanted to take advantage of that, what would I be able to do, which would not get me blown up by risk limits and the other things that we had in that part of the simulation." But and so, presumably after you've had some opportunity to do the table, talk with people and say, "Okay, because this person is doing X, we see X and Y and Z in the order book," then you can take away the identifiers and the heuristics that make that obvious and just hope that people retain the sense in their fingers that, "Okay, the market is moving in this fashion. I've seen that before. And then, or maybe 60% confident. I've seen that before. If I'm not confident how much should I be updating my behavior based on that?"

Ricki: Yep. Yeah, I think that's exactly right. I think causing people to be able to identify certain patterns in the market because they've actually seen those patterns themselves goes a huge ways. I think that we just keep seeing time and again, that nothing works, at least for teaching trading, like causing people to actually trade and to notice certain patterns and certain behaviors to have certain muscle memory in their fingers of what different kinds of trades look like and that that is going to get them so much closer to the goal of being able to do good trades when the opportunities actually arise than any amount of trying to reason from first principles or any amount of trying to memorize the full range of what kinds of trading behaviors could happen.

Patrick: The things that trivially falls out of this model, which I think you mentioned last year, is that the people who come in with the best approximation of the number of people in the room, even if they bingo it directly, are not necessarily the people who make the most money trading, right?

Ricki: Yep, that's correct. I think a lot of it will be about a sense of savvy about not just having a good model, but having a good sense of what the models of the other people in the room are like, and at what points your models are more sophisticated, at what points, who's moving, how quickly, etc. Which will often be a pretty good way of getting people to be the ones who are making better trades because they understand who they're trading against, and that for any trade that happens, there's somebody on the other side of it that makes it go right. It's not just who wrote down the best prediction at the beginning of class of what the eventual number would be. It's who did the best trades. And who got to actually profit on the difference between the trades they did manage to execute, and other people being willing to trade with them as a result and the final settlement value.

One thing that we will often do. When it comes to helping people improve their trading and their models, is when they come up with a model of how the world works, similar to what you just mentioned in Starfighter. We'll want to see what does that look like in the extreme case? If their model actually makes sense. Is it true that as you push it toward the extreme, as you say, "Okay, well if in fact the price is going to move this many clips for every additional share that someone buys based on the market maker's function, is it in fact the case that if you buy 10 times as many of those shares, then the price will move 10 times as many clips upwards as a result of that behavior." Hmm. Let's actually do that. Let's type in those numbers and see what happens. And this is one of the reasons I love working with play money here is we can actually test these hypotheses. "Okay, you just typed in a contract size that is 10 times as large, but the market didn't move 10 times as many clips upwards as you were expecting. Why that happen? Is it that the equation looks different? Oh, is it that you ran into capital limits? What does that look like, etc." And can actually make people produce hypotheses. Figure out what kinds of extreme cases will best either falsify those hypotheses or corroborate them and allow them to keep seeking more information and then just directly go forward and test it. And this will ingrain those ideas much more clearly in people's heads than any other method that we've been able to come up with so far.

Patrick: So a thing that happens over and over again in financial markets, despite any number of smart people writing code, heuristics, procedures, etc., to avoid it is the fat finger problem where the order that the trader thought they were executing and the order that they physically typed into the system we're not quite the same order.

Ricki: Oh, yeah.

Patrick: What happens for fat fingers when they happen in your simulation? Yeah.

Ricki: Another reason I'm thrilled that we are using a play currency.

Patrick: Yeah.

Ricki: Those orders are indistinguishable to our exchange, to our electronic trading system from the orders that are intentional and therefore they go through. The thing that we do about the fact that fat fingers exist is we make sure that someone isn't in a situation where one fat finger is going to lose them their entire clips balance and take them out of commission for the rest of the trading session.

Patrick: Mm-hmm.

Ricki: So we will put capital limits on people during our trading scenarios where we essentially say, "Instead of trading out of your normal account, you are going to trade together with a team. Each of you will seed your team account with a certain number of clips from your personal balance. If you want, you can put more in and you'll just take out a a fraction of clips at the end of this trading round that is proportional to how much money you've seeded it with. But by default, we try to encourage everyone to put in the same amount." And that way if you lo and behold hit a fat finger mistake and lose all of your money and your team's money, at least you are, aren't all bankrupt for the rest of the entire trading camp long.

Patrick: Mm-hmm.

Ricki: This also helps people understand better parameters around capital constraints, which we haven't gone into at all yet. But under what circumstances do you want to be. Paying attention to how much money you're using in different markets where all of the trades you're doing might be positive, but having certain orders on the book might tie up your ability to go and perform other trades in other places and therefore might not be worth it for the expected profit. You'd be able to get there.

But even going back for a moment, talking about the ways in which these risk limits for people will prevent them from just having one fat finger error blow things for everyone. A case that we often see is one of our teams will have somebody make some kind of error, whether it be a fat finger error or a reasoning error or decide they want to do something in their self-interest that is in in the interest of everyone on the team. And other people on the team will be upset about this and they'll be like, "Hey, this other person did trades that caused me to lose money as a result." And we will get to do the very fun emotional processing activity of arbitrating, what it is that should happen as a result of this. And this is one of many different ways in which one of the things that we're trying to teach at Trading Bootcamp isn't just how to do good trades as an individual, but what it means to do good trades as a team. What it means to work together with others through the difficult parts, like somebody makes a mistake and you've all collectively lost money to the good parts. Like two people trading together can be even better than additive because there's multiple different things you can see at once. And when there's just one market with super simple parameters at the very beginning, it might not be obvious why trading with other people is important. But once we've entered situations that have 12 different markets at the same time, and no one person can focus on everything, having a team working together is pretty good.

Patrick: I'd love to just dig in a little bit on the team dynamics and communication styles and similar that I assume people pick up as a result of this. Because like we said, when it's solo player mode on the most trivial market imaginable, there's still a speed premium for getting ahead of other people on realizing things. Is there a premium for, this team just talks better to each other than other teams do?

Ricki: There is such a huge premium on this. Here are some of the places where that comes in. Take for example, what we were talking about in terms of building bots and building arbitrage strategies.

Patrick: Mm-hmm.

Ricki: Last week we ran one of our trading bootcamps here at the Light Haven summer Camp Festival season event. And there were a handful of teams where each of the three people on the team were kind of doing their own thing.

Patrick: Mm-hmm.

Ricki: One of them was building an arbitrage strategy. One of them was building some kind of fundamental model of the world in which you would use the information that your team privately had to infer from that and market behavior what you thought other teams thought and come to some Galaxy Brain conclusion. And one team member was kind of floundering, uncertain what to do and maybe looking through old trade logs but not really being that helpful to toward the team's goals. And I would go around and I would ask teams, "Hey, in the past five minutes, how has your trading been going? Have you made or lost money?" This was often a pretty hard question for people to answer and a pretty good prompt that having some kind of UI that gives you insight into this is useful. "How would you go about figuring this out? Well, I would look at the trades we've done and I would compare it to where the market is, okay, how are you even seeing your positions? What page do you have those on?" etc.

But it would also cause people to realize that very often the the two members of the team would be doing. Trades that had no knowledge of what other members of their team were doing. And as a result, sometimes were based on models that were inconsistent with each other, sometimes effectively looked like trading with themselves, whether it be through actually accidentally executing self trades or through effectively trading with themselves when trading on different markets. That represented the same underlying positions because of the ways in which many of these markets included ETFs or other arbitrable assets. And sometimes would include things where, because they were trading in the same overall account. A person who would've successfully been able to track their own profits and losses, if that was all they were doing, wasn't able to, because trades that their partner was doing would now muddy the waters and make it look as though they had some trading that they had done together. That netted out to a different number from what it is that they would've done individually.

That last problem can be solved through account management, but the first few problems, especially ones where each of you are taking in information about the world and coming to conclusions, but never sharing that information with each other is going to be really, really hard for turning into doing good trades as a team when no one person can look at and see everything and when choices that you're going to make are going to be dependent on what choices other people are making. When you have capital constraints, that means that you can only spend some of your clips in some of those markets. Deciding which markets are more profitable is way more important when you have 12 different markets that you're looking at. Often it'll be the case, whether by our design or by our Knight Capital style bug. One of those markets becomes 10 to 100 times more profitable than other markets. Having everyone on your team be able to quickly become aware of that fact and then focus all of your effort on the strategy that looks at that market instead of each of you just plugging and chugging away at the thing you're doing on your corner of the universe and not paying attention to other people is worth a ton at the times that are most important to be doing the right things.

An exercise that we've recently started developing is "sit down. You're going to work on your bot and your UI and your dashboard and which markets you want to open on the screen, and then you're going to swap and your partner's going to sit down and you are going to have 90 seconds to explain to them what they're looking at, what you've set up your screen to include, and then you're going to go silent. And they have to do all of the trading themselves for the next 10 minutes." And if the end of those 10 minutes, your takeaway is, "How was my partner so stupid? They got this wrong and made us no money, even though I built a great thing," then you are not learning one of the basic concepts of trading here, which is that. Trading with another person means coming up with a system that will maximize for how many dollars you guys make in total. Including in circumstances where you individually don't get 100% of the glory 'cause you're not doing 100% of the work. And will result in you having a setup that is more robust to the different opportunity sets that your group as a whole is going to be exposed to.

Patrick: Mm-hmm.

Ricki: 'cause just like sometimes systems break and accidentally buy the same position multiple times. 'cause there's an infinite loop bug where you have some for loop keep buying the same position or the wifi goes down and all of a sudden your internet is out. Or you meant to type 10 and instead you typed 100 or you switched position and you switched your size and your price numbers, or you got a negative sign where you wish you didn't have one. So too, sometimes your teammate has the flu. And the dashboard that they built the previous day. Still exists, but they don't. Or they're not reachable by phone or they're in the bathroom, or they got confused and overwhelmed, or they're going to look at a different market that's more important, but the market that they were watching would now benefit from someone watching it.

And having a dashboard that is easily legible to a human being. And when I say legible, I mean bright reds and greens that are more bright the more important the thing is, and large numbers where there should be large numbers and don't require parsing a whole bunch of text and understanding each and every line of that text. 'cause humans are really bad at reading paragraphs and paragraphs of words. But really legible and really good at drawing your attention where it's most important to have it. You're setting not just yourself, but your team as a whole up for success. Way, way better than just building something that makes sense in your head, or to your best approximation of what your code now says.

Patrick: That would be, I would bet that some of the best team is by the—is it a consistent team over the few days or do people keep rotating?

Ricki: We switch things up the final day, you're with the same team all day. So up until then, you've typically done these two or three hour exercises with the same team, and then you rotate. We want people to get the sense of what it's like to work with different people, and by the final day we throw you on one team and kind of keep you on that team the entire day.

Patrick: Got it. So I imagine that many of the successful teams on the final day are ones that come up with a visual language of style guide to their tools, some level of consistency, maybe even some level of "we have a shared our goal around what we're doing right now. This emoji always means, much more attention here," yada, yada.

Ricki: Yep. Exactly. Some kind of effect. And these are not huge teams. These are going to be like two, three, maybe four people. So it's not as though we need a hyper sophisticated thing that total strangers could plug into, but having a system where everyone on the team can quickly and efficiently communicate with each other what's going on, and why is that important and where is it that we should focus attention and what to do when this dashboard looks green, what buttons to press as a result of that can be really valuable.

I think the one other lesson that teams often learn pretty well here is the value of redundancy, the value of having multiple different parts of the system that apparently seem to have the same goal. A claim for which I very often get the pushback of, "Oh, well those should just be the same value, so why is it that you would need to do that computation twice? That's a waste of resources." I think that redundancy is your friend in a bunch of different ways. One of the obvious ways is sometimes one of your setups will fail. And then having another setup is good, which is why in addition to having a wifi setup, you probably also want an ethernet cord, and you probably also want mobile hotspots that you can plug in all of a sudden, because when your wifi goes down, you better believe a bunch of people's wifi went down and a bunch of the trading is really good. All of a sudden.

Patrick: This sounds awkwardly specific. Did wifi actually go down at one of these events?

Ricki: Wifi has gone down at many of these events. Yeah. Not limited to, but definitely including last week's event at a certain point in time. But you also see this when it comes to how to estimate a number quickly.

Patrick: Mm-hmm.

Ricki: A very common thing that will happen is for a number that you need, let's say not even a precise thing, but an order of magnitude thing. Whoever speaks up first will just anchor everybody else to that number, and now we're all stuck with that same belief, even if we would've produced other things. At Trading Bootcamp at Arbor, the company I run, we have a norm that anytime someone's going to say a number, whether it be in writing or out loud, we basically censor that number out, give everybody 10 seconds to produce their best guess. And then everyone says their best guess. And unless we have better reason to, to do otherwise, we'll typically just go with the median.

Patrick: Mm-hmm.

Ricki: This will be great because most of the time we have numbers that are within 50% of one another. Sometimes we will produce numbers that are orders of magnitude apart, and this will be a good prompt that we need to stop and think for a moment, "What is it in our models that resulted in us getting at such different answers." If we have, let's say, multiple different ways of measuring how profitable a certain market is, where one is like, I don't know how many dollars or clips in our case were traded in that market in the past 20 minutes, and another is how many different people are trading in that market, and another is how many shares have traded in that market, etc.

Then at a time that. There's some kind of mistake like "hmm, our metric for how many people are trading in a market was dependent on having a list of the accounts trading in the market. But now all of a sudden the accounts trading in the market have changed because the account monitoring system changed. So that number is now showing, N/A on our dashboard, at least we still have other systems set up." If we are doing something that requires dividing by a number that we think can't possibly be zero. 'cause we thought prices must always be positive, but now it turns out prices are zero 'cause we forgot what thing we were measuring and got something wrong. We want a dashboard that doesn't just start blinking Infinity or NaN, but still has some sort of numbers we can anchor ourselves to. So having multiple different ways of deriving an answer, of getting to some kind of approximation or heuristic for the thing we care about, can be extra useful for points where those break.

And same thing with people. You want it to be the case that for a given UI you've designed or a trading strategy, if one person gets the flu, someone else can come over and plug in or you can check whether multiple people are able to operate a thing to figure out how good is that thing itself.

Patrick: I'd imagine that in the case where your subsystems are disagreeing with each other, taking notice of the fact that your subsystems are disagreeing with each other, particularly if in recent ticks recent time units in the market, like, "Oh, we went up from these. Basically never disagree to each other, to, they're always disagreeing with each other, to something that has never disagreed once with anybody, is now also in a correlated fashion. Disagreeing." Like, maybe you should be spending a lot more time thinking about what's going on than one would be.

Ricki: Exactly right. Exactly right. I think it's very often there mainly as a catch to make sure we're paying attention when we need to be, because something's different from usual. Something is broken in one of our systems that's causing it to output a very different number from what we'd expect. "Okay. Now we should all take a breath and look at it." Whereas if four people are able to very quickly produce numbers that are approximately the same, any of those numbers would probably be right. The median is probably slightly more right. But whatever, they're all probably fine. But that makes us a lot more confident to go forth and use that number and operate, assuming that that's right. And being able to do things quickly in a way that a trader often needs to instead of having to go through many more painstaking steps. And the more uncorrelated our ways of getting at that answer are. For example, instead of one person just saying an answer and ever being anchored to it, but instead each of us computing it differently and even better through different mechanisms the better, the more likely we are to succeed at getting values that we can feel genuinely more confident in.

Patrick: And of course there's the meta rationality step here of if you have four people who are all coming to approximately the same. The, you know, this is perpetually a resource allocation game where we have four smart brains here that we can put on any problem. If we're all coming up with the same answer all the time, then three of us come up with the same answer as efficiently as four of us coming up with the same answer. And maybe the fourth one should be thinking of something else rather than being ditto in here.

Ricki: This is true. And of course we're always having to make optimization decisions like this when there are trade offs about resources. And at the same time, it is so overwhelmingly the case that I see people error in the direction of not enough redundancy that even hearing you say that, I notice myself flinching and wanting to make a case for why no, you need more brains on this. Because I'm so used to seeing people in the reference class of "I'm so used to being right. I almost always get the right answer. It's a waste of time for me to have to do a computation that someone else is also doing. Why bother?"

I think the other thing that people underappreciate is. Often they think what it means for me to do this equation requires me solving all the math problems to the last decimal place. Whereas what I'm actually looking for often is "what is the number that you just pull out of your butt if you have one second to pull out a number?" Often that number is still a pretty useful piece of information.

Patrick: Mm-hmm.

Ricki: But one other thing I'll say on the topic of getting at that solution via different methods is we have a new tool since maybe last time you and I talked on this podcast basically. Which is asking an AI. We now have another algorithm through which to come up with an answer to a lot of our questions quickly for which the mistakes that a human might make and the mistakes an AI might make. Are definitely correlated. But less correlated than a human and another human sitting next to them. And one of the tools that I will often remind students are in their toolkit when trying to estimate, "Okay, if we're going to roll four D20, what's the minimum value of those going to be? How do we calculate something like that?" You can do something using certain tools you've learned in math class. You can do something by writing a one line Python script to execute a bunch of Monte Carlo simulations. You can do something by just pulling a random number out of your butt and seeing approximately what is it. You can do it by taking a physical die and rolling it and seeing what happens when you try doing that a few times. And now you can also do it by asking a large language model, by asking ChatGPT, or Claude, or whatever model you might want to use, and seeing "are these answers remotely in the same ballpark? If they're not, let's dig into why" often we can make good use of that LLM and find out why from there in a way that ends up helping us.

Patrick: And I immediately start thinking in the lines of, "Okay, the people who are selected into this class know that LLMs are an option for them, even if, I'm pretty sure I have the right answer already. I want to know the answer in LLM gives to someone asking the LLM because I think I want to trade against LLMs a lot, at least as of this point in life."

Ricki: Exactly right. Exactly right. I think learning not just "what methods I can use to make my answer better," but "if I and a few others come up with different answers, it gives us data that we should dig in more and it tells us what kinds of answers the other people in our environment are going to be coming up with," can be extremely useful toward that goal.

Patrick: So, relatively few of us weren't accepted. Probably had alums will be fundamental to craft of trading on the boards a year ago. And I guess you can check manifold records for that too. But what are the other interesting things that as you've iterated on your software and systems for this, for example, you've discovered "Yep. This is really something that we want to put in the toolkit for everyone."

Ricki: Hmm. That's a good question.

Patrick: So you're doing this in person, so the answer is not Zoom or Discord or similar. But are there things that you do to make communication out the box easier for teams?

Ricki: Yeah. So. People have actually asked us a number of times, "When are you going to have an online class? And we'd like to take this class, but we live in Europe." Or "We can only do the evenings and can't do a block of several days in a row." And we've been thinking a lot about what are the best ways to design these classes in ways that will be more scalable and ways that will allow more people to gain access to them. One of these ways might be figuring out how to design these trading scenarios in ways that are basically like trading games in a box that you can just spin up on your own computer and play, that you can open up to our exchange and run with a few friends that don't require us to be in the room.

I'm answering a slightly different question from the one you asked.

Patrick: Sure. Here we are. Nevertheless, this is, by the way, a classic PR training is if you ever get a question you're not prepped for, answer the question you prepped for instead. And so few people catch that in the world.

Ricki: True enough, true enough. I'm always very happy to readily let people know that I'm deliberately sidestepping their question, but we'll come back to it in a minute. I think that we are still trying to figure out how can we do online education or education that doesn't include us in the room in a way that will still achieve a lot of the goals that we have here. I personally feel extremely resistant to having Zoom or other mechanisms like that be the way that we are teaching things. Because it's so important to me to get quality time with students. That might just be selfish 'cause of how much I enjoy it. I love teaching people in person. I love seeing their faces light up and I'm pretty skeptical about a method of teaching people that has everyone only in directing online.

Patrick: Mm-hmm.

Ricki: It's also 'cause a lot of the way that we do teaching trading isn't just the content in the classes themselves, but actually immersing people in an environment of it over the course of several days where they're also eating meals together, making side markets on things, having conversations that overflow late into the night, sitting around the bonfire, talking to each other about trading and life goals and philosophy and everything else on their minds. But I think that. We are at the point where there's enough demand for online trading classes and online trading games. That figuring out how we want to build that out is one of our clear next steps over the coming several months.

Okay, let's go back to the question you actually asked though. So you want to know what sets of tools are people using now in trading or, or be recommending for our students to use in trading? And I think my answer here is not super impressive because a lot of the tools that we really hammer on the head are, "do the more basic thing, not the more sophisticated thing." This idea that people think, "Oh, okay, I need a super fancy advanced LLM model set up in order to have any edge. 'cause the market is so competitive" and there's a sense in which that is true, that the market is really competitive, that the counterparties are trading against or likely to be really sophisticated. And therefore, if you want to have any edge, you're going to need to be doing some multi-factor decomposition, complicated math equations in order to get there. But that there are ways in which learning the basic heuristic do need to come first. Do need to come from having simpler models of the world that you can build those into.

Patrick: Mm-hmm.

Ricki: And can allow you to do trades in environments that are selected for being less competitive, less efficient. The Efficient Markets Hypothesis is that the best markets already have everything squeezed everything out of it. 

"Can I identify which markets still have good trading opportunities for the kinds of trades that I do know how to do, the kinds that I am able to cover in the first few days of a curriculum like this, or through what my comparative advantage is, maybe I'm really, really good at predicting the outcomes of legal cases or something like that. 'cause I just am a huge nerd for reading case law and I have a really good sense of how pharmaceutical companies are going to get ruled on compared to all of the existing prediction markets or financial markets or whatever it may be." Okay. I found my comparative advantage. I have a story for why I would be getting something right, where other people might not be getting it right because I have a high stamina for reading case law and as a result I'm going to find the markets that might have opportunities available for me where I'm actually adding something instead of having to build a hyper sophisticated model. I don't understand in an environment already saturated. With players who are a lot more advanced than I am.

The thing that we're trying to do in our class is less build the most sophisticated advanced models and more build up people's skillset at identifying what different environments look like and whether those are the right environments for them to be engaging with. In the case of many of our students, that does include building really sophisticated models because the comparative advantage that those students have is being really good at a specific kind of math or a specific kind of options pricing or a specific kind of trading strategy, and helping them figure out which environments are places where that trading strategy is most prevalent, where they can go and make good use of that scrappiness, of that math skill set of that computer programming ability.

Patrick: Mm-hmm

Ricki: is a great thing that we can empower people with. But we are not trying to tell them, "Here are the complicated skills that you need to know all of in order to take on the world at the highest profitability places." We are trying to tell them, "Here's how to find where you want to be with the skills you do have, and figure out which skills you're excited about investing in more and getting better at. And sometimes that might be using an LLM to sanity check an answer, but rarely is that going to be us pushing on you. A thing that we predict will be way more complicated than you're going to be able to grasp in the amount of time you're going to realistically throw at it before trying to implement it. 'cause we don't want people to be overconfident to lead themselves astray by chasing down a path toward riches that isn't really there because it has way too many adversaries along the route. And instead give people the ability to use the skill sets they have and the opportunity sets they're presented with to look out for their own interests using trader like skill sets and frameworks.

Patrick: Nice. So on that note, I think this conversation unfortunately has to come to an end somewhere. Ricki, where can people learn more about you and about Arbor?

Ricki: Great. So our trading bootcamp gets run now at once every six weeks or so. You can go to trading.camp in order to find it. And for listeners to this podcast, you can go to trading.camp/complex to get an extra discount in there. In general, Arbor's working on a bunch of projects across markets, pedagogy and game design.

We have a game design conference coming up that will itself be a giant game in September here at Light Haven Campus. That's called Metagame 2025. And you can go to Metagame.games to learn more there. And in general, anyone who wants to reach me can reach out Ricki at trading.camp or come hunt me down in person somewhere. I will talk to you until the sun sets and the cows go home.

Patrick: I'm also going to try to write a game for third or so time in my life to have something to share at Metagame. So, knock on wood, hopefully that our project will go well, but I won't spoil it yet. Alright, well thanks very much for Ricki for coming on the podcast again and for the rest of you, see you next week on Complex Systems.

Ricki: Thanks so much, Patrick. Bye everybody.