We should stop burning pharma trials’ lab notes, with Ruxandra Teslo

This week, Patrick McKenzie (patio11) sits down with Ruxandra Teslo to discuss why drug development keeps getting more expensive and her Common Technical Document Project. They also talk about why it takes courage to leave a well-defined career path to work on systemic problems, and why a single tweet from Tyler Cowen can matter more than you'd think.

Sponsor: Framer

Framer is a design and publishing platform that collapses the toolchain between wireframes and production-ready websites. Design, iterate, and publish in one workspace. Start free at framer.com/design with code COMPLEXSYSTEMS for a free month of Framer Pro.

Timestamps:

(00:00) Intro
(00:56) Challenges in biopharma productivity
(03:12) Understanding clinical development
(04:59) The role of basic science in drug development
(07:39) Clinical development process explained
(09:25) Issues in clinical trials and development
(19:33) The role of information in clinical trials
(20:30) Sponsor: Framer
(21:42) The role of information in clinical trials (continued)
(32:55) Proposed solutions for clinical development
(40:31) Consultant opinions and regulatory documents
(41:28) Streamlining the regulatory process
(43:06) Understanding FDA interactions
(45:35) Building a public library of regulatory documents
(48:18) Encouraging novel approaches in biotech
(50:06) Addressing risk aversion in the industry
(51:52) Analyzing FDA consistency and reviewer heterogeneity
(01:02:15) The importance of courage in professional growth
(01:06:39) Supporting young professionals and catalyzing change
(01:16:14) Wrap

Transcript

Patrick McKenzie: Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does.

Hi everybody. My name is Patrick McKenzie, better known as Patio11 on the internet, and I'm here with Ruxandra Teslo.

Ruxandra Teslo: Hi, and thank you so much for having me on.

Patrick: Thanks very much for coming.

You are a public intellectual and scientist who works in the field that I would broadly call clinical abundance, and I would love to hear some of your thoughts on that. And you've recently announced a new initiative called the Common Technical Document project, and I think that relates to clinical abundance. But I would love to give perhaps a less specialist audience some sort of lay of the terrain first and then say how CTD fits into that.

Challenges in biopharma productivity

Ruxandra: Yeah, thank you. That's a very nice intro. And indeed, I have very recently finished my PhD in genomics at the University of Cambridge Sanger Institute. And in the last few, like maybe one or two years of PhD, I got very, very interested in how to improve biopharma productivity. So basically the amount of drugs that we get per dollar of we invested in research and development. And maybe some of the listeners have heard about this, but in biopharma we have something called Eroom's Law. [Patrick notes: Of much interest in the field.]

So that is the sort of Moore's Law in reverse. [Patrick notes: While reviewing the transcript I suddenly understood that this was not simply a researcher named Eroom being surprised that results outside the field failed to replicate within it.]

So unlike in semiconductors, we are getting worse at producing drugs per amount of dollar invested in R&D. So basically the productivity of the biopharma enterprise is getting worse. It's gotten worse in the last six decades. There's been a plateauing in the last 10 years or so.

Patrick: And just to give people some color on that, my understanding is it takes about a billion dollars to successfully get a drug to market these days. Right? That's the rough order of magnitude. [Patrick notes: See previous episode with Ross Rheingans-Yoo for an explanation accessible to non-specialists on how we count  that high.]

Ruxandra: Yeah, that's correct. So basically, you know, there are various estimates. Some people say 1 billion, some people say 1.5, some people say two. But that's basically the right order of magnitude. That creates a lot of issues in the biopharma industry because it creates a lot of misaligned incentives because you have very few shots on goal, so to speak, because of how expensive drug development programs are. Although the 1 billion figure also accounts for failures because most drug development programs fail.

The 1 billion figure accounts for failures as well. So if you just take successful drugs, it's lower, but it's still quite high because the successful drugs would've been taken through all of the clinical development steps.

Patrick: And for people who are non-specialists, failure can mean a lot of things in a lot of fields. When a drug fails, that means somewhere along the de-risking process prior to it being given to patients in the phase one trial, phase two trial, et cetera—perhaps it wasn't effective. OK, it actually remediated the condition, perhaps. Well, it works great in a Petri dish, but if you put it in a person that kills them. Et cetera, et cetera. There's many ways that it could fail and failure is the default, even when everything is right, basically.

Ruxandra: Yeah, exactly. And I became very interested in how we can improve the productivity of the biopharma industry and why it's getting lower, despite the fact that we've had enormous advances in basic science.

Understanding clinical development

So just in the last 30 years, we have so many new modalities, for example, in biology. So modality is a way to attack biology. So in the past we only had small molecules, so most drugs were part of this category called small molecules. We now have peptides, which people must be familiar with from GLP-1 agonists. We have antibodies, we have checkpoint inhibitors, we have CAR-T cells, which are the type of immunotherapy for cancer. Those are little cells that we modify. We have gene therapies. So we really had an explosion of sort of technical capabilities. And given this, it's somewhat surprising that we don't have a cornucopia of treatments, and you know, there are various explanations for that.

Patrick: I think it's also, it would be a priori surprising to someone who didn't know the shape of the curve. That knowledge work itself must have gotten more productive in the last 40 years. We have computers now. We are no longer collecting clinical data by having someone write things down on paper by hand and fax it to a principal investigator. Email can happen, international collaboration can happen, et cetera, et cetera. And not only did we not see any empirical benefit from all of that, which should have helped us, even if, you know, science had stood still, we've seen the opposite of improvement.

So I guess that asks the natural question—why? Why are we seeing disimprovement?

Ruxandra: So that's a good question. So basically I think, you know, there's always the sort of "we have picked the low hanging fruit" explanation. So obviously the drugs that we are making have to beat drugs that already exist, and that creates all sorts of problems, including making clinical development harder because you need larger samples of patients.

There are other theories about the fact that yes, we are doing more science and we have all these technical capabilities, but we have really deviated from how we think about the science and whether it is actually applicable and whether it produces practical results. So Jack Scannell, who has talked about this, coined the term "predictive validity." About whether the models that we have for whether drugs work—whether the models that we have before we actually try them in patients—whether they are very accurate at actually predicting what will happen in patients. And it seems like for a long time we kind of deviated from that. We actually had lower predictability of our models. That being said, I don't think we couldn't definitely improve on that side. And there's always room for improvement, but it does seem like we are actually improving and Jack has recently said that, you know, he has noticed that things are improving.

The role of basic science in drug development

So, but I think that a big part of it is something that often gets ignored. So that is the big elephant in the room, which is clinical development. So clinical development and drug discovery encompasses everything that happens once you actually start to develop a drug as a drug and try to get it to test it into patients.

So everything—so basically drug development is imagined as a funnel. So at the top you have all of the hypothesis and all of the basic science, and you try it in all sorts of in vitro systems because you cannot try it in humans because it would be dangerous and even if it wasn't dangerous, you know, humans are limited and so on. So you try a lot of things in vitro and this is where these preclinical models come into place because what you're trying to do with these models is you're trying to emulate what happens in humans. Obviously we're quite bad at that because, you know, all models are either animal models or they're individual cells or organized groups of cells in a dish. But they all have their own, you know, flaws and usually you need to sort of interplay between them to figure out what is good, but they're just not humans.

Clinical development process explained

But after, you know, you have gained some level of confidence through sort of across all these models and so on, you will initiate a phase one trial. So that's the first part of clinical development. And once you do that, you start the interaction with the FDA, and obviously it starts before you actually enter human trials—it starts with presenting these preclinical data to convince the FDA. So basically at that stage where you transition from in vitro models to humans, that's kind of where clinical development starts.

And it ends when—it depends on the disease area. But you know, the classical model of this funnel is that it ends at phase three trials where you show finally efficacy. So phase one and two are safety, and then phase three is a bit of safety and efficacy as well. Sometimes phase one can also be efficacy, but for simplicity it's safety and some efficacy for phase two, and then really trials for approval, where you really show efficacy in phase three. So that's when clinical development stops. And it includes the trials themselves, all of the regulatory interactions around preclinical data, but also all the manufacturing that you do for these trials, which is a bit different from the manufacturing that you do afterwards when you actually commercialize a drug if you get approved.

And this process of clinical development really encompasses—it's really a very, very long process. It takes about 70% of drug development costs, also very long timelines. And it's really a huge bottleneck. And despite the fact that it's such a huge bottleneck, obviously when I saw that this is so important, such a big part of R&D cost and such a big part of the timelines for drug development, I was like, well let's Google, let's see what's happening in this space. Who is writing about how to make this better? What reviews are there? You know, because I was expecting from coming from science, you know, for every topic there's a review, there are papers, there's stuff published on it. And I was surprised by how little is out there about even how clinical development works.

So not almost nothing. And there is some research, there are some papers, but it's very, very scattered. There isn't—you would think that given the importance of this, there's entire institutes dedicated to studying clinical trials, to figuring out how to make trial design better, to figuring out what policy changes we could make. And really there's no such thing.

Patrick: I think that this is a problem almost endemic among lots of knowledge-based institutions where there are no even high-quality ethnographies of how do these people spend every day. And so we are limited to an apprenticeship-based system where the only way to learn how to do clinical development is to sign away X years of your life doing the least productive part of clinical development to get a mental model for how to do the more productive parts. Where that's probably not how we would order the world if we were being rational about how much we want to get an Alzheimer's drug in the next 10 years. But yeah, as you say, you know, a rational world would have university courses devoted to nothing but here's what you can expect when you get into drug trials and those courses don't exist.

There are vanishingly few—there's no "The Social Network" of clinical development where the arc of a movie is just phase one, phase two, phase three trials triumphing at the end. [Patrick notes: The most compelling work of fiction I can remember on this topic has the takeaway “Make sure you are quite protective of the rainforest because you never know which species of ants holds the cure to cancer.” And, accepting for the sake of argument that we have a molecule that is the literal cure to all cancers, that does not necessarily successfully result in a drug to treat cancer, to say nothing of how unlikely a premise that is.] 

There are no TV series about clinical development. There are no pop science Malcolm Gladwell books about clinical development relative to say Michael Lewis dives into various facets of finance. [Patrick notes: And this means no scripts get optioned to turn into Big Short, movie edition.] And thus people we have educated at enormous expense to become research scientists are flying blind when they get into the industry.

Ruxandra: Yeah, 100%. And I think this shocked me. I mean, even within academia, I was in a biology PhD program and before that I studied biochemistry at the good university at Oxford. And I never even heard about, for example, Eroom's Law, about predictive validity, about not even about clinical development as in the process of it, but just about how the science that we do feeds into clinical development and feeds into what you actually care about, which is making a drug that works and how does this relate to the science? And that doesn't mean that I think basic science shouldn't exist. Of course I think it should exist. But I do think that just seeing these worlds as two separate things—the world of basic science and the world of clinical development—is just wrong.

Because at the end of the day, clinical development is just sort of the infrastructure on which you put your drug through so that it produces things in patients. And at the end of the day, a lot of the reasons why we fund biomedical science so much more than other science is because we assume that it will feed into this, it'll make trials work because that's what it means for drugs to work—it means that the clinical process succeeds. So, you know, the fact that we don't discuss about whether science, different branches of science help the success rate per trial, it's weird. So that's one problem. But then the other problem is just the process of clinical development.

Patrick: Yeah. The distinction between basic science and applied science and development was covered in our previous episode with Ben Reinhardt. It is a distinction written into US law, but the US law doesn't cleave reality at the joints. It was largely not written by people who spent 10 years in the lab trying to save lives. And so when you're trying to make something in the world, there is this push and pull where, you know, there are these concrete problems we are trying to solve in lab or solve at a certain portion of the trial.

And sometimes it's not like, well, the state of basic science has advanced as such that I can solve this problem right now. And so rather than saying, well, I guess that kills this $1 billion cost and expectation trial dead, you say: no, we have smart people here, do the basic research to figure it out.

The specific way that that basic research gets captured by scientific establishment and kind of the common knowledge of humanity is different when it occurs in the context of a drug trial than when it occurs in the context of say, basic research that was funded by the NIH in a lab under a grant program. Does that largely track with your understanding?

Ruxandra: Yeah, that's true. I would say that I would also like to distinguish between two things about clinical development. So a lot of people talk about when they hear that I work on making clinical development better, or I try to, they immediately ask me, so are you trying to improve success rates per trial?

And that is a bit of a separate question. I touch upon that incidentally in my work, and I'll explain later how. But that is basically the goal, the implicit goal of basic science, right? So the implicit goal of biomedical science is that it increases the probability that when you put a drug through a trial, that trial will succeed. That's why we hope to learn more biology, right? Because we want to model—we want to put better, higher quality drugs through the clinical development process.

And so many people are asking me, do you know a biotech company that works on improving clinical trial success rates? Because they hear that I work on clinical trials. And to me that's really surprising because my answer is, well, that's what every biotech company should do. That is the implicit reason why you found biotech companies, because you think that you have some technology that will increase your rate of success in trials. It's almost like asking, do you know a bank that lends money?

Right? Like literally that's what they're supposed to do. Like improving success rate per trial, that's what biotech companies should do. And to, what I focus on instead, which is not success rates per trial—it's about the entire clinical development infrastructure and the costs and the timelines of the trials. And I think that improving that infrastructure is the main way we can get more clinical abundance. And I think it's an area which is very understudied and people don't realize how much we can improve it because the process is so poorly understood.

Issues in clinical trials and development

Patrick: And just to give people some numbers, if I recall correctly from your writing, a phase three trial means we're talking about an expectation of hundreds of millions of dollars and many years?

Ruxandra: Yeah, exactly. So trials can vary a lot, but the rough approximation is that a phase three trial can be between like 100 to 500 million, something like that. And it can be anything from like three to seven years or even more. So it's a very, very long and expensive process. And, you know, I remember being—when I started learning about this, I was like, well, okay, this is a very expensive process. But, you know, surely a lot of these costs are just inherent to the nature of what we're doing. Like, you know, you have to recruit patients, you have to monitor them, you have to give them drugs, you have to measure outcomes.

But the more I learned about it, the more I realized that so much of the cost is just like pure waste. Like it's just things that don't need to be done or could be done much more efficiently. And the same with timelines. Like a lot of the delays are just because of inefficiencies in the system, not because of any fundamental constraint.

Patrick: I think the startup world has a useful framing here, which is—and this applies very explicitly to software startups, but I think it applies more broadly—if you model your entire career as having one bet, and your career will succeed or fail based on whether that one bet blows up on the launch pad or not, you get extremely conservative on every possible dimension of that bet. This is true even though blowing up rockets is cheap as long as they don’t have humans in them.

And that is basically the SpaceX story. We'll blow up as many of these things as we need to. In a startup, you might spend one or two years of your career learning, "Oh, okay, that one failed early." Well, I'll get 10 to 15 shots on goal in the course of my career. There's relatively little career risk for having something shut out from under me. And because there's relatively little career risk, my peers won't tell me to stick to the straight and narrow. I won't be judged by people for failing once because you'll develop community knowledge that everybody fails. Yeah, it happens. I personally had a startup shut out from under me in 2015, 2016. Four people in the world remember that fact. [Patrick notes: An exaggeration but not much of one. And, while I think that sometimes startup mythology successfully predicts e.g. VC partner decisions and sometimes it does not, I do perceive that Starfighter made the next venture more fundable than the resume prior to Starfighter, as it demonstrated a missing check box: this person knows what a company that plausibly scales to $100 million in revenue looks like. Of course, almost ten years later, relatively few decisions swing uniquely on that resume line, one would imagine.]

But for the benefit of people who don't know, what is the percentage of your career that you sacrifice if you do a clinical trial and it doesn't work out?

Ruxandra: Yeah, so basically the clinical development process is something like, depends on the area, about 7 to 10 years, maybe more. That's from start to end. What ends up happening is that most junior people don't work on the entirety of it. You often own parts of it. And that sounds as if it decreases the risk, but actually it creates other bad incentives because it means that people often don't have the incentive to see it through and see it succeed. They have an incentive to basically push it to the next stage and show that they have pushed something to the next stage, but it doesn't necessarily mean that it will succeed in the end, right? So that's another disadvantage of long feedback loops.

Obviously there is also a lot of risk aversion for people who coordinate a specific trial because, you know, even if it's not ten years, it's three years, but it's tens or hundreds of millions of dollars. There's also the cost, so decreased risk aversion.

So basically the length and the cost of clinical trials creates two pathologies. On one hand, the risk aversion, especially for people who coordinate individual trials. But also a sort of siloing of different areas within the pharma industry such that each person ends up—it's hard to explain exact details, but basically you have people working on individual things or individual parts, and there's almost—in the end, few people who know everything from preclinical to end of clinical. And you have this splitting, which I described before, which is that people almost sometimes working on the basic science and the blueprint but you even forget that what you're doing is ultimately for the purpose of getting something to succeed in a trial, to show efficacy in a trial. So it's two pathologies that this creates, I would say.

The role of information in clinical trials

Patrick: If I can tell an anecdote that I've told before, I was once in the position that I was supporting a clinical trial that was done at my alma mater. And it was a clinical trial under sort of the worst circumstances. It was during the COVID-19 pandemic, testing the efficacy of a treatment against COVID-19. And so, obviously, we as a society want this to get through the hoops as fast as possible. And because the supply chains for knowledge and clinical trials are so siloed, this trial was, for various reasons, attempting to recruit patients through the internet, which requires someone to click on buttons on a website and sign up to get a drug in the mail.

And the questionnaire to get into the trial to get, you know, either the drug or placebo mailed to you, was 400 questions long. And I asked the PI, "Please help me understand, because you've articulated to me you're not getting enough patients. I look at this website that has 400 questions on it. And I think from my perspective, as someone who does conversion optimization for a living, no one is going to answer 400 questions. But I don't know anything about medicine. I don't know anything about the institutional review board. Can you tell me, what's the constraint that's generating 400 questions?"

And he said something which rounded to, "Oh, the grad student coded up 400 questions." "Oh, great. Why?" "I don't know. Grad student." And I said, "Can we bring her into this meeting and reduce it to the actually necessary set of questions? How many is that, 360?" And I'm thinking 10% improvement off the top. That would be great. It's like, "Oh, we really need four things. Maybe five."

"You're kidding me. Why don't we just delete 395 questions immediately?" "Well, nobody on the team but the grad student speaks PHP." And I'm very loudly thinking to myself, "Are we seriously at a multi-billion dollar research institution in the United States, constrained by not having someone who can press the delete key in PHP?" Yep.

And so, the literal Too Many Cooks horror bit. I'm like, "I don't write PHP. I think I read enough PHP to hit the delete key. But be that as it may, the best man at my wedding definitely knows enough PHP to do that. Let's bring him in here immediately. He can hit the delete key and we can get more patients to do this."

This is one of these pathologies and it is fractal. Also, even the fact that I feel a little askance as someone who is, you know, somewhat involved in this, to say this anecdote out loud because I don't want to get a reputation for betraying the confidence of people, et cetera, et cetera. The cumulative result of institutional omerta is there is some cumulative knowledge base of—in this example, what should you do to recruit patients over the internet—which is not being exposed to people. And as a result, people who should have all the advantages of this is a funded trial, it has gone through approvals, it has been granted the expedited authority by the FDA, et cetera, et cetera—are stumbling over what should be very low stumbling blocks.

[Patrick notes: A genuine strength of the U.S. medical establishment is there is some pathway by which someone whose expertise in conversion optimization comes from slinging bingo cards to elementary schoolteachers successfully ends up in a conversation with a PI at a major research institution. A weakness of the U.S. medical establishment is that it does not possess a deep literature which would bury me in effectiveness. A further weakness is, after being apprised of this gap, I confidently predict that no U.S. med school is going to add my writing on conversion optimization to the curriculum tomorrow. To do that would be to admit a weakness, and institutions regard that as a loathsome necessity which they will retreat to only when forced.] 

Ruxandra: Yeah, exactly. So that's exactly my experience. And that's why it's such a hard problem is because of the siloing and because these inefficiencies accumulate. And each part is known by one person or a few people who work on that specific thing. And as you said, there is this culture of secrecy and it's very hard to professionally be a whistleblower, so to speak, so you just have these things that accumulate. But collectively, they basically clog our ability to make drugs. And we have all this amazing science—it just clogs for these reasons that you've just highlighted.

And maybe that's the perfect place to talk about the information problem in biotech and in clinical trials. And which we're trying to solve with the CTD initiative.

Proposed solutions for clinical development

So basically in trying to tackle clinical trials, there are very many avenues that one can take. And maybe when I started, I thought that, you know, if we just slash regulations and if we just make the FDA allow people to do whatever they want, or closer to what they want, then everything will be fine. And we just need to slash regulations and whatnot. And I'm not saying that there aren't bad regulations, and sometimes they're also intertwining with the law. So it's not as easy as taking down the regulations. There's various things that we can discuss, but something that I hadn't really considered—and again, that's just one angle of improving the process—is that even if you keep regulations constant, one of the big problems in clinical development that I kind of touched upon before is the siloing thing and the lack of information that people have.

Just if I want to study clinical development, as I said, I don't even know what the steps are because it's all done in companies and it's protected under commercial interests, trade secrets. And you can't release it. And it's just individual people knowing each part of the process.

[Patrick notes: A point I desperately wanted to interject live, but didn’t want to interrupt: no, you can’t simply ask the FDA to redact the trade secrets but leave the boring parts. Look at the vast armies of paraprofessionals employed during litigation to do document review, redacting legally privileged communications and excluding non-responsive communications. Industry spends billions of dollars on that line item. Relativity, a software company that just helps manage the process, was worth about $3.6 billion. The FDA’s entire budget is only $7 billion per year, and this out-of-mandate improvement without a specific contingent of Congresscritters and advocates clamoring for it will not successfully get them another billion to play with.]  

Ruxandra continues: The interesting thing that I found out was that actually a lot of this information around clinical development is actually stored in documents. So because you have to present everything to the FDA—all of your preclinical data, all of your trial results, all of your clinical trial design plans and so on—you store all this information. And that information, again, is very valuable because if you are someone that is just starting or is new to clinical development, you again, you don't know anything about how it works.

Patrick: Much of this is encoded in what might be called in other contexts tacit knowledge or process knowledge. In, say, a semiconductor context where semiconductors are their own ball of wax, but, you know, there is some series of steps that you have to do in the laboratory, and maybe there's multiple alternatives for that series of steps. And you have to describe to the FDA in painstaking detail which one you did. And many people have gone through this.

And so if there are six plausible alternatives based on the state of the art of chemistry / biology, very plausibly two of those alternatives are very easy to get through the FDA. They're quite comfortable with them. And the other four will result in multiple back and forths.

And if you're ambivalent on which of the six you use because they're scientifically equivalent, you would sure wish to know which one gets through instantly versus which one is going to add six months to your discovery process. And yet, assuming in this hypothetical example that these are six things that any undergraduate knows about how to synthesize something and put it in a petri dish, there is vanishingly little IP in just—of six equivalent things that are printed in a bio 101 textbook—"use A or B, but not C, D, or F." That doesn't seem like it is valuable IP to anyone. There is no company that will live or die based on—nobody, you know, "our great secret is that the magic way to put agar on a plate is the one on page 37 and not 42."

But because the FDA feels perhaps rightly, that when people file all these documents with us, those include so much trade secret data, we couldn't possibly expose them to the rest of the world. We as a society essentially burn the lab notes every time we go through a $1 billion process to get a drug into play. And so this Common Technical Documents thing that you're proposing is a proposal to not burn the lab notes anymore.

But what is the actual mechanism of that?

Ruxandra: Yeah, that's a very good description. And I would add that, so this is another interesting thing where people think that, you know, the sort of villain in clinical trials is the FDA. It's not the FDA that thinks there are trade secrets. So actually there is a statutory thing. So basically the FDA is required by law to keep confidential all of this data that companies submit to the FDA. And this is in part because of trade secrets. So Congress decided that, you know, companies will not want to submit their data if the FDA will just release it immediately. So therefore we will have this law that says the FDA has to keep it confidential.

Now, in practice, what this means is that all of this incredibly valuable information that could help other companies develop drugs better, that could help academics study clinical development, that could help policy makers understand what's actually happening in clinical trials—all of that is just locked away. And the FDA can't release it even if they wanted to, because it's prohibited by law.

And what we found is that actually a lot of this data is not really trade secret at all. Like it's just the kind of stuff that Patrick was describing—it's just like basic process knowledge. It's like, you know, "we used this formulation," "we used this endpoint," "we designed the trial this way." And that's not really a competitive advantage. That's just knowledge about how to do clinical development. But because everything is bundled together and the FDA can't release any of it, we lose all of this knowledge.

So the Common Technical Document Project is essentially trying to change that. And the way we're trying to do it is by creating a system where companies can voluntarily share parts of their regulatory submissions that don't contain trade secrets. So we're not trying to force companies to share their crown jewels. We're just saying, look, there's a lot of stuff in these documents that is not actually competitively sensitive. It's just process knowledge. And if you share that, it would be hugely beneficial to the entire field.

And the mechanism is that we're working with companies to identify what parts of their submissions they would be comfortable sharing. And then we're working on creating a repository where this information can be stored and made accessible to researchers, to other companies, to policy makers. And the idea is that over time, as more companies contribute, we'll build up this really valuable knowledge base about how clinical development actually works in practice.

Patrick: And I think there's an interesting dynamic here which is that in many industries, there's a lot of shared infrastructure and shared knowledge that everyone benefits from. Like in software, we have open source. We have Stack Overflow. We have all these ways where people share knowledge about how to do things. And everyone benefits from that. And it doesn't mean that there's no competition in software. There's intense competition in software. But the competition is on like, who can execute better? Who can build a better product? Not on who has secret knowledge about: how do you configure a web server?

[Patrick notes: Readers outside the software industry can read “configure a web server” as a deep technical domain that specialists can give rousing 60 minute talks about for particular cases in deployments which scale worldwide, but where the common techniques are boring, well-understood, and exhaustively documented. No company goes to market thinking its web server configuration is the secret sauce that will make them worth a billion dollars.] 

And I think clinical development could benefit enormously from having more of that shared infrastructure and shared knowledge. Where you're not competing on like, do you know the magic incantation to get the FDA to approve your trial design? You're competing on, do you have a good drug? Do you have good science? And I think that would be much better for everyone.

Ruxandra: Exactly. And I think the analogy to software is really good because I think that's exactly what we're trying to create. Like we're trying to create the Stack Overflow of clinical development. And I think, you know, one thing that people sometimes worry about is like, well, if you share this information, won't that help your competitors? And the answer is yes, it will help your competitors. But it will also help you.

Because the thing is, in clinical development, you're not really competing with other companies in the same way that you might be in other industries. Like if you're developing a drug for Alzheimer's and I'm developing a drug for Alzheimer's, we're both trying to solve the same problem. And if your drug works, that's great. That doesn't mean my drug won't work. Like there can be multiple drugs for the same disease. And in fact, having multiple drugs is usually better because different drugs work for different people.

And so the idea that we should be hoarding all of this process knowledge and making it harder for everyone to develop drugs—that just doesn't make sense. Like we're all trying to solve the same problem, which is getting drugs to patients. And if we can all do that more efficiently, everyone wins.

And I think there's also a really important point which is that a lot of the companies that we've talked to, they actually want to share this information. Like they're frustrated by the fact that they can't learn from what other companies have done. They're frustrated by the fact that they have to reinvent the wheel every time they do a trial. And they would love to have a repository where they could go and say, okay, how did other people design trials for this disease? What endpoints did they use? What were the common pitfalls?

And so I think there's actually a lot of appetite for this kind of knowledge sharing. The problem is just that the current system doesn't support it. And so we're trying to create the infrastructure to make it possible.

Consultant opinions and regulatory documents

Patrick: So just to make this concrete for people, when you say Common Technical Document, what's actually in one of these documents?

Ruxandra: Yeah, so the common technical document, or CTD, is the format that companies use to submit their drug applications to the FDA. And it's actually an internationally harmonized format, so it's the same format that's used in Europe, in Japan, in other countries. And it's a very standardized format.

And it contains everything about the drug. So it contains all of the preclinical data—so all of the studies that were done in animals or in cells before the drug was tested in humans. It contains all of the clinical data—so all of the results from the phase one, phase two, phase three trials. It contains information about the manufacturing process—how the drug is made, what the quality controls are. It contains information about the proposed labeling—so what the drug is supposed to be used for, what the dosing is, what the warnings are.

And it's a very comprehensive document. Like for a typical drug, the CTD can be tens of thousands of pages. And it's all very carefully organized according to this standardized format. So there's a section for preclinical data, a section for clinical data, a section for manufacturing, et cetera.

And what we're interested in is particularly the parts of the CTD that describe the trial design and the trial results. Because that's where a lot of this process knowledge is encoded. Like, how did they design the trial? What endpoints did they use? How did they recruit patients? How did they handle adverse events? All of that stuff is in there. And that's the kind of knowledge that could be really valuable to share.

Patrick: And just to make sure I'm understanding this correctly, currently every company that wants to submit a drug to the FDA has to create one of these documents from scratch, essentially?

Ruxandra: Well, they have to create it for their specific drug, yes. But they don't have access to examples of what other companies have done. So if you're a small biotech company and you're trying to figure out how to design a trial for a particular disease, you can't go and look at what Pfizer did or what Merck did. You just have to figure it out yourself.

And that means that you might make mistakes that other people have already made and learned from. Or you might not know about best practices that other people have discovered. And that's a huge inefficiency.

Patrick: I'm reminded of a story that I heard about the early days of the internet, where there was this problem that every university had to figure out how to connect to the internet. And it was this very complicated technical process. And at the beginning, every university was just figuring it out themselves. And it was very slow and very expensive.

And then someone had the brilliant idea of just writing down how to do it and sharing it with everyone. And suddenly it became much easier for universities to connect to the internet. And I think that's kind of what we're talking about here. It's like, let's just write down how to do clinical development and share it with everyone.

Ruxandra: Exactly. And I think that's a great analogy. And I think, you know, one of the things that I find really exciting about this project is that it could have such a big impact with relatively little effort. Because the information already exists. It's already been created by companies. It's already been submitted to the FDA. We're just trying to make it accessible.

And I think if we can do that, it could really transform clinical development. Because suddenly you'd have this knowledge base where you could go and learn from what other people have done. And that could save so much time and money. And it could help more drugs get to patients faster.

Streamlining the regulatory process

Patrick: So what are the main obstacles to making this happen?

Ruxandra: So there are a few main obstacles. One is the legal framework. So as I mentioned, there's a law that says the FDA has to keep this information confidential. And so we need to work within that legal framework. And the way we're trying to do that is by having companies voluntarily share information that they don't consider to be trade secret.

And the key word there is "voluntarily." Because companies have to be willing to participate. And so we need to make the case to companies that this is in their interest. And I think there are a few arguments for why it is. One is that they will benefit from being able to access other companies' information. So it's kind of a reciprocal arrangement. You share your information, you get access to everyone else's information.

Another argument is that this could actually help them get their drugs approved faster. Because if the FDA can see that a particular trial design has been successful for other drugs, they might be more likely to approve that trial design. And that could save time in the back and forth with the FDA.

And then there's also kind of a public goods argument, which is that this is just the right thing to do for society. Like we all benefit when drugs get developed more efficiently. And companies are part of society, and they benefit too.

But I think the biggest obstacle is just inertia and coordination. Like this is a big change from how things are currently done. And it requires companies to do something that they're not currently doing. And that always takes effort. And so we need to make it as easy as possible for companies to participate. And that means building the infrastructure, building the repository, figuring out the legal mechanisms, all of that stuff.

Patrick: And have you started talking to companies about this? What has the response been?

Ruxandra: Yeah, we have. And the response has actually been quite positive. Like I mentioned, a lot of companies are frustrated with the current system. And they see the value in having more shared knowledge. And so there's definitely appetite for this.

The challenge is just getting them to actually commit to participating. Because it requires effort on their part. They have to go through their documents and identify what they're comfortable sharing. They have to work with their legal teams to make sure they're not sharing anything that's actually trade secret. And that all takes time and resources.

And so I think what we need to do is make the value proposition very clear. And show them that the benefits of participating outweigh the costs. And I think as we build up the repository and as more companies participate, that value proposition will become more and more compelling. Because the more information is in the repository, the more valuable it becomes.

And I think there's also a kind of first-mover advantage in some sense. Like the companies that participate early will be seen as leaders in the field. They'll be seen as companies that care about improving clinical development, that care about getting drugs to patients faster. And I think that's valuable from a reputational standpoint.

Building a public library of regulatory documents

Patrick: So if this works, if you can get companies to start sharing this information, what do you think the impact will be?

Ruxandra: I think the impact could be really significant. I think in the short term, you'd see a reduction in the time and cost of clinical trials. Because companies would be able to learn from what others have done. They wouldn't have to reinvent the wheel. They could avoid mistakes that others have already made. And that could save months or years and millions of dollars.

In the medium term, I think you'd see more innovation in trial design. Because once you have this knowledge base, you can start to analyze it. You can start to see patterns. You can start to identify what works and what doesn't work. And that could lead to new and better ways of designing trials.

And in the long term, I think this could fundamentally change the culture around clinical development. Right now, it's very secretive, very siloed. And I think if we can create a culture of more openness and more knowledge sharing, that could have all sorts of positive effects. It could make clinical development more collaborative. It could make it easier for new people to enter the field. It could make it easier for academics to study it and suggest improvements.

And ultimately, I think this could lead to more drugs getting to patients faster. Which is the whole point, right? Like the reason we care about making clinical development more efficient is because we want to help people. We want to get treatments to people who need them. And if this project can contribute to that, I think that would be really meaningful.

Patrick: I think there's also an interesting dynamic where if you make the process more transparent and more accessible, you lower the barriers to entry. And that could lead to more competition, which could lead to more innovation. Because right now, if you're a small biotech company, it's very hard to navigate the clinical development process. You don't have the resources of a big pharma company. You don't have the institutional knowledge. And so you're at a huge disadvantage.

But if you could go to a repository and learn from what others have done, that levels the playing field a bit. And that could mean that more innovative ideas actually make it through the process. Because it's not just the companies with the most resources that can navigate the process. It's the companies with the best ideas.

Ruxandra: Exactly. And I think that's a really important point. Because I think one of the things that's really problematic about the current system is that it favors incumbents. It favors companies that have already done this many times, that have all the institutional knowledge, that have the relationships with the FDA. And that creates a barrier to entry for new companies.

And I think if we can lower that barrier to entry, that would be really valuable. Because a lot of innovation comes from small companies. A lot of breakthrough drugs come from small biotechs, not from big pharma. And so if we can make it easier for those small companies to navigate the clinical development process, that could lead to more innovation and more new drugs.

And I think there's also a geographic dimension to this. Because right now, the vast majority of clinical trials are conducted in the US and Europe. And that's partly because those are where the companies are, but it's also partly because those are where the knowledge is. If you're trying to conduct a clinical trial in, say, India or Brazil or somewhere else, it's very hard to get the knowledge you need about how to do it.

But if we had a global repository of this information, that could make it easier for companies anywhere in the world to conduct clinical trials. And that could lead to more diversity in where trials are conducted, which could be really valuable.

Encouraging novel approaches in biotech

Patrick: So for people who are listening to this and thinking, "This sounds great, what can I do?" What would you say to them?

Ruxandra: I think there are a few things. One is if you work at a company that does clinical trials, talk to people at your company about this. See if there's interest in participating. See if you can be a champion for this within your organization.

If you're an academic or a policy maker, you can help by advocating for this. By making the case that this is important, that this could have a big impact. By helping to build support for it.

And for everyone, you can follow the project. We're going to have a website launching soon where we'll share more information about what we're doing and how people can get involved. And you can share it with people you know who might be interested. Because I think the more people who know about this, the more momentum we can build.

And I think there's also a role for policy makers here. Because while we're trying to do this through voluntary participation, there could also be policy changes that would support it. For example, there could be incentives for companies to share information. Or there could be changes to the legal framework around trade secrets that would make it easier to share information that's not actually competitively sensitive.

So I think there are multiple paths to making this happen. And we're pursuing all of them.

Patrick: And just to be clear, this is something that you're working on personally? Or is this an organization?

Ruxandra: So this is a project that I'm leading, and we're working with a number of partners. We're working with some academics who are experts in clinical trials. We're working with some policy organizations. And we're talking to companies. But it's still in the early stages. We're still building it.

And that's actually one of the things I'm excited about is that this is very actionable. This is something that we can actually make happen. It's not like we need to wait for some huge policy change or some technological breakthrough. We can start doing this now. We can start building this repository. We can start getting companies to participate. And we can start seeing the benefits relatively quickly.

Addressing risk aversion in the industry

Patrick: I want to zoom out a bit because I think there are some broader lessons here about how innovation happens and how we can make it happen better. And one of the things that strikes me about your work is that you're not trying to invent some completely new technology. You're not trying to discover some new scientific principle. You're trying to make the existing process work better. And I think that's a really important kind of innovation that often gets overlooked.

Because we tend to think of innovation as being about new discoveries, new inventions. But a lot of the time, the most impactful innovations are about making existing things work better. About removing friction from existing processes. About making knowledge more accessible.

And I think that's harder in some ways because it requires you to understand the existing system really deeply. And it requires you to navigate all the institutional barriers and the political challenges and all of that. Whereas if you're inventing something completely new, you can kind of do it in your lab and not worry about all that stuff until later.

Ruxandra: Yeah, I completely agree. And I think that's one of the things that's been really interesting about this work is that it's forced me to really understand how these institutions work. How the FDA works, how pharmaceutical companies work, how the legal framework works. And I think that understanding is really valuable because it allows you to see where the opportunities are for improvement.

And I think you're right that this kind of innovation is often undervalued. Like if you look at what gets celebrated in biotech, it's usually the companies that are developing some completely new modality or some breakthrough technology. And those things are important. But I think improving the infrastructure, improving the process—that can be just as impactful.

And I think there's a parallel here to what you were saying earlier about the internet. Like the people who built the infrastructure of the internet—the protocols, the standards, the knowledge sharing mechanisms—they didn't get as much attention as the people who built the flashy consumer applications. But they were arguably just as important, if not more important, because they enabled everything else.

And I think that's what we're trying to do with clinical development. We're trying to build the infrastructure that will enable everything else. That will make it easier for everyone to develop drugs.

Patrick: I think there's also an interesting point about leverage here. Because if you improve the efficiency of clinical trials by even a small amount—say 10% or 20%—that has a huge impact. Because clinical trials are so expensive and there are so many of them. And so even a small improvement multiplied across all the trials that get conducted every year, that adds up to a lot of money and a lot of time saved. And ultimately, a lot more drugs getting to patients.

And I think that's one of the things that's exciting about working on systems and processes and infrastructure. Is that you can have this kind of leverage where a relatively small change can have a very large impact.

Ruxandra: Exactly. And I think that's something that I try to communicate to people who are thinking about where to focus their efforts. Like if you want to have a big impact on health, you can try to discover a new drug. And that's great, and we need people doing that. But you can also try to make the drug development process more efficient. And that might have an even bigger impact because it affects every drug that gets developed.

And I think the same logic applies to a lot of other areas. Like if you want to improve education, you can try to be a great teacher. But you can also try to improve the education system. And the latter might have more leverage.

The importance of courage in professional growth

Patrick: So I want to talk a bit about your career trajectory because I think it's interesting and I think it might be instructive for other people. You did a PhD in genomics, which is very technical, very scientific. And now you're working on policy and institutions and systems. How did you make that transition?

Ruxandra: Yeah, it's been a bit of an unusual path. I think when I started my PhD, I had a pretty traditional view of what a scientific career would look like. Like I thought I would do my PhD, maybe do a postdoc, maybe become a professor. And I would spend my time doing research, publishing papers, all of that.

But somewhere along the way, I realized that the questions I was most interested in were not really scientific questions. They were questions about how do we make the scientific process work better? How do we get discoveries translated into things that actually help people? How do we make the drug development process more efficient?

And those are not questions that you can answer by doing more bench science. Those are questions about systems and institutions and policy. And so I started reading about those topics and thinking about them and writing about them. And I realized that this was actually what I wanted to work on.

And I think there was definitely some risk in making that transition. Because I was leaving a relatively well-defined career path and going into something that was much less clear. Like there's no obvious career ladder for someone who wants to work on improving clinical development. It's not like you can just go and get a job as a "clinical development improver."

But I think I also felt like the potential impact was so much greater. Like if I spent my career doing genomics research, I might publish some papers that other genomics researchers would read and cite. And that would be fine. But if I can help make clinical development even a little bit more efficient, that could affect millions of people.

Patrick: I think there's an interesting question about how institutions and society can better support people making those kinds of transitions. Because I think there are probably a lot of people who have technical skills and domain expertise, and who could be really valuable working on these kinds of systemic problems. But the incentives and the career structures don't really support that.

Like if you're a scientist, the way you succeed is by publishing papers and getting grants and building a reputation in your specific subfield. And there's not really a parallel structure for people who want to work on improving the scientific process itself.

Ruxandra: Yeah, I think that's absolutely right. And I think it's particularly hard when you're early in your career. Like when you're a PhD student or a postdoc, you're very vulnerable. You don't have tenure, you don't have job security. And so taking risks is really scary.

And I think there's a broader problem here which is that our institutions are really good at rewarding people who stay in their lane. Like if you're a genomics researcher, and you just do genomics research, and you publish papers in genomics journals, and you go to genomics conferences—that's what gets rewarded. But if you try to do something different, if you try to work across disciplines or work on systemic problems, that's much harder to get recognized for.

And I think we need to change that. I think we need to create more paths for people to work on these kinds of problems. And I think we need to celebrate and reward the people who do it.

Patrick: I think it's particularly hard when you're young because you haven't built up the social capital and the credibility that would give you the freedom to take those risks. Like if you're a senior professor with tenure, you can afford to say, "I'm going to spend the next five years working on this weird interdisciplinary project." But if you're a PhD student, you can't really do that because you need to graduate and get a job.

Ruxandra: Exactly. And I think one of the things that helped me was having people who believed in me and who were willing to vouch for me. Like I mentioned Tyler Cowen earlier, but there were other people too. People who were willing to say, "Hey, this person is doing interesting work, you should pay attention to them." And I think that made a huge difference.

And I think that's something that people who are more established in their careers can do. They can lend their credibility to younger people who are trying to do interesting things. They can make introductions, they can provide advice, they can publicly support their work. And that can make a huge difference.

Patrick: I think that's a really important point. Tyler Cowen has also been enormously helpful to me, too, and several people I know.

And I think it's something that doesn't require a huge amount of effort but can have a disproportionate impact. Like it doesn't take that much time to read someone's work and tweet about it or introduce them to someone. But it can make a huge difference to that person.

If you're someone who's in a position to support younger people, you should think of it as a portfolio. Not every bet is going to pay off. Some people you support are not going to end up doing anything particularly impactful. But some of them will. And the upside is so asymmetric that it's worth it.

Ruxandra: Yeah, I completely agree. And I think there's also a kind of cultural shift that needs to happen where we celebrate risk-taking more. Like right now, I think particularly in academia, the culture is very risk-averse. People are very afraid of failure. And I think that's problematic because it means that people don't try ambitious things.

And I think we need to create a culture where it's okay to fail. Where people who try ambitious things and fail are still respected. Because the alternative is that people just play it safe. And if everyone plays it safe, we don't get innovation.

Patrick: I think it's particularly hard when the feedback loops are so long. Like in startups, if you try something and it doesn't work, you find out relatively quickly. Like within a year or two, you know whether your startup is going to succeed or fail. But in academia or in clinical development, the feedback loops are so much longer. Like you might spend five years working on something before you know whether it was worth it.

And I think that makes it harder to take risks because the cost of failure is so high.

Ruxandra: Yeah, exactly. And I think that's why it's so important to have support from other people. Because when the feedback loops are that long, you need people who can give you encouragement and validation along the way. You need people who can say, "Yes, this is important work, keep going." Because otherwise it's very easy to get discouraged.

And I think that's another way that more senior people can help. They can provide that encouragement and validation. They can help younger people stay motivated when things are hard.

Patrick: I want to talk a bit about the specifics of how you built credibility in this space. Because I think you came from genomics, which is not the same as clinical development. And yet you've been able to establish yourself as someone who people take seriously on these topics. How did you do that?

Ruxandra: I think a lot of it was just putting in the work to really understand the topic. Like I spent a lot of time reading about clinical trials, talking to people who work in clinical development, trying to understand how the process actually works. And I think that deep understanding was important.

And then I started writing about it. I started publishing pieces on my Substack and in other venues where I would explain what I had learned and propose ideas for how to make things better. And I think the combination of deep understanding plus clear communication was important.

And I think I was also willing to take positions that were maybe a bit controversial or that challenged the conventional wisdom. Like I wasn't just summarizing what everyone already knew. I was trying to say new things, to point out problems that people weren't talking about, to propose solutions that people hadn't considered.

And I think that got people's attention. Because there's a lot of conventional wisdom in this space, and there's a lot of people who just repeat the same things over and over. And so when someone comes along and says something new, that stands out.

Patrick: I think there's also an interesting point about being an outsider. Like sometimes coming from outside a field can actually be an advantage because you're not bound by all the assumptions and conventions of that field. You can see things with fresh eyes.

Ruxandra: Yeah, I think that's true. I think there were definitely times when my lack of experience in clinical development was an advantage. Because I would look at something and think, "Well, that's obviously inefficient, why doesn't anyone fix it?" And people who had been in the field for a long time would say, "Well, that's just how it's always been done."

And I think that fresh perspective was valuable. But I also think it's important to balance that with genuine expertise. Like you can't just come in as a complete outsider and think you know better than everyone. You need to actually do the work to understand the field. But once you have that understanding, I think being an outsider can help you see opportunities for improvement that insiders might miss.

Supporting young professionals and catalyzing change

Patrick: You mentioned writing, and I think that's been a big part of how you've built your reputation. Can you talk a bit about your approach to writing and why you think it's been effective?

Ruxandra: Yeah, I think writing has been really important for me. I think partly it's just a way to clarify my own thinking. Like when I write about something, I have to really understand it. I have to be able to explain it clearly. And that forces me to think through the details and make sure I actually know what I'm talking about.

But I think writing is also a way to reach people. Like if you just talk to people one-on-one, you can only reach so many people. But if you write something and put it on the internet, potentially thousands of people can read it. And some of those people might be in a position to actually do something about the problems you're describing.

And I think I've tried to write in a way that's accessible. Like I'm not writing for other academics. I'm writing for a general audience. And I think that's important because the problems I'm interested in are not just academic problems. They're practical problems that affect real people. And so I want to communicate with the people who are actually working on these problems, not just other researchers.

Patrick: I think there's also something valuable about writing in public, where you're putting your ideas out there and inviting feedback and criticism. Like it's much easier to have bad ideas in private than in public. Because when you put something out in public, people will tell you if it's wrong or if you're missing something.

Ruxandra: Yeah, exactly. And I think I've definitely had experiences where I've written something and then people have responded and pointed out things I hadn't considered or disagreed with my conclusions. And that's been really valuable. Because it's made my thinking better.

And I think there's also a kind of accountability that comes with writing in public. Like if you say something in private, you can always change your mind later and no one will remember. But if you write something and publish it, it's there forever. And so it forces you to be more careful about what you say.

Patrick: I think there's also a network effect where the more you write, the more people discover your work, and the more opportunities come to you. People reach out with ideas, with offers to collaborate, with introductions to other people. And that can be really valuable.

Ruxandra: Yeah, absolutely. I think some of the most valuable connections I've made have come from people who read something I wrote and reached out. And I think that's one of the things that's great about the internet is that it makes it possible to connect with people who you would never otherwise meet.

And I think that's particularly valuable when you're working on interdisciplinary problems. Because the people you need to connect with might not be in your immediate network. They might be in a completely different field or industry. And writing is a way to reach those people.

Patrick: So as we're wrapping up, I'm curious—what do you see as the next steps for the Common Technical Document Project? What needs to happen in the next year or two?

Ruxandra: I think the immediate next step is to get a few companies to commit to participating. Because we need to prove that this can work. We need to show that companies are willing to share information and that there's value in the repository.

And then once we have a few companies participating, I think we can build momentum. Because other companies will see that it's working and they'll want to join. And the more companies we have, the more valuable the repository becomes.

And in parallel with that, we're working on building the infrastructure. Figuring out the technical side of how to store and organize this information. Figuring out the legal side of how to make sure we're not running afoul of any trade secret laws. All of that.

And then I think in the longer term, we want to think about how to make this information more useful. Like it's one thing to just have a repository where you can go and search for information. But ideally, we want to be able to analyze this information and derive insights from it. We want to be able to say, "Here are the common patterns in successful trials. Here are the common mistakes that lead to failures." And that requires not just collecting the data, but actually analyzing it.

Patrick: And what can people do if they want to help?

Ruxandra: I think the main thing is just to spread the word. If you know people at pharmaceutical companies or biotech companies, tell them about this project. If you're a policy maker or work in policy, think about how policy could support this kind of knowledge sharing.

And if you're interested in following the project, we'll have a website launching soon where you can learn more and sign up for updates. And I think we'll also be looking for people to help with various aspects of the project—technical people to help build the infrastructure, legal people to help navigate the trade secret issues, domain experts to help curate and organize the information.

So there will be lots of ways to get involved.

Patrick: Great. And where can people follow your work more generally?

Ruxandra: So I'm on Twitter at @RuxandraTeslo. And I write on my Substack, which is writing.ruxandra.bio. And I also write for various publications—Works in Progress, Institute for Progress, I recently wrote for the New York Times. [Patrick notes: I think people who caught that ordering and are miffed should spend more cycles thinking why it is a natural ordering for people who write for the New York Times.] So you can follow me there and see what I'm working on.

Patrick: Wonderful. Well, Ruxandra, thank you so much for your time today and for all the work you're doing on this. I think it's really important work and I'm excited to see where it goes.

Ruxandra: Thank you so much for having me. This was great.

Patrick: And for everyone else, thanks so much for listening to Complex Systems and we'll see you next week.

Thanks for tuning in to this week's episode of Complex Systems. If you have comments, drop me an email or hit me up at @patio11 on Twitter. Ratings and reviews are the lifeblood of new podcasts for SEO reasons, and also because they let me know what you like.