The economics of discovery, with Ben Reinhardt

This week, Patrick McKenzie (patio11) sits down with Ben Reinhardt, founder of Speculative Technologies, to examine how science gets funded in the United States and why the current system leaves much to be desired.

Sponsor: GiveWell & Framer

Support proven charities that deliver measurable results and learn how to maximize your charitable impact with GiveWell. First-time donors can go to givewell.org, pick “Podcast” and enter COMPLEXSYSTEMS at checkout to get $100 matched.

Framer is a design and publishing platform that collapses the toolchain between wireframes and production-ready websites. Design, iterate, and publish in one workspace. Start free at framer.com/design with code COMPLEXSYSTEMS for a free month of Framer Pro.

Timestamps

(00:00) Intro
(00:26) Understanding focused research organizations (FROs)
(01:52) The evolution of science funding
(03:59) Taxonomy of research: basic, applied, and development
(06:14) Challenges in science funding and research
(08:12) The role of process knowledge in research
(18:52) The bureaucracy of tech transfer offices
(20:00) Sponsors: GiveWell & Framer
(22:33) Critique of tech transfer offices
(25:20) The burden of bureaucracy on researchers
(44:34) Emerging solutions and optimism in research
(46:58) Wrap

Transcript

Patrick McKenzie:

Welcome to Complex Systems, where we discuss the technical, organizational and human factors underpinning why the world works the way it does. Hi to everybody. My name is Patrick McKenzie, better known as Patio11 on the Internet, and I'm here with my buddy Ben Reinhardt, who's the founder of Speculative Technologies, a focused research organization.

Understanding focused research organizations (FROs)

Fros, if you're not familiar, are a sort of innovation that is returning to tradition in terms of how we do science funding. But we've had some episodes recently about charitable giving and some episodes recently about sort of for profit investing. And science funding sits uncomfortably in the spiritual intersection of those two, where the gains from core science funding are often not directly captured in the way that the gains from funding a company would be captured. But they're not exactly charitable either. For one thing, those of us who have worked in tech have had long and prosperous careers due to someone in the past making a decision to fund some amount of research that we eventually built on top of. And with that super broad prompt, I just want to talk about that, what we currently do for the funding ecosystem in the United States and how we could improve it.

Ben Reinhardt:

I'm excited to be here. One quick correction is Speculative Technologies is not itself a focused research organization. We help people start them and aspire to run them internally. And I can get into the nitty gritties of what the classic definition of a focused research organization, as classic as a five year old definition can be, but that is the one little asterisk on that.

Patrick McKenzie:

Yeah, my bad.

The evolution of science funding

So from a super high level, I think the common narrative in tech spaces of the funding for basic research in the United States and elsewhere was that it was at one point in the past largely an industrial game with places like Bell Labs that had large semi monopoly powers investing a large amount of economic rent into their internal laboratories, which deployed huge amounts of capital by the standards at the time, against basic research. And then that fell out of favor for a variety of reasons, in favor of the federal government funding almost all basic research through a couple of different funding authorities and organizations. And there has been something of a shift of that in the course of the last few years, with increasingly a lot of research in particular areas of interest funded through industry again, but the federal government remains the largest funder by quite a bit. Does that broadly capture the shape of the curve?

Ben Reinhardt:

That broadly captures the shape of the curve. 

[Patrick notes: For people who do not have these numbers cached, but benefit from understanding via numbers, see e.g. this report.]

I think it obviously it depends on when you start, when you start history, what is your baseline? So if we look at sort of the post- World War II system, which is kind of what we live in today, the government has sort of always been funding a lot of basic research and the industrial funding of basic research. I will put a star on this because I think the categorization of research into basic and advanced and development is a bad bucketing system. But companies do not do much less basic research than they used to. And now if you look at the breakdown, the vast majority of it is funded by the federal government.

Patrick McKenzie:

And I would love your take on why these three buckets are not a great taxonomy for organizing the world of research. But just for the benefit of people who have never considered that question before, what is the typical taxonomy? And then we can go into reasons why that might not leave reality at the joints.

Taxonomy of research: basic, applied, and development

Ben Reinhardt:

So the typical taxonomy of research, and this is like encoded into law, is that there is basic research which sort of, in the sort of classic sense of your scientist who's like, oh, I wonder how snails meet.

[Patrick notes: If you’re the kind of person who geeks out on this stuff, and if you’re not I question what you’re doing reading this transcript, you might find OMB Circular A-11, Section 84 interesting. Per the National Academies:

"OMB Circular A-11 (2003): Basic research is defined as systematic study directed toward fuller knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind."

"Applied research is defined as systematic study to gain knowledge or understanding necessary to determine the means by which a recognized and specific need may be met."

]

Right. Like I think is sort of like a prototypical is research done into the nature of the universe with no mind towards how this might be useful for anybody. It's just like pure curiosity. And then there is applied research which you are then still very much. They're still very open ended questions, but you are now trying to do something useful. So now it is like how can we make the structure of this material have this specific set of properties in a way that we don't know how to yet. Then there's development, which is the last piece before making a product. So to a large extent the work to make SpaceX’s Starship is development in the sense that they are trying to do something that nobody has done before and it is very hard, but there is a clear sort of product at the end of it.

Patrick McKenzie:

This is the quote unquote “‘just’ an engineering problem, no new science required.”

[Patrick notes: This is a common enough bromide in my circles, sometimes used with varying emotional valence. Occasionally it is said positively, meaning the difficulty of a task/goal is bounded and relatively small. Occasionally it is said derisively, because e.g. the global financial system is just engineering, not science to this conception of reality, and to the extent that one grants that for the sake of argument, that is an awful lot of engineering.]

Ben Reinhardt:

Yeah, I hate it.

Patrick McKenzie:

Something that engineers never love hearing, particularly from the CS field, given the existence of Turing complete languages. Everything is an engineering problem.

Yes, I think it is sometimes useful in terms of my mental taxonomy to think, okay, there is foundational research done into electromagnetics and then in the middle of we discovered that LEDs are a concept. And then actually getting a blue LED to exist in the physical universe at a price people can afford is more applied research.

But the United States, through the genius of its elected representatives, has encoded this tripartite definition into law. Why might that not have been a great idea?

Challenges in science funding and research

Ben Reinhardt:

Well, I think semiconductors are a great example of why.

So you think about point at the history of the transistor. And it starts off with sort of this realization that there are these things called semiconductors where they're not quite conductors and they're not quite insulators and they have weird properties about how electrons move through them. And that is certainly basic research. 

Then people realized: we might be able to use this to make better amplifiers, we might be able to replace these vacuum tubes. And so the way that you actually do the work there is a mixture of applied research where you're trying to get these semiconductors to be useful at the same time as trying to figure out what the actual shape of the product is that you're going to make out of them. But then you run into this situation where you realize that actually our understanding of the laws of physics is insufficient to explain how electrons move through these semiconductors.

You basically need to do some updates on our understandings of quantum mechanics in order to actually properly model how electrons move through the semiconductor. And you're doing this sort of at the same time as thinking about the product. And so it becomes this wild entangled mess with a whole bunch of loops. And so if you have a team and one guy is trying to figure out new laws of quantum mechanics while talking to a guy who's trying to bond hunks of semiconductor together to try to make this amplifier that is actually useful, what kind of research are you doing? Right. It goes in no bucket. And the reality is that if you zoom out, the vast majority of useful research projects look like this.

The role of process knowledge in research

Patrick McKenzie:

Yeah, I think it often gets a short shrift in discussions, even though it's been highlighted for decades in a number of places. In Japan, the magic word is kaizen, which got appropriated and/or homaged by a US-based consulting community. And often with regards to semiconductors, the jargon thrown around is “process knowledge”, where there is something that hasn't been reduced to a paper yet, but you can't build a chip fab without having it where just the designs and just the description of all the specs for all the machines, you wire them all up together and you will not get a usable wafer at the end of it because you lack the process knowledge.

[Patrick notes: See generally Nelson & Winter, An Evolutionary Theory of Economic Change, a more philosophical take by Michael Polanyi, The Tacit Dimension, and basically anything written in the last few years about semiconductor manufacturing and how our entire civilization rests on about 200 sets of shoulders in Taiwan.

Incidentally, if you have a particular set of interests, one memorable articulation of the claim: Fantasy story pitch: Imagine there's a magical herb that everything in the world relies on but it only grows on a single island and there are like 200 [people] who can harvest it and that island is precariously placed between two kingdoms...that's basically the semiconductor industry.]

Ben Reinhardt:

And the dirty secret of science is that even when there is a paper, there's still process knowledge. There are many, many situations where the only lab that can actually do the thing that is reported in the paper is this one lab that has the process knowledge that even if they are trying to communicate it is very hard because you're like, oh, you have to fiddle with this thing. Just the Right way. And so I think process knowledge is shot through everything. Even when there are actually papers, it is difficult to reproduce them without having been on the writing team. [Patrick notes: This gestures in the direction of the most hopeful take I’ve ever heard about the replication crisis.] 

This sidebar is why the idea that we are going to teach AI to do all of the science just by feeding it a bunch of training it on a bunch of papers is one that I personally think is not going to happen.

Patrick McKenzie:

I think there's a number of reasons that one could doubt that broad hypothesis. Even though, goodness, we hope that AI results in an acceleration of science or else what the heck are we doing with it?

Ben Reinhardt:

Yeah, no, it certainly could.

Patrick McKenzie:

We will lack digital twins for a while, to use another magic jargon word that people really like. 

[Patrick notes: A “digital twin” is a sufficiently high-fidelity simulation of a complex system (ba dum bum) in the real world to be able to run the system in silico. These would, in theory, allow you to test changes to the system or subsystems at a very high cycle rate because you would not be rate-limited by getting things done in physical reality. Applied to AI, sufficiently good digital twins might enable so called “self-play”, where computer systems do reinforcement learning or similar techniques not off of a dataset produced at high cost by humans but out of a massive dataset, including less-than-high-quality approaches, produced by computer systems. This has been successful in some domains; it is called “self-play” because we actually can design a digital twin of a chess board or go board which reasonably models the same one a human would actually use. Designing a digital twin geothermal power plant is… somewhat trickier.]

And so to some amount of the science is going to be rate limited by people reading the AI from a screen and then using that to do something in the physical world and then iterate that loop a lot. At least until the point where we have a high fidelity simulation of the world that can be run in a computer. Currently the high fidelity simulation of the world uses human language as the substrate or some derivative of human language in the model weights, which is a wild world to have ended up in. I don't think we appreciate how science fiction reality is at the moment anyhow. Worthwhile tangent to explore some other time. But we were talking about scientific funding.

Ben Reinhardt:

We could just talk about how science funding works in the United States right now. Let's do a broad overview of the system. I will flag that how science is funded is deeply coupled to where the science is done, so there are these two parallel tracks that are deeply interrelated.

Using the basic, applied, and development buckets as they exist right now, I think roughly $900 billion goes into science research funding in the United States. We can slice that up in a couple of different ways. The vast majority actually goes towards development. Don't quote me on these numbers, but it's like $700 of that $900 going towards development, then roughly two thirds of the remainder goes towards applied research, and the rest goes towards basic. So basic research is the least expensive. And this makes sense, right? You don't need to be building huge machines. You don't need to be blowing up spaceships when you're doing basic research.

Another way of chunking up that money is by source. You can largely put it into three buckets: money from the government, money from private corporations, and money from private foundations and other nonprofits. A lot of the money actually does come from businesses. I think it's roughly $600 of that $900, and then another $200 is from the government, and less than $100 is from private organizations.

[Patrick notes: I live in terror of having to quote stats correctly extemporaneously. That said, if you want authoritative numbers:

]

Looking at this, many people say, "Oh, business funding for R&D is very high. What's the problem?" The thing to note is that the vast majority of the R&D money that businesses put in is D—the development. I'm going to go slightly deep here: a thing to keep in mind whenever you hear this number is what can get coded as development spending by these businesses. Some of your audience may be aware of this, but for example, building a new feature in a piece of software can get coded as a development expense. So when you hear that there's all this money going into R&D from businesses, some of it is very legitimate research—Microsoft and Google are building quantum computers and discovering LLMs and all this stuff—but the numbers are a little bit skewed.

Patrick McKenzie:

It's always felt something like an impedance mismatch of how tax policy around software specific has huge distortionary effects around one our understanding of just the fundamental domain of science and around allocation of resources in the economy.

So you were mentioning that businesses can code this, which basically means if the business identifies a particular line item, whether that's an engineer's time or similar as laddering up to research, then they get the R&D tax credit and the companies are incentivized in many cases to try to maximally claim for all the R&D, which pushes up the claimed amount of research and development work done every year.

However, as everyone who has ever worked in a software company before knows: a certain amount of effort in a software business with an existing product, by engineers and product managers and similar, is effectively OPEX [operating expense]. It doesn't look like OPEX in a certain view of your balance sheet / profit and loss statement and isn't coded as 100% OPEX when the accountants or consultants file the tax return. In the United States, if it is cherry-flavored OPEX, you get a substantial tax credit from Uncle Sam for it.

Where if it's like simple normal garden-variety OPEX, you don't. As a result we have incentivized some of the largest organizations in capitalism to say, well, we have an awful lot of cherry-flavored OPEX every year.

That doesn't change a fact of the physical universe. It doesn't change the rate we are learning about reality all that much. It does change what we perceive the shape of the graph of science funding to be, though.

[Patrick notes: For a more scholarly take on a similar argument, see: "Do Tax Credits Stimulate R&D Spending? The Effect of the R&D Tax Credit in Its First Decade." Journal of Public Economics 140 (2016): 1–12.]

As you mentioned, the largest firms in capitalism also do a whole lot of legitimately cutting edge research on everything from quantum computing systems to the Attention Is All You Need you need. That paper is classic basic research where there was no application that could actually be made for it at the point it was written. And then a number of firms, interestingly not the ones that wrote the original paper, chased after it with vim and vigor, and now we have magic answer boxes on our phones. [Patrick notes: And helping me format this transcript, and be the research gopher for finding citations, and many other things in my business besides.]

Ben Reinhardt:

A couple of other things to flag. We've talked about where the money is coming from; there's also the question of where the money is going. The vast, vast majority of money that goes towards what I call "pre-commercial research"—work that may be targeted at an application but isn't yet a thing that makes sense as an investment because of the uncertainty, timescales, and public-goodsiness of the research—goes to universities. A rational investor will look at it and say, "I do not want to put my money into that if I want my money to make more money." So when you hear people complaining about the academic system, that is these universities. There's also a good chunk of work that happens in national labs and a long tail of other organizations.

One of my hobby horses—I have several; you could say I have a hobby chariot pulled by my hobby horses—is that we have developed a system in the United States where, when you ask someone "how does technology happen?", the response will be: "Well, someone does pre-commercial work in a university until it makes sense as a company, they spin it off into a company, that company raises a bunch of money and makes products, and lo, there will be technology." Similar to how the basic/applied/development model of research is a bit flawed, so too is this "how does technology magically happen" model.

Patrick McKenzie:

There are all sorts of institutional incentives that throw wrenches in the works here, but one in particular: at the point where you are spinning something out from a university into its own private company, or selling the technology to someone, our friends at the technology transfer office get involved. You've had some choice words for them in the past. Do you want to spin out that thought on air?

The bureaucracy of tech transfer offices

Ben Reinhardt:

The explicit choice words are that I'm generally a big believer in Chesterton's fence—that most institutions that exist, exist for a reason and need to be reformed, but by and large should exist. That is not the case for tech transfer offices. I think the world would be better if we burned them to the ground. This is not against any individual in a tech transfer office; there are many wonderful, lovely people who work there. But tech transfer offices are serving none of their purported purposes.

If you think about what a tech transfer office ideally should do, it's twofold. One, make sure that technology invented in the university gets out into the world so the world can benefit from it. Two, make it so that the university can capture some of that value, which will then hopefully be plowed back into more research to create more technology and more science. Both of these things seem like good things.

But the reality on the ground is that tech transfer offices serve neither purpose. I looked it up beforehand: the fraction of tech transfer offices that are profitable is 16%. The vast majority actively lose money. Of the ones that do make money, the total amount across the entire U.S. is single-digit billions of dollars a year. That's every single spinout from every single university—the mRNA vaccine, Google, Gatorade, which makes a shockingly large amount of money. The amount of money that tech transfer offices generate is tiny compared to the amount of money that research actually needs.

The biggest thing is the amount of pain people have to go through to get technology out of the university, which is kind of mind-boggling. When someone signs an employment contract with a university, part of that contract is that they do not own anything they invent using university resources or on the clock—the university owns it. So if you're a scientist and you invent something at a university and think, "this would make a great product," you must go to the tech transfer office and say, "May I license my own invention from the university in order to spin out a startup?"

Then the tech transfer office will hem and haw. First, they will feel no urgency—and as your listeners know, time is of the essence when you're starting a startup. Then they will often want fairly onerous deals: monthly payments starting at day zero, or an unreasonably large amount of equity. There are real questions about how much technology has counterfactually not spun out of universities because it's such a pain to go through the tech transfer office—how many companies have died because people tried and either were discouraged or the deals were such that it was impossible to raise more money.

Patrick McKenzie:

They function as just one more imposition on a PI's time doing paperwork instead of doing research. [Patrick notes: Ben later defines this for the benefit of you the audience, but a PI is the “principal investigator”—chief theorist, writer, marketer, fundraiser, and manager—in the industrial organization of science in the United States. Yes, we expect one person to do all of that. Yes, it is insane, which we will shortly discuss at length. Another similar issue is in Inadequate Equilbria, mostly focusing on how the practice of medicine is twelve jobs rolled into the word “doctor”, but if I recall correctly Eliezer also mentions research as suffering from this affliction.]

The burden of bureaucracy on reseaerchers

A long time ago, in a place far, far away, I was an undergraduate research assistant. And my understanding, as someone who had been a student until a hot minute ago, was that I am working for the princely sum of $12 an hour doing this undergrad research. Obviously, I'm not going to get any sort of upside in this research. That's not the point of the exercise. I'm learning things, et cetera, et cetera. This will be a good way to use the summer.

I spent more than 90 minutes that summer myself doing paperwork from the university tech transfer office, and the PI spent tens of hours over the course of his year. That's wild. I don't think there's any PI in a university who only actually works 2000 hours. But if it were a 1% tax on all research produced by the university, you'd want to know, what are we getting for paying 1% of all potential research output? And if that answer is like, it's literally negative, then just cancel it without replacement.

Perhaps if the university wants to do something instead, they'll just say, yeah, we're going to put a $100k check into anyone who spins out something that seems like we'll be in the things that are on the far right tail of outcomes. And for the rest, that's cheaper than maintaining an office of full-time employees. [Patrick notes: Of course, purpose of a system being what it does, to a very real extent the organization that is the tech transfer office exists to provide jobs to the people who work at the tech transfer office. This is not a phenomenon which only happens in government work.]

Ben Reinhardt:

Yes, that would be very wise. And we've run this experiment—there are natural experiments. The University of Waterloo lets inventors own their own inventions. Basically they say, "Hey, if you go off and make a ton of money, give some of it back." I'm an optimist about human nature, and I think people do give back if it really is a situation where they couldn't have done it without the university. Waterloo is doing quite well off of some of the things invented there.

The riff on this is: your listeners know the dynamics of startups very well, where it's an extreme tail-dominated thing. The vast majority of spinouts from a university are not going to be incredibly lucrative. Trying to squeeze everyone the right amount does nothing besides give you fewer shots on goal.

Patrick McKenzie:

(Stripe doesn't necessarily endorse representations I make in my own spaces.) A thing we saw over and over again at Stripe Atlas is that startups in their early days are just so fragile—particularly in the proto-startup form, where a founder's state of mind is "I'm not exactly sure what I'm going to do with my life next year." The PI is asking: Am I going to continue climbing the ladder? Am I going to try to go after that ambitious research project I just got a grant for—grant-making being its own ball of wax—or should I try commercializing my last research project?

In those early days, just the notion of "well, it's 600 pages of forms to do option B versus option A"... it's still work. I work for a living. But it's work that feels less draining than "I won't do the one that has 600 pages of forms in front of it." And that's barely an exaggeration, by the way.

It's not like 600 is a magic number where people have an internal capacity of 450 forms and then the startup dies. We were tracking this down to literally the form-field level. Eventually you play this out over a few years and it's probably visible in macroeconomic indicators: we have intentionally, and often for good reasons, strangled innovation at the earliest stages due to bureaucratic and administrative overhead. There's a wag quote that the bureaucracy is expanding to fill the needs of the expanding bureaucracy. But having interfaced with some of these processes directly, you definitely feel the indicative truth of that.

Ben Reinhardt:

Oh yes. I've both gone through them myself and am very close to many professors, so I hear on a daily basis what that looks like. The data bears this out too—they did surveys and I think bureaucracy now occupies roughly 40% of logged hours by professors. It's a very real thing.

[Patrick notes: The 42% figure comes from the Federal Demonstration Partnership Faculty Burden Survey (Decker et al. 2007; Rockwell 2009), which surveyed over 6,000 PIs on federally funded grants. Follow-up surveys have confirmed similar levels of administrative burden.]

If I can double-click quickly on a term you've been using: you've said "PI" a lot and I don't know that we've defined it. PI is "principal investigator," a term of art in our modern system of research funding—the person who applies for and gets the money. In a university, usually the PI is a professor; sometimes it's a postdoc or research assistant, but the vast majority of the time it's a professor.

The entire grant-making system is organized around this assumption that there will be a PI who will write the proposals for research, be responsible for executing on the research, and be leading the research—applying to do a very specific project with very specific scopes and aims.

I flag this because it's certainly a fine system—it has gotten us here, it has produced many amazing things—but as a way of doing everything, it's kind of shocking. In startup terms, that would be like: you're not allowed to have a CTO. You're only allowed to have one executive who must do all the fundraising and run the team, and they can only raise money for four very specific projects—"we need to develop this new feature" or "roll out this specific app." And heaven forbid you find a different opportunity and want to use that money for something else, because you certainly can't.

Patrick McKenzie:

This goes back decades. But I sometimes wonder if we're not reacting to the fact of science as conducted centuries ago, where—if you look in the history books—a number of foundational mathematics and foundational science were done by essentially bored noblemen, or patent clerks (infamously). One person with a relatively tiny amount of money, a single practitioner advancing the state of the art, with time to think about the problem, draw things on the blackboard, and have correspondence with peers.

Increasingly, that's not how science is conducted. We have plucked much of our low-hanging fruit, and now you need a large team in a lab environment for much of the research we're discussing. But the funding mechanism assumes: "Yes, we understand you will need a large lab. However, for whatever reason, we can't really interface with large labs. We have to pretend it is one scientist who has all the ideas in their own head." We essentially ban via statute any notion of specialization.

In private industry, granted, the CEO almost always owns the fundraising ball as one of their things—but you're not also forced to be the person doing all the hiring, filing all the timecards, and doing server administration. Whereas under grant contracts, it's all literally on you. You have to be the chief hirer, the chief correspondent with your funding agency, and most of the brain trust.

This doesn't do wonderful things for scientists who are good at science but not great at admin or managing people. They get essentially frozen out of the funding pipeline—unless they have one of our supremely talented polymath PIs who can step in and write the proposal for them while humming over the required line that says it's your work you're applying for, not somebody else's.

Ben Reinhardt:

Yeah, that is spot on. Another one of my hobby horses is trying to create more systems that are sort of outside of this, where you can sort of have funding at some level above the level of a lab and a PI in the same way that you have a company and within that company there are several departments and they're doing all these different things and not needing to go to someone outside and justify on a project by project basis why they need that funding for.

Patrick McKenzie:

This is the benefit of people who have only worked in private industry. Obviously we spend a lot of intellectual calories on fundraising—whether through investment or through sales—but the calories largely get spent at the company level. And while budgeting is a process, teams have some flex in how they spend their department's budget; often it only takes an email to get more allocated to you. If a PI at a university lab has their ambitious research project and needs an extra $200,000, how much additional difficulty does that add to their life?

Ben Reinhardt:

Ben audibly pauses. I pause only because it's almost unquantifiable. If that money is for an entirely new project, there are processes for that—"I have this idea, this project, it needs $200,000." Professors, when they start their jobs, get what's called a startup budget—a slush fund the university seeds the lab with. If they have some of that left, they could deploy it. But if they want to save it for a rainy day, which is very reasonable, and they want extra money for an existing project... there are no standard channels for that.

Patrick McKenzie:

Right?

Ben Reinhardt:

The way you would get that money is if you happen to know an incredibly wealthy person, or you would secretly apply for money for a different project and then do some very sketchy book-cooking: "Oh, well, this grad student who's on this other project is just going to be spending some of their time on this other project." And banish the thought if you want to spend that money on equipment.

Something we didn't mention: most pre-commercial research funding specifies how the money must be spent—what you're allowed to spend it on. You need to say, "Yes, this amount will be spent on consumables like chemicals and reactants, this amount will be spent on personnel." Sidebar: that personnel money is usually earmarked specifically for grad students or postdocs, so you're not really allowed to hire specialized professionals to come in. And a very small amount will be earmarked for equipment.

This creates very interesting incentives not to automate anything in science. Everybody's wringing their hands about how science productivity has not increased as much as many areas of the economy. I buy the low-hanging-fruit argument to some extent, but I don't think it's a complete explanation. When it's relatively easy to get money for grad students and relatively hard to get money for equipment, you're very disincentivized from getting more capex to substitute for opex—even if it would make you much more productive.

I've heard very explicit conversations with PIs: "So why don't you install some robots to do this incredibly repetitive task that right now you have five different grad students doing—basic pipetting, dropping one drop of chemical into another?" And they say, "Well, it's easy for me to get money for grad students, so why would I buy robots?"

Patrick McKenzie:

This also has impacts on the quality of the science done, because we reach for the grad student since it's the one option in the toolbox. You mentioned it's difficult to hire external professionals—both because the social system in the university makes that harder than just promoting a new person to grad student, and because the numbers thrown around might not match industry wages for very long.

I remember there was a funded project at my alma mater, many years after I left, doing urgent research into the effectiveness of a particular drug for treating Covid. They had a 200-question intake questionnaire for patients they were attempting to get into the trial, and were realizing—go figure—that not many patients were getting through an online 200-question questionnaire to get approved.

Through a random pathway, I ended up discussing this with the PI. I said, "Help me understand, as someone who has not done medical research for his entire career: why is it 200 questions?" My hypothesis was it must be the institutional review board or something—they've required these questions and aren't accountable for the actual success of the trial.

He said, "Oh, that's how many the grad student coded."

"So how many are required?"

"Four."

"Well, how about we delete the other 196?"

"No one on the team knows how to do it because it's written in PHP. Only the grad student knows PHP."

It's a big Internet out there. You can find a lot of people who can press the delete key on a PHP app. But no—they had to use the grad student. And why wasn't she in this meeting already? For the usual organizational reasons, she wasn't available and wouldn't be for days.

In this period of intense criticality for the project and the broader society that research is supposed to support, their funding mechanism had me asking the best man at my wedding—who does code PHP—"Hey, can you look at this thing and press the delete key for us?" And indeed it was very effective.

Ben Reinhardt:

That's wild.

Patrick McKenzie:

Grad students do a lot of very excellent work. I've known some extremely talented people who were taking an incredible pay cut relative to working in industry because they loved the science and wanted to spend their life doing it. But I think we are making poor use of their time and brain sweat as a society if we have them pipetting chemicals into containers just because that's the easy thing to put on a grant application.

Ben Reinhardt:

Two quick riffs on that. One is that the requirement to use grad students comes down all the way from legislation, because the United States government sees grad students doing research as fulfilling two different things: one, "we must train the scientific workforce," and two, "we must create research." This sounds great at the level of the Senate, but on the ground it has increasingly led to tensions.

First, we're kind of overproducing grad students—if the vast majority of scientific research needs to be done with grad students, then every time we want more science, we need more grad students. Second, I'd argue we get lower-quality research because, to your point, many grad students are really amazing, but at the end of the day they are trainees. I liken this to a software company where every time you wanted a new feature, you said, "Okay, cool, let's have the interns do it." You not only have issues with people learning and doing at the same time—which can work, but usually is helped by having more senior people around—but also continuity, because the interns leave after a while. It's one thing if the grad student isn't in the room; often the grad student who coded the thing has graduated and is off doing their own thing. You have to do a lot of code archaeology. That's another way the system is a bit rickety.

Patrick McKenzie:

Well, at least you can look in the source control for what they did. This is a dark joke. How many grad programs that I've interacted with actually teach people to use source control? [Patrick notes: I may, possibly, have interacted with some, but I am failing to recall any specific examples.]

Ben Reinhardt:

I tried to introduce GitHub to my lab during grad school. Literally everything was just files on computers. I said, "Hey, there's this thing called Git. We could use it for source control." And everybody was like, "Meh, that sounds like work."

Patrick McKenzie:

Do you mind if I ask what year that conversation happened?

Ben Reinhardt:

I believe that happened in 2013.

Patrick McKenzie:

Okay. So the Joel Test, which came out around 2005 or 2006, was a ten-question test by Joel Spolsky essentially helping people who hadn't yet joined a company evaluate whether that company had taste. Already by 2005-ish in the private tech industry, "do you use source control?" was a pass/fail question. And that's roughly contemporaneous with my own research experience—but even ten years later, it was still taking its time to percolate into the research community.

[Patrick notes: Correcting the record: the Joel Test was 12 questions and from 2000; I originally read it in 2005 or 2006, when I started reading Joel’s writing. It is an interesting time capsule.]

We have abbreviated time together today, and I'd hate to just dwell on the problems. Obviously one can't thank the research system in the United States enough for all the wonderful things in the built environment around us. But what are the reasons for optimism? What has been working well recently?

Emerging solutions and optimism in research

Ben Reinhardt:

There's an emerging new world of people who have seen these problems, realized that the system can be changed, and are working to build new institutions. I call this a small group of misfits trying all sorts of institutional experiments. The realization that we need to change this has started to percolate through more of the culture—that's one reason for optimism.

The other is that, as much as I think it could be better, the U.S. research system is still the thing that everybody looks up to and emulates. Despite all of this, there are still amazing people doing amazing work, and we should be very supportive of them.

Going a little deeper on what's happening: we mentioned FROs at the beginning—focused research organizations. The idea is, what if we organized research more like a startup, with a core group of people working on a very specific problem? That's one thing going on.

There are also attempts in various governments to reorganize research funding. You have ARIA in Britain, modeled after DARPA in the United States. You have organizations on both the for-profit and nonprofit side—like Speculative Technologies—trying to take all these things I've been critiquing, invert them, and say: what if we do have teams of professionals who are not burdened by bureaucracy, working on problems that aren't bucketed by whether they're basic or applied, trying to do very useful things?

There are many ways of implementing that, and I'm excited because many people are trying many different experiments. That's a good reason to be optimistic.

Patrick McKenzie:

Well, more experiments on how to do experiments better never hurt anybody. I regret that that's all the time we have to chat today. But Ben, where can people follow your work on the Internet?

Ben Reinhardt:

You can find me on X: @BenReinhardt. I write both my substack and my website benjaminreinhart.com.

Patrick McKenzie:

Thanks Ben for your time today. For the rest of you, thanks very much and we'll see you next week on Complex Systems. Thank you.