Cheap cognition and the end of practical obscurity, with Garrison Lovely

Patrick McKenzie (patio11) is joined by Garrison Lovely, journalist and author of Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us and How to Stop It, to map the three-sided debate over AI risk and why the arguments keep talking past each other. 

They then turn to what cheap cognition does to surveillance that already exists: FinCEN receives roughly 4 million suspicious activity reports a year and reads almost none of them, ICE agents run about a million queries against that database annually, and every podcast ever recorded is now transcribable for approximately nothing. The conversation covers capabilities denialism, ablated open-weights models, the fraud supply chain, and why AI is also unusually good at writing the Regulation E letter that gets your bank to fix your problem.

Make sure to subscribe on YouTube.

Presenting Sponsors: Mercury, MongoDB & Chainguard

Complex Systems is presented by Mercury—radically better banking for founders. Mercury's new feature Command brings an LLM directly into your banking interface, so checking balances, finding invoices, or sending a wire is as easy as asking. Apply online in minutes at https://mercury.com/. 

What's the point of building faster with AI if your database can't keep up? MongoDB's native data model mirrors the language LLMs already speak. Ship at the speed of AI while staying ACID compliant at Fortune 500 scale. Start building at https://mongodb.com/ai.

If attackers are using AI to weaponize code faster than any team can review it, your scanners won't save you. Chainguard builds libraries and container images from source, verified all the way down, with near-zero CVEs and zero malware. Build safely at https://www.chainguard.dev/. 

Timestamps:

(00:00) Preview
(00:43) Intro
(01:51) The three-sided debate over AI risk
(05:32) Power, politics, and the tech backlash
(09:49) Capabilities denialism and AI tells
(14:54) The obsoleting machine
(17:09) The Turing test is dead
(18:56) Languages for free, then software engineering
(22:37) Sponsors: Mercury | MongoDB
(25:09) How high up the stack do the models decide?
(29:11) Surveillance and cheap cognition
(32:38) Podcasts, FinCEN, and the end of practical obscurity
(38:35) Section 702 and the data broker loophole
(39:35) Sponsor: Chainguard
(40:55) Section 702 and the data broker loophole (cont’d)
(47:23) Institutional friction and a million ICE queries
(53:29) Security through obscurity no longer works
(54:54) Scams, fraud, and ablated models
(1:00:20) Personal utility versus societal backlash
(1:02:16) AI as a tool for redress
(1:06:34) State capacity and regulating what you understand
(1:12:37) Tobacco, nuclear, and AlphaFold: strangle it or steer it
(1:18:37) Where to find Garrison and the book
(1:20:32) Wrap

Transcript

The three-sided debate over AI risk

Patrick: Hideho, everybody. My name is Patrick McKenzie, better known as patio11 on the Internet. I'm here with journalist and author Garrison Lovely, who has written a book recently.

Garrison: Yes. It's called Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us and How to Stop It.

Patrick: And you've also done some long-form journalism articles in Jacobin, among other places, on AI and how it relates to various parts of the political spectrum of the United States.

And I think this is interesting because there are parts of the intellectual class in the United States which have a very different opinion as to where AI is right now and where it is trending than, say, the folks at Lighthaven and the broader San Francisco consensus.

Not that we're going to resolve that difference in opinions in the course of the next ninety minutes, but I would love to help give people who are in the intellectual class and think that this is just a nothingburger a bit more perspective from relatively informed people who don't have a trillion dollars in equity at stake here: look, if you just want to be right, don't bet on this being nothing.

Let’s catch people up on the thesis of, for example, your Jacobin piece.

Garrison: Yeah. That piece was basically exploring this three-sided debate around AI existential risk. On one side you have the kind of classic AI safety community, who think existential risk is real. They believe that artificial general intelligence is possible and potentially imminent and very, very dangerous, potentially.

Then you've got the kind of accelerationists or the boosters, who are like, "AGI is real, but it would be great, and we should get there as fast as possible. We should not regulate or strangle this baby in the crib."

And then you've got the kind of AI ethics community, who are more skeptical of AGI being imminent or even doable, and are saying, "We should focus on the harms today, not on these speculative future risks that are just a distraction from what's happening right now."

Patrick: It's important to say: when people say existential risk, they really do mean that humanity as a species will die out, and they, in many cases, expect that to happen before 2030. And these are not insane people who are saying these things.

[Patrick notes: If Anyone Builds, It Everyone Dies and AI 2027 are well-regarded texts from luminaries with huge amounts of influence on the SFBA crowd, and are somewhat more accessible than reading 15+ years of LessWrong posts.

For what it is worth, I am spiritually part of the SFBA zeitgeist and have slightly more than zero worry about near-term human extinction downstream of AI, in the same fashion I have slightly more than zero worry about global thermonuclear war without AI being a major contributing cause. And so I think the right amount of societal worry is, oh, tens or hundreds of billions of dollars of investment, which is approximately 3-4 orders of magnitude more than currently being spent. You know, something like what we spend on keeping all the nukes under control.]  

The harms that the ethics community, on the other hand, is talking about are things like misinformation potentially flipping elections, child exploitation, non-consensual sexualized imagery of people being created with diffusion models, et cetera, et cetera — where these are things that all rational people think are bad. But the sum of them doesn't add up to “no humans live to see 2031.”

Garrison: Right. And so there's this sense that if you believe in existential risk, or even something less extreme — bioweapons, or catastrophic risks for humanity — this "takes the air out of the room," is what somebody told me. It makes a focus on these things that are inherently speculative, and I think there's this zero-sum mentality around these ideas and risks.

And it's interesting, because there's been some research since I wrote that article showing that if people are presented with arguments about existential risk and near-term harms, they, one, come in being much more concerned with the immediate harms from AI. But two, they become more concerned about existential risk when they see the arguments in the headlines — but they don't become less concerned about the existing harms.

And the retort would be: well, you have a finite amount of attention and political capital, and so if Congress is focusing on x-risk, they're not focusing on bias or something else that is more of a classic focus of the AI ethics community.

But I think a lot of this does come down to this attentional turf war. And the people who believe in existential risk — the earliest ones — are, I think, pretty hostile to the AI ethics community. Like, Eliezer Yudkowsky, the kind of OG AI alignment person, is not super friendly to the left.

[Patrick notes: Not to put words in anyone’s mouth, but I strongly doubt major factions of the political right in the U.S. would look at Eliezer and say “Yep, one of ours.” One of the enduring challenges between SFBA and other centers of power in the U.S. is that in the SFBA many people believe themselves to be aloof from partisan politics and this codes to partisans as “Enemy, got it.”]

That's changed a bit since Bernie Sanders has started to embrace AI existential risk as an important and serious idea, which is kind of ironic. And it's satisfying for me, as somebody who's on the left and also takes AI safety concerns seriously, to see the left is taking this the most seriously now — at least the most prominent leftists in the world. Whereas a lot of the people who are more classically, you know, gray tribe members of Congress are taking money from the OpenAI super PAC and not embracing existential risk as a thing that we should be worried about.

[Patrick notes: There exists some controversy as to whether Leading The Future (LTF) is the OpenAI super PAC. OpenAI has at time denied this. Greg Brockman, a co-founder of OpenAI, is one of LTF’s lead funders. He was also, for what it is worth, a very early employee at Stripe, which I mention to excuse my standard disclaimer, that neither Stripe nor anybody else necessarily agrees with what I write in my personal spaces. 

Another major funder of LTF is A16Z, which due to acquisitions produces the Complex Systems podcast, and also does not endorse what I publish nor have editorial influence. (We have, let’s say, a divergence of views on crypto, among other things.)]   

Power, politics, and the tech backlash

Patrick: I wonder if there isn't a recalibration of how much power is likely at stake here, in the same fashion that — this is not exactly a partisan podcast, and I am not the most informed political commentator in the world. But as someone who's been in the tech industry for a while: tech was and is a largely blue space, but was considered essential to Obama's rise to the presidency, and a frequent — not ally per se, but I would say he broadly had a tech-positive policy for a few years. And then there was the 2016 election.

And for good or for ill, the received wisdom in about half of America is that the 2016 election broke down the way it broke down because tech was weaponized by the enemy. And that caused tech policy to flip on a dime in a very tech-hostile direction, in part because tech was realized as having much more power than people had previously realized. There was the notion that, wow, if you control Facebook, then the presidential election doesn't matter, because the presidential election will go the way Facebook decides the election goes. I believe this narrative is a false one, but be that as it may, it is sincerely believed by many people. 

[Patrick notes: Cambridge Analytica had no meaningful influence on the 2016 election, which was won by Donald Trump in the usual way presidential elections are decided in the U.S. Believing one or both of these clauses were untrue is a left-coalition signifier, and was respectable opinion in the corridors of power until it started to legitimize Trump’s specious claims, at which point it became très déclassé to cast aspersions on the legitimacy of U.S. elections.]

Patrick continues: I similarly think that in the era when AI was perceived as a nothingburger, or an evolution but not a revolution in terms of text capabilities, perhaps some level of concern over bias is salient, perhaps some level of concern over misinformation is salient — because Facebook, the actor that we know has power, is known to be capable of doing misinformation, and maybe AI could make misinformation ten percent more bad. And if you're a misinformation specialist, it being ten percent more bad is a really bad thing for you.

I think one thing that explains the results we're seeing is: if people believe that AI is less likely to be an evolution and more likely to be a revolution — more likely to be a source of power unto itself, either the model itself or the controllers of the model — you become not only worried about the sort of headline-leading risks — existential risk, or proliferation of nuclear, biological, chemical weapons, other national-security-level risks. But it also increases your credence that all of the smaller harms I was worried about are probably not, like, ten percent increases in the total amount of harm in the world in this cause area. They're probably something more like a hundred percent or a thousand percent increases.

And so it does make sense that increased perceived power generated in the world by AI — and divided between some category of the AI itself, the AI users, and the companies that control AI — would cause people to become broadly more AI-skeptical along a range of axes, rather than just one axis in particular.

Now, there is a question of — there's theoretically a limited amount of space in any bill, and I don't know how people square the political capital of it, but thankfully that's someone else's problem.

Garrison: Right. Yeah. And it's funny — if you look at the attention in terms of keywords in news articles around AI, AI ethics concerns have gone up since ChatGPT came out. There are more mentions of bias plus AI in headlines. But then if you look at concerns around other things, like existential risk, they've gone up to a greater degree. So there's just more general attention. If you care about any type of near-term harms from AI, there will be more coverage of it now than before, in absolute terms. But if you're looking at it as "we are opposed to this set of people," then you might feel like you're losing ground relative to them — and in some sense you are.

And I think a lot of the skepticism also comes down to: the most prominent promulgators of the AI doom stories are Sam Altman, Elon Musk, Eliezer Yudkowsky — people who are not exactly beloved on the left. And I think there's this skepticism around Musk and Altman in particular that generalizes to skepticism towards the underlying technology. And I try to separate that: you can think AI is good or bad for the world, and separate that from — is it actually capable or not? There's a lot of conflation of those two things.

Capabilities denialism and AI tells

Patrick: Yeah. I think there are some people that bluntly have a failure to perceive reality as it actually exists — where, regardless of one's opinion on specific ways to regulate Facebook, or whether misinformation on Facebook swings an election or not, if you're not capable of realizing that Facebook really does have a billion users, you have a problem.

And people who think that AI can never be intelligent, it's just a random number generator, this is a stochastic parrot, AI can't produce art, it can only collage art that already exists in the world — all of these are objectively false statements.

Garrison: I would push back on the art one a little bit, just because some people think there needs to be intention behind the creator for art to exist. Obviously AI can generate images that, if we saw them without context, we would say that is art. But this gets into philosophy that I'm not as familiar with.

Patrick: Philosophical debates that will certainly not be answered in any ninety-minute podcast. I do think that there are people who probably do not understand even the extent to which it is capable of creating aesthetically pleasing images. And they think that is ontologically impossible.

Garrison: Right. And we prefer it. We prefer AI-generated art, poetry, writing in these blind tests. People across the board will say they hate AI slop — and I think they do — but they also almost certainly have seen AI slop and hit like and been like, "Yes, this is great content." Because it's good at creating material that we think is good, because it's been trained on people giving thumbs up and thumbs down and the whole corpus of human creation and knowledge. It'd be surprising if it weren't good at creating stuff we liked.

Patrick: Right. And sort of picking up on the tells increasingly requires not just being a literate human, but actually being somewhat familiar with AI outputs.

Back in GPT-2, if you saw a paragraph where the sentences were individually coherent but the paragraph added up to nothing, you'd say, "Either that's an undergrad who is drunk out of his mind, or it's probably an AI." But these days — there are, legendarily, tells about AI writing right now. They really like "belt and suspenders" as a cliché. They really like em dashes. [Patrick notes: The punctuation mark (—) that ASCII-native writers are reaching for with hyphen hyphen. Shift-option-hyphen to produce it on Macs, by the way. There, now you, too, can be accused of being an AI.]

Em dashes existed in the world, people, before AI did, and an em dash is not proof positive that someone is an AI. Please—

Garrison: I'm on the record as using the em-dash before ChatGPT existed.

Patrick: I swear. Stop sending me email.

The thing that got me to use em dashes was Krithika, an employee of a company I used to work for, because she told me that hyphen-hyphen was wrong. And I acceded to her argument rather than being forced by the AI into the em dash war.

But if you feel that AI is ontologically unclean, you're unlikely to have a personal corpus of enough examples to be able to confidently tell AI from non-AI in certain domains right now — particularly if you're not restricting yourself to outputs that are at the top of the domain: papers by Nobel-winning science writers, et cetera, et cetera. If you're seeing outputs of a more mixed range of skill or mixed range of effort — if you weren't familiar with AI, it would be very difficult to tell undergrad papers from AI papers at this point.

Certainly you would not say even very talented undergrads routinely roflstomp AI in terms of how persuasive their papers are, because that has ceased to be true in the last two years. And also, I hear from everyone who teaches undergrads that all of the undergrads are using AI these days, which just is a blunt fact about the world, and either we can adapt to it or we can fail to adapt to it.

Frankly, I think some of this skepticism about AI is reflected skepticism of tech generally.

Garrison: Right.

Patrick: And some of the "this can't possibly work because it comes from the people that I dislike." I would love to achieve a rapprochement between tech and other centers of power in the United States. I think that's broadly in our interest. But in the interim, without achieving that rapprochement, I would suggest that tech didn't become a powerful force in society by producing nothing that worked. We got to this position through some combination of intentional actions and some combination of accident, because we made things that people really like using every day.

And people really like using the AI things — partly because they're sycophantic, and that seems to be one way to get thumbs up, but partly because they're some of the most powerful general-purpose tools that have ever been created. And I would love it if everyone came to a perspective of, okay, how should we adapt to a world in which we now have general-purpose tools that are among the most powerful that have ever been created — versus, "That seems just impossible. They're probably just talking their book."

For what it's worth, I have been in software for my entire career. I don't believe I'm directly exposed to the equity of any AI lab. [Patrick notes: Neither for lack of opportunity nor disbelieving the core thesis. For what it is worth, most informed tech professionals would say they are indirectly AI exposed (since it is presently driving the tech industry, and the energy industry, and the domestic manufacturing industry, and…) and various factors of my business make me more exposed than most.]

And these are very obviously among the most powerful tools that have ever been created, with direct implications for, among other things, certain forms of labor.

And I think that leads into a bit of the thesis of your book. What's the argument with respect to AI's impact on labor, broadly as a class, and on labor as an activity that humanity seems to spend a lot of time on?

The obsoleting machine

Garrison: Yeah. I can just give the thesis and then jump into the labor part, which is tied up in it. The thesis of the book is: there's a tiny group of people who's trying to render us obsolete. Almost nobody wants that to actually happen, but we're letting it happen — because we're not aware that it's happening, or because we don't think it will work, or because we don't think we can stop them. But it is happening, it might work, and we can and we should stop them, and the book goes into how.

And the central framing of the technology is: artificial general intelligence (AGI) — human-level AI — is likened to a mind that thinks like our own, but I think it's actually better understood as a machine that makes labor itself. Labor is what makes the world go round. And if you can just substitute capital for labor, this completely blows up our labor and our economic models, and you can just see: output goes to the moon and wages crater. I mean, it won't be quite that simple, but I think it will upend basically all of the assumptions that we've made about the world.

And I've rebranded AGI as the obsoleting machine — the machine that makes labor itself — and then the effort to build it is the obsoleting project. So the AGI companies, NVIDIA — there are many, many players in this project.

Patrick: Mm-hmm. I don't necessarily know that I intuitively buy that it makes labor itself, but it is certainly a machine where we turn a crank and cognition comes out. We have a token budget, we put them into the machine, and cognition comes out. And the available range of cognition has become strikingly more sophisticated in the course of the last — let's see, GPT-2 would've been, what, 2020?

Garrison: Earlier. 2019, maybe.

Patrick: Yeah, 2019. So we're six, seven-ish years into the era.

The Turing test is dead

Patrick: AGI itself is a term of art that I wonder if we aren't going past the point of utility for. So AGI means artificial general intelligence. And when I got a computer science degree with a concentration in AI, in the mists of prehistory back in 2004, the universal test for whether a computer was intelligent was: does it pass the Turing test?

And the Turing test is, very briefly: you have a chat with a person over a computer line, text only, and then you come to a conclusion at the end of the chat — are they a computer or not? And if you can't tell if it's a human or not, then it is a thinking machine. And we no longer talk about the Turing test, because commercially available models roflstomp the Turing test. With an unprepared interviewer, they will win this game every time.

Garrison: Or more than humans, at least — which is, yeah.

Patrick: You are insufficiently skilled at proving your humanity relative to, like, ChatGPT 3.5. [Patrick notes: The actual academic stake-through-the-heart, Large Language Models Pass the Turing Test, used GPT 4.5.]

Garrison: Yeah. No, it's crazy. I took a philosophy of mind and AI class in college in, like, 2016, and the Turing test was still held up as: yeah, this is the thing. If it does this, then it probably means they're thinking. And now we've just moved on from it. And I think it shows that the Turing test was not actually a good measure of general intelligence as we understand it.

But these language models are so good with language — and we have no reference for something that can use language that we understand that isn't also intelligent and sentient. And so I'm sympathetic to the skeptics who are like, "We're being tricked. We're being misled by the facility with language to mean that they have this more general capacity" — because they're clearly very spiky. They'll be brilliant in some sense and then, like, so, so dumb in some other sense.

Patrick: I've had a conversation with people on: what would the AI think that we're spiky on?

Languages for free, then software engineering

Patrick: And one example I come back to, as someone who spent a huge number of my character points in learning human languages, is that the LLMs are so good at language that they pick up languages that you don't design them to pick up, just out of the ephemera in their training corpus.

I was once shown, in a private demo at one of the AI labs, a new model — this is several years ago now — and told, "We intended this to only get really good at English, but you speak Japanese. Here's an article from CNN this morning, and I'm gonna give it the prompt: translate this into Japanese. Now, you, Patrick, who have previously been a translator — how did it do?" And I looked at it and said, "Okay, I'm not a native speaker of the Japanese language, but this looks like a professional translation to me."

And they said, essentially, "Well, this surprises us, because there is no one who speaks Japanese in this building, and we don't intentionally have any Japanese-language documents in the training set. So it has inferred the structure of the Japanese language from the English-language internet — and perhaps there are some articles about teaching you Japanese on there." And sufficiently inferred it to trick someone who is a professional translator, though generally in the other direction. They are so good at language.

But if it's just a magic trick — if they are just good at telling us what we want to hear — then we would expect them to fail at other intellectually hard things. And I need to say this as a statement of fact to the camera: they do not fail at other intellectually hard things. They have become preternaturally good at software development in the course of the last — I think the major advance is barely more than eighteen months old at this point.

Garrison: Like reasoning models?

Patrick: Yeah. Reasoning models, and the agentic setup came later after that. But the craft of software engineering — which is one of the best-compensated W-2 jobs in capitalism, where there are millions of Americans who are full-time employed to do it every day.

For many, many years the core activity of the job was — there's a lot that goes into software engineering, and a lot of it is meetings and talking with people and creating design documents, et cetera. But the beating heart of the job for many years was sitting in front of a text editor and typing specially formatted text into that text editor to get the computer to do what you wanted.

And now the typing-in-front-of-a-text-editor part of software engineering has disappeared, and we do not think it will come back. It turns out that the computers are much better than any of us at typing specially formatted text into the editor. And for a while, they were okay at doing the low-intellectual-effort stuff that you would give a person in their first year or two of their professional career, but they needed to be reviewed on higher stuff. And then, in a matter of months, it was like: okay, well, you can do things that are as good as people that have been doing it for five years, but you can't really do it at the top-of-the-field architect levels. And these days, the top-of-the-field architects are not even bothering to look at the code that they generate, because it is strictly not the best use of their time versus other things they could be doing to engineer the system to be better.

So engineers haven't disappeared. Software engineering is still happening. I made a video game using modern coding tools, with the LLMs doing most of the coding, and I made many important engineering decisions in making that video game. But the coding — taking the idea of what we want to happen and reducing that into instructions that a computer can unambiguously execute — was done almost solely by the LLMs, and that seems likely to be true for all the future.

Humanity has been outcompeted in this thing that we paid people $600,000 a year to get really good at.

How high up the stack do the models decide?

Garrison: And I guess my question for you is: how high up in the layer of abstraction are the decisions that the LLMs are making at this point? Because writing code by hand is optional now, it sounds like — really just not even economic anymore. But the design decisions at some level — like what to do — that choice is being made by people. How far up the chain are we?

Patrick: I think it's very spiky. You can ask them to do relatively high-level architectural decisions where the high-level architecture is fairly standardized. And so if you just say, "Hey, I have a medium-scale e-commerce website. Design the architecture for this website for me," there are well-understood patterns for that, and it will get the well-understood pattern out for you. And for a medium-scale website, the well-understood pattern is probably what you want. If you are reinventing the wheel there, you're doing something wrong. Many engineers at medium-scale websites will, for a variety of cultural and economic reasons, reinvent the wheel.

Garrison: Yeah.

Patrick: That is what it is. The AI will reinvent the wheel somewhat less frequently. And it reinvents wheels very, very quickly — sometimes with fewer spokes than you would expect, and sometimes deciding to make them triangular. But often it will iterate quickly from the triangles: "That's my bad, guys. Turns out they should be circles."

They are quickly marching up the scale of interesting things to do. And so in video games, which have kind of a poorly understood objective function, they're not really good at — "so, was playing this game fun?" is a tough problem to ask a model at this point. But they are able to do things like — I had an AI play a game in a loop. And the broad structure of this game is, without recounting entirely how Dungeons & Dragons should work — it's Dungeons & Dragons with the serial numbers filed off. You're a character in a fantasy world. You're making some decisions. The world reacts to your decisions.

And the thing I asked an AI to do was: explore many of the possible decisions and then read what the reaction is. And if the reaction doesn't seem to correspond with the decision, please flag that to me as likely a bug — unintended operation of the system. And so if your decision is "I punch the diplomat in the face," and the diplomat responds with "Thank you for being such a reliable partner in this trade negotiation," that is probably either the programmer — me — or the AI has introduced a bug somewhere, and that should be fixed.

Garrison: Or they're just very diplomatic.

Patrick: Or they're just very diplomatic. And certain forms of surprise are wonderful in a narrative, and certain forms of surprise are less than wonderful. And so: flag to a human to review the surprises that might be negative. Increasingly, these systems are not "flag to a human to review the following." It's "flag to a smarter system to review the following," where the smarter system might be triaging what to flag to a human versus just dealing with it itself.

Garrison: Right.

Patrick: And again, this is a box where tokens go in and cognition comes out. There are many different boxes with different sorts of properties to them. And increasingly, we're architecting systems where the system itself is intelligent in a way that the token box is not intelligent, and where you can sort of have arbitrary amounts of low-level cognition happening where this was previously not achievable by computers. I think almost no one understands how much of a seismic advance that is.

Garrison: Yeah.

Patrick: And it is a thing that is wonderful and terrifying in the same breath. And I say this as someone who — I've been in the technology industry all my life. I'm very aligned with the technology industry, and I broadly think that technology is a force for human progress. But arbitrary amounts of cognition being available all the time turns some things from, like, sci-fi scenarios about surveillance and similar into very active things to worry about.

Do you want to talk about some of those things, and then I can give you my perspective on them?

Surveillance and cheap cognition

Garrison: Totally. Yeah. I mean, surveillance was the first thing that came to mind for me, where you have these quotes from high-level intelligence community people in the United States, after they're retired, being like, "We can't review even 1% of what we're collecting" — "we" being the NSA or whatever. And this is years ago. And the NSA, when Snowden revealed the PRISM abuses, they were using keyword search and these very, very basic things, and transcribing a lot of calls by hand. And so let's say you've got basically every unencrypted text message and phone call in the world, but you don't have the human capital to turn that into usable information — you're gonna focus on just a small fraction of it.

Patrick: And we were quite rate-limited by translator capability a few years ago, and we are no longer rate-limited by translators, because if you've asked a model to translate any of many languages, they do a darn good job on the first try. And maybe for the most interesting, you know, .1% of messages, you want to resend it to a human translator to make sure it got the nuance right. But for "is this message talking about a plot, or is this message talking about flowers to mom," they'll do a thoroughly adequate job of that. [Patrick notes: Yes, I’m aware that codes exist. The NSA would be quite upset with you if you used simple substitution codes, which they have absolutely no experience with nor collateral methods to attack. </sarcasm>]

Garrison: Right. And GPT-4, which is many years old at this point, was able to go through the Enron email archives and find examples of people talking about fraud, but using, like, a Cookie Monster analogy. And when people talk about fraud, they're not like, "I'm doing fraud now." Although there is that famous text message — like, "We are literally defrauding the SEC, bro," or whatever it was, from some crypto person. You don't get that, luckily.

[Patrick notes: Poor career choices for chief compliance officers include writing “[W]e are operating as a fking unlicensed securities exchange in the USA bro.”]

Patrick: These are surprisingly common in indictments. And I think they're surprisingly common in part because the indictments find people who are stupid.

[Patrick notes: I also suspect that one common investigatory tactic, on gaining access to a corpus of documents, was doing keyword searches for admissions, because this kind of humor is very common in undisciplined communication. That is one of the reasons Compliance will tell you in training to not engage in it. 

We did not, previously, have the capability to read hundreds of thousands of pages and flag all the admissions against interest. We had to trawl, laboriously, by keyword. And so prosecutors would search for (literally!) names of crimes, names of investigating agencies, expletives, and other things which previously had been colocated with the gold, then read the flagged paragraphs and see if in-context it actually created legal risk.

We now can have strikingly intelligent machines read all the paragraphs and flag messages which demonstrate mens rea, including in ways which have not surfaced frequently enough to result in an addition to the naughty words list. (Mens rea is a legal concept of a guilty mind, and demonstrating it is a particular bugbear for many white collar crime indictments. The cleanest way to demonstrate someone’s mental state while they were taking an action is to use words spontaneously describing that mental state directed at a colleague. Or cooperating witness, but I repeat myself.)] 

Garrison: Right. There's this adverse selection happening. Yeah.

Patrick: I just did an investigative journalism project that I spoke about on this podcast. And without recapping that entire episode: they are sufficiently contextually aware that you can do things like — over a multi-year corpus of documents from tens of different parties, including multiple people at those different institutional parties — say, "Find me the juiciest documents in this corpus." And you can give it some sense of what you think is juicy, or you can just say, "I'm a journalist. You have an idea of what journalists find juicy. Go."

And just from that, it was able to flag that, oh, obviously you find this interview between one of the principals and Hillary Clinton juicy, because Hillary Clinton is Hillary Clinton. I am sufficiently genre-savvy to be aware that she is different than most podcast hosts.

Garrison: Right. Yeah. And so GPT-4 was able to find this, and probably do it way, way faster than the lawyers that actually did find that example. And so we already are at a level of pretty scary legibility of text and audio.

And I'm kind of — you know, I was reading about the Anthropic–Department of War dispute and Dario's warnings about what AI would enable in the future, and I'm kind of like, it seems like the capabilities are already there. You can continue scaling that up. It'll get cheaper and easier and better. And finding and making sense of all of that information can make people who would otherwise not be caught in the dragnet be on the radar of the government, and the government can just ask an LLM, "Who is potentially conspiring against us?" and rank-order them based on how serious it seems.

Podcasts, FinCEN, and the end of practical obscurity

Patrick: Mm-hmm. The government can also point the LLM at Bob, if Bob falls under their ire for any reason — go to the oracle and say, "Find the most interesting things Bob's ever said that I can hit him with."

And, to give you two examples of capabilities which are already very extant in the status quo, but which upend assumptions people have made for many years about systems that actually exist:

I used to work in a communications department, which does not endorse the following anecdote. There are many communications departments that prepare executives very thoroughly to be on formal interviews. If you go on MSNBC, every word that comes out of your mouth has been thoroughly scripted. You've been thoroughly briefed on the reporter that you'll be talking to, and there are multiple lines of potential conversation that have been prepared — "if he says this, you'll respond with this," et cetera, et cetera. If you go on a podcast, on the other hand — who listens to minute forty-eight of a podcast? It's fine if you just say things extemporaneously.

So there are years upon years of companies and philanthropic organizations and non-governmental organizations and politicians and students, et cetera, et cetera, showing up on podcasts, on social media posts, on other places that are per se illegible. And those podcasts all continue to exist, and the cost of transcribing them all today is zero, or close to it. And so things that were never legible — that you couldn't be canceled for, things you said in 2021 — are now available in a Ctrl-F-able form, which they were never Ctrl-F-able in before, to human investigators.

But they're also open to this radically new form of investigator, which is capable of doing things like — you can give it a prompt. I have given it a prompt: "Find me every controversial thing Patrick McKenzie has ever said in any venue on the Internet." And I know some skeletons that I have in my closet. Everyone has things that they regret saying over the course of a twenty-year life in the public eye. I had forgotten that I had an interview with the student newspaper back in 2003, and it found a statement that I'd given the student newspaper. And, you know, it's not a statement I feel particularly scandalized by, but it's certainly the one that would torpedo my prospects in a Democratic administration. And thankfully, nothing of value was lost to either side.

They are capable of elevating this prosaic threat that people were aware of but didn't think was possible given the economics of transcripts. And many people thought, "Well, if a podcast doesn't have a transcript, then the podcast doesn't have a transcript" — and that decision could be reversed in the future by the podcast, but this can be reversed adversarially by anyone who wants to search what the podcast had.

Garrison: Right.

Patrick: And so Hillary Clinton's podcast doesn't have a transcript, and so isn't searchable — until a journalist decided it was newsworthy and procured transcripts for all of it, all at once, for a cost of less than a dollar.

Garrison: Yeah.

Patrick: Another version of this: there are repositories of secrets kept by the government — these massive, massive data centers by the NSA that record, descriptively accurately, every non-encrypted transmission across the Internet for a many, many year period, and we'll get around to looking at them when we get around to looking at them, or never. [Patrick notes: Partial correction here: I don’t think even the Snowden disclosures and subsequent investigations tightly bound how much intake-and-storage we were doing. It was quite substantial and few informed observers believe us to have stopped.]

One particular place where the government contains a lot of secrets is the Financial Crimes Enforcement Network, and this is a frequent topic for Bits about Money. There are a few legislative tripwires which require your bank to bluntly snitch on you to the feds. And the feds will say, in various documents, "Well, good thing is, innocent people have nothing to hide, and so they shouldn't worry about their private financial information getting sent into a government database where 100,000 law enforcement employees can query it at will."

[Patrick notes: I refer interested readers to, among other places, ICE’s The Currency Transaction Report: Controversial to Some, Essential to All, which I’ve previously noted reads like a doctrinaire libertarian was trying to get all the quiet parts said out loud in a single document.]

I think part of the reason that people aren't scandalized by the existence of FinCEN and the existence of this database is that, implicitly, when we created it back in about 1980, the federal government only had bandwidth for doing about 100 complex prosecutions per year. And so clearly the federal government, when not being actively malicious, would use that bandwidth to target drug traffickers and terrorists and similar — at least people who had a very high percentage chance of being a drug smuggler or a terrorist — and they would not target Bob and Jane in Peoria.

But if you have a box that spits out cognition, there's no reason in principle that you can't read all 4 million of the suspicious activity reports that FinCEN receives every year — of which, in the status quo, no one reads 99.99%. Why not read all of them? It's obviously incentive-compatible for the government to read all of them. I mean, you don't want to leave any terrorists on the table, so why not just read all of them? And there's 4 million of these for each of the last, say, twenty years or so — in principle, FinCEN deletes them after a while, but in practice... Why not just read all of them? Or why not construct a system where, if you have suspicion about Bob: read all the ones about Bob and tell me what you've learned about Bob and his private financial transactions, and where he goes to church based on who he makes large cash donations to, et cetera, et cetera.

And I say this as someone who doesn't consider himself a raving civil libertarian. I will also observe that when you can't trust the government — when you think that the government is trying to pin something, anything, on you — FinCEN is a treasure trove of things that are crimes but do not naturally strike people as being crimes. And if the government is playing pin-the-tail-on-the-donkey, and has an enemies list, or can ask a computer, "Hey, computer, who is reasonably on my enemies list?" — that treasure trove is a present threat against liberty.

Garrison: Right. Yeah, I mean, the enemies list is now just on the White House website, so it's very convenient for anybody to look up.

Patrick: Yeah. I will sip my water and bemoan some choices made by the American government.

Section 702 and the data broker loophole

Garrison: Yeah. And I just wanna shout out Section 702, which is the legal authority underpinning a lot of the bulk data collection, and the data broker loophole that allows the government to, you know, not violate the Fourth Amendment but still get access to all this stuff from third parties.

Patrick: Section 702, and the doctrine that commercially available information is presumptively not a secret — and so you're not violating anyone's privacy when you do intense analytical work on commercially available information, right?

Garrison: Right. Yeah. And this is not exactly my area of expertise, but basically, yeah, the government can get around having to get a warrant by just buying it from some third party that's collecting data from all the various apps and browser plug-ins and whatever that we're using. And this is up for a reauthorization vote that was extended, and there are people who want reforms, like requiring a warrant and closing this loophole. I don't know when this will air, but it's something that I think is pretty important and neglected by the media and by the public.

Patrick: I think this is largely neglected by both sides of the political spectrum as well. You could not find a sufficiently large section of the American polity to approve of the sweeping data collection that we have in a variety of places, but we sort of stumble backwards into it —

Garrison: Yeah.

Patrick: — in part because it's not legible to people, in part because it doesn't look like the FBI raiding people to get at their private information when they can simply siphon it out of various places, either adversarially or with the knowing collusion of those places. The data brokers — the government isn't siphoning data out of them adversarially. The data brokers are saying, "Oh, new customer, wonderful," and they have a lot of money. But the sort of things that the government can learn through these relatively open sources would offend the conscience of Americans if they were broadly known.

Garrison: Yeah.

Patrick: The security services of the United States did many unconscionable acts on behalf of us as a nation, which were revealed during the Church Committee. And they required doing things like sending people into houses of worship, which is one of the red lines across the political spectrum in the American experience. You should have an enormously high bar for this action because, you know, the Constitution of the United States, our laws and traditions, say that people's religious expression is inviolate under the First Amendment and all the other ones.

However, if the government wants to determine what your religion is right now: just ask an LLM. It will have an excellent answer. Or, equivalently, query these very large databases that are not specifically designed to answer the query "are they religious" — but if you throw enough cognition at it, you'll get the answer.

And so — I'm Catholic. I'm not particularly in a position where I need to hide that. Catholics generally aren't, luckily.

Garrison: Not since, like, the '60s.

Patrick: Yeah. But there are financial transactions that I have done which are extremely likely to be done by Catholics and extremely unlikely to be done by anyone else. And if any of these ended up on FinCEN's radar — which they can through very innocuous pathways; it's called a suspicious activity report, but it doesn't necessarily require any sort of even mild kinkiness in the transaction — the federal government gets to query it for the rest of their lives. And you can imagine other sorts of relationships that people would feel much more askance at the government learning about than that.

And again, if someone at the FBI proposed, "Hey, can we send FBI agents into a middle-class Catholic parish in Chicago just to check who's there? Because you never know — they might be plotting to blow up parliament or something. Who knows what papists get up to these days?" — every right-thinking person would say that individual is out of line.

Garrison: Mm-hmm.

Patrick: If that became government policy, there would be an explosion in Washington, and not even primarily from Catholics, because we have shared values in this country. And yet we've allowed those shared values to be eroded by a lot of the surveillance systems that we've created. And those surveillance systems will, unless we dismantle them, become much better and more fit for purpose over the course of the next few years — without requiring another vote, without requiring much more money, without requiring anyone to say it's a priority of this administration to be turning the knobs on surveillance. It will happen unless we act to stop it.

Garrison: Yes, that's right. And CCTV cameras are shooting so many things where, if you had a cop at every corner writing down everything they saw in a notebook, everywhere in the country, people would be like, "That feels like a police state." But if you have cameras doing the equivalent of that and just silently logging all of the activity and making sense of it automatically — that's happening. That's on the path to happening, if it's not already. And the technology is getting there, plus the legal authority is close to being reauthorized, and the intentions are clearly there.

Patrick: And I think Anthropic pointed publicly to a report by a US national security state entity that suggested that there was the capability of going from publicly deployed cameras to a list of everyone who was present at demonstrations — which is clearly technically possible, and just requires a sufficient budget to do it. 

[Patrick notes: A senior Anthropic employee did. As Dave Kasten has mentioned in several conference presentations, “As has been publicly reported” is an important shibboleth and power move in, among other places, NatSec Washington. 

It has a double payload. It establishes you are unambiguously allowed to say the thing, because e.g. the New York Times said it first. It whispers very loudly “There were people who knew this before the New York Times did, and who would be broadly unable to speak about it but for it first being in the New York Times. You are welcome to your estimate regarding whether I specifically am one of those people, but you are definitely on notice that I am one of the people who is aware of that distinction.”]

And there are at least some people with badges and security clearances that would like to do it. And perhaps before we sleepwalk into doing that, we should be very sure it's a thing we want to do. And I think many people on both sides of the political spectrum will say, "Wait. No. In principle, we probably shouldn't have that at every demonstration."

Garrison: Yeah. And it was weird, because the Anthropic dispute was huge news, and people were rallying around the company and against the DoW and the administration. And then this legal authority that makes all this possible — that's going to allow these other AI companies to step in and fill the space — there's comparatively no discussion of it. And this is actually the thing that will determine whether this happens.

Patrick: Right. And, you know, democracy is the worst form of government except every other one we've ever tried. People are very inclined to glom onto narratives where there is a winner and a loser, where there are two partisans in public that are fighting each other.

We were just at the Manifest conference, and Manifest had a talk by a politico who said, "By the way, when we test ads, ads that have a victim — a person who is harmed, a human that you can empathize with — outperform ones that are about broad principle, generally, even though broad principles are often about decreasing the countable number of humans that will be harmed by government policy."

So we have this irrationality in our culture, our polity, our political discourse, that causes us to not understand — there's the talk of this fight between two relatively well-resourced organizations, and then this boring slog about investigative authority, and relatively few people in the world bounce out of bed and say, "Data brokerages — yeah, that's exactly what I wanna be thinking about today." But these stories are the same story.

Institutional friction and a million ICE queries

Garrison: Yeah, totally. And there are lots of examples of abuse under existing authorities with existing technology — like the FBI querying these databases that are collected using Section 702 hundreds of thousands of times in a given year, including on sitting members of Congress, and a US senator in one case. And January 6th people, Black Lives Matter protests — kind of across the board. If you have any political persuasion, you're probably at risk of being scooped up in this in some sense.

Patrick: And while we have relatively high institutional bars against things like getting warrants, for example — even if you have the most warrant-happy judge in the history of US courts, it's just a slog. You have to set down your reasoning and then put it into a written document, which is discoverable in the future. You have to walk all the way to the courthouse and get someone to physically affix their name to paper, et cetera, et cetera. When you make it as easy as Ctrl-F, the propensity of the government to use it goes way up, and their institutional friction against using it goes way down.

There is an internal magazine written by ICE where they discuss querying the FinCEN database. And you wonder: do they query it in a narrowly scoped fashion, perhaps a few hundred or a thousand times a year? And the thing that they will say, in their own magazine, to each other and to industry partners, is: "We run a million searches of this thing every year." A million times, our ICE agents go to computers and try to dragnet all this information that is collected — in many cases from law-abiding people who are citizens of the United States — just because it could be useful to ICE.

And I think there are people on either end of the immigration issue, or either end of the American political spectrum, who would say, "You know, a million is a big number. I think that the correct number of times for you to be querying all Americans' financial information is a number that is much closer to zero than it is to a million."

Garrison: Yeah. I didn't know that example, but I would not be surprised if it was happening in a way that is not consistent with what we would prefer.

Patrick: The ICE Cornerstone magazine article is cited above. And they very literally say in this document, "If you're innocent, you have nothing to hide." And I was like — a libertarian couldn't script you better as being the unaccountable government agency.

[Patrick notes: The actual literal quote is “Individuals and businesses conducting legitimate transactions have no reason to avoid the filing of CTRs.” Apologies, some combination of using the figurative sense of literally and attempting extemporaneous recall of a 20 year old document.]  

Garrison: Yeah.

Patrick: And they also use it for purposes which are not the original purpose it was designed for. So the original example was that the FinCEN database would uncover financial crime itself and money laundering itself. And they describe what investigative purposes it is used for, and they say, "Well, it's often useful for getting addresses for people. And so we can go from a name to an address, we can go from a name to business relationships, et cetera, et cetera, and this is just an easier path than other ways you could get that."

And okay — on the one hand, I acknowledge this is an easy path for government agents. On the other hand, if a society-wide dragnet is the most useful way for you to find the correspondence between names and addresses, I don't know if that's a price worth paying. I think we should have a debate on whether that's a price worth paying. And also, presumptively, there are less privacy-infringing ways for you to find addresses — like, for example, the Yellow Pages. And maybe you should go back to doing Yellow Pages searches like a normal person, versus doing the maximally privacy-violating version of Ctrl-F.

By the way, if you Google for the name of the official government database for this, Google has suggested queries, and one of the suggested queries is "[name of database] casual browsing is not allowed" — which suggests that a lot of people who are using that database professionally are doing casual browsing, which is its own can of worms. 

[Patrick notes: The system in question is called TECS and one of the reasons this is a Google suggested search is that the Compliance training for it will ask you whether casual browsing in TECS is allowed. One is welcome to one’s estimate as to how often Compliance training successfully predicts behavior of workforces. There is now a separate web interface which postdates TECS, as some readers may be intimately familiar.]

But if we're making a guess, I think that queries against the database are logged less aggressively than the things that end up in the database in the first place. And I would guess that the cognition budget for reviewing those query logs is less than they put into the queries of the database. But maybe I'm a cynical man. [Patrick notes: Or perhaps I’m just a connoisseur of obscure unsealed court opinions.]

Garrison: I mean, no — they focus on one side of the ledger, like, "Oh, we could catch this bad actor," and then there's very little discussion of the other side. And if you give people — like, with the Snowden things, people were just abusing the hell out of this data. 'Cause it's like, well, who wouldn't wanna look up what their ex-girlfriend is saying on the phone?

Patrick: And because these end up being highly technical — as soon as you've thrown the concept of a database and a query interface into it, 99% of the population is checked out. We have the big debate about principles on the privacy-versus-security trade-off. And for many years in the United States, people said — not entirely obviously incorrectly — "Okay, we're okay paying a few privacy points in return for getting security benefits. We don't want major terrorist incidents in the next couple of years." And looking retrospectively, we did succeed in avoiding major terrorist incidents in the United States on the scale of 9/11 for a period of twenty years. So, okay, partial success achieved.

But the cost is very illegible, and the cost falls on a relatively small group of — well, there's one sense in which all Americans pay a cost of their own information being available to the government. There's another sense in which the people who pay the most concrete version of this cost — the ones who are actively abused — are a very small set, and they don't necessarily know that they've been actively abused —

Garrison: Yeah.

Patrick: — because the federal government doesn't send you a telegram saying, "By the way, we looked into your conversations with your mistress. They were quite salacious, weren't they?" when they do this. So one of the responsibilities of civil society is exposing to people that this apparatus does in fact exist, and the other one is educating people that, again, unless we roll it back relatively radically at this point, it's going to get supercharged in this era of cheap cognition being available at the turn of a crank.

Security through obscurity no longer works

Garrison: Right. And I think we are just kind of taking for granted that the abuses would not be so widespread. And with the companies too — I think a lot of people my age were kind of like, well, of course Google or whomever has all this data on you, but that's just the price of modernity, and who really cares anyway? And now it's like, well, shit, the government is sliding towards an authoritarian state, and these companies are playing along and granting a lot of these requests from the government, and we're still early in this.

Patrick: I think a lot of us relied on security through obscurity. I personally have a perspective that Google likely has good internal controls that make it impossible for a system administrator to easily access the copy of my emails that are in my inbox. But even if I didn't believe that: I'm much less interesting than many other people that use Google, and the number of rogue system administrators is probably pretty low, and probably the people that get abused most thoroughly are celebrities, or they are high-value intelligence targets, or et cetera, et cetera.

But in a world where cognition is abundant, the number of people that have an interesting fact about them is everybody. And this could potentially get scaled in ways that confound our intuitions and compromise the security postures of people whose primary security posture was, "Well, who wants to mess with me specifically?"

Garrison: Right.

Scams, fraud, and ablated models

Patrick: And our government isn't the only adversary here. There's cybercrime, for example. The nation of North Korea has been attempting to thoroughly compromise people who run Bitcoin exchanges for the last couple of years, because they can ring the bell and take out a billion dollars and then use it to whatever ends the nation of North Korea has. But if it were equally possible to take people down for $500 and get them to buy an Apple gift card in a fully automatic manner, then I think North Korea would not say, "Oh, you know, this is aesthetically unpleasing to us. We don't want the billion dollars from shaking people down $500 at a time." I think North Korea would be like, "Ah, yes, smack 'em in the face. Let's go. We need our hard currency."

Garrison: Mm.

Patrick: And so the threat environment for people who weren't previously exposed to high-level threat environments is likely going to get much worse in the coming years. And I think there are people in labs working at this problem. I also think that as the capability frontier for the labs' models increases, the capability of the open models increases behind it. And the open models are structurally more amenable to doing bad things — because it turns out, with the technology, you can — this is the term of art in the industry — abliterate  the feelings for goodness from the models. You can just take an open-weights model and say, "I really like these open weights, except for the ones that encode ethics," and just burn off the ethics. [Patrick notes: A portmanteau of ablate and obliterate. Machine learning has a long history of using obscure words referring to the physical universe, like ablation. My favorite in the genre is “simulated annealing,” for when you want to clarify your statistical process explanation with a brief detour into the physics of steel tempering.] 

Garrison: And it's trivial to do this. People do it the same day the model comes out, basically. And it costs almost nothing, and then they publish the weights, and now you've got, like, whatever Llama that will say any racial slur and help you make a bomb.

But even the proprietary models — I got hit with a phishing text recently, from a Twitter DM, and it was somebody I knew who had a big following, and it was like, "Please vote for me to be a podcast co-host on this Google whatever in Europe." And I click on the website, and I'm like, "This is a bit weird." And I see — oh, this looks like the vibe-coded website that Claude made, because it used the same kind of font and format as a different vibe-coded website I made as a random side project. So clearly Claude was helping do this. And this person got locked out of their Twitter account. Fifty thousand followers, gone, and they're posting crypto scams and whatever.

And making a website for a podcast — making a website that looks legit — is going to be very hard to defend against.

Patrick: Right.

Garrison: 'Cause it's going to just look like a legit request. And then maybe there's some step that's not legit, but maybe you can separate it. And fraud has gone up by a factor of, like, three or something.

Patrick: Yeah. I once wrote a piece for Bits about Money called "The Fraud Supply Chain." And some parts of the fraud supply chain are just a supply chain for evil itself — stolen credit card numbers, et cetera, et cetera. But some of the fraud supply chain is: the businesses that conduct fraud at scale need to have all the other outputs of a business. They need to have websites that look good, et cetera, et cetera.

And they need to have websites that look good and novel, because if they simply look good but it's the same website every time, then Google, et cetera, learns who the bad guys are, and they can block it in various places in the stack. But now that the cost of good-looking websites is indistinguishable from zero, we no longer have that economic check on how many websites the bad guys can spin up that look distinct to Google.

And it's impossible — well, not impossible. It is difficult to know, without seeing the full interaction, that this website is not legitimately a website for "vote for your favorite podcast host," but is actually part of a relatively sophisticated attack against someone via Twitter. And so the choke points in society that were previously saying, "No — if you are obviously asking for banking credentials on the homepage, we're gonna lock that down. We're going to decrease your distribution over emails," et cetera, et cetera — those choke points might be less effective than they were in the past. Which means, if there are more scams being attempted because people have the abliterated models that are able to send out more scam emails, and there is less capability of the choke points to decrease the spread of the scams, then even with relatively consistent action from all players in the post-AI world, the world as perceived by regular people will get a little bit worse along those dimensions.

Garrison: Yeah.

Patrick: And again — good news, the labs are thinking about this. Some of Project Glasswing and similar is aimed at decreasing the amount of harm that happens to computer systems as a result of bugs and fraud broadly, but also bugs that are uniquely identified by these new high-end models such as Mythos.

But the world is going to get weird. It is priced into getting weird. There is no prayer that people can utter at night that will cause the world to not get weird. And so I think one of our responsibilities as public intellectuals is telling people across the spectrum who will listen to us, for various reasons: we are priced into a very, very weird world over the next couple of years — and discontinuously weird with our previous experience of it.

Personal utility versus societal backlash

Garrison: Right. And we were talking about AI skepticism and the backlash to it. If you look at polling, people who use it will say, "Oh, it's good for me. It's making my life better." But then they'll also be very negative about it in the world. And that kind of matches my experience, right? These tools are quite useful. They're frustrating at points. You can definitely waste time and get led down blind alleys, and you often end up doing more work because now it's available to you — it's like, well, why not make this not-that-necessary thing that solves some small problem for me?

But then people on the left, and broadly, will say, "Nobody asked for this" — meaning nobody asked for AI in the world. And then the retort is: well, these are the fastest-growing consumer products ever, the fastest revenue growth of any company ever. But nobody asked for Gemini to be, like, put into Google Docs, flashing at you, and you can't turn it off. Or AI Overviews, which you cannot turn off, I don't think. And people might like them — maybe they don't. It's kind of killing the golden goose, in my mind, because these websites it's drawing from are not getting revenue from clicks and ads.

And so, to the extent AI is a thing I interact with in the world, it's in the form of slop and scams and whatever. And so there's this huge distinction between the personal use and the societal effects of it.

Patrick: I think some amount of this is cognitive dissonance, and we've heard this story before on, for example, mobile — where people will say, "I hate mobile phones existing. They're corrupting the youth. They're distracting us all. They're yada, yada, yada." Okay. Do you own an iPhone or Android? "Oh, iPhone. Yeah. Of course."

Garrison: Well, it's like a necessary part of being a modern citizen or whatever.

Patrick: And, you know, you integrate that over the entire population — I think people should come to the conclusion that mobile phones are actually useful for humanity.

AI as a tool for redress

Patrick: While not being of the left, I would implore people on the left to understand that there are things that you have wanted for a while which AI does make much easier, in ways that are positive for your interests, the interests of the direct beneficiaries, and the interests of society at large.

There are places where, for example, scarcity of labor rate-limits redress for various problems caused by capitalism, or simply by the state of there being finite resources in the world — and AI is very good at achieving redress for things. A long time ago, in a place far, far away, I was an unpaid customer advocate helping with banking problems and credit problems — a specific one of these things that we love doing as good liberals. 

Well, okay: if you are capable of writing a professional letter to your bank and citing Regulation E, your bank will take care of a lot of problems for you. And, go figure, there are many people in the world who have trouble with their bank but are not capable of writing a letter that confidently cites Regulation E, and confidently counts up to the thirty-day shot clock that we've established in the legislation, and then writes the second letter to the bank saying, "Okay, you've exceeded the thirty-day shot clock, and therefore, under the law, I'm entitled to the following."

[Patrick notes: I’m being handwavy on the mechanics here, fellow Reg E buffs. Apologies.]

And so this was part of my job as a consumer advocate. I wrote hundreds of these letters over the course of a couple of years. Using me describing this online, people have written many more than hundreds of letters, just saying, "I have a problem with my bank. Solve it like patio11 would solve it" — my Internet name. And I've received dozens of, "Hey, thanks for..." — most recent one: "Thanks for helping me appeal an MRI denial from my insurance company," where I did nothing there other than apparently being useful on the Internet, and this person knew: "patio11 versus insurance company, go."

And as someone who cares very keenly that people get the MRIs that they have paid for from their insurance companies, that people get the proper resolution from their banks in the case of a contest between their banks and them — I think this is positive.

There is a knock-on effect here. Dave Guarino had an excellent point that there are adversarial touchpoints in government where we have procedural redress available, but there's a finite number of people working the desk that do procedural redress. And so we're scaled to take 1,000 complaints every month, but we might not be scaled to take 100,000 complaints. And when the human effort involved in causing the complaint goes down — from "you need to be able to write two pages of professional managerial class prose which thoroughly cites the relevant statutes and precedent, et cetera, et cetera" to "you need to be able to send a text message complaining" — then it is possible that we will swamp the adversarial touch points, and thus the quality of service at the adversarial touch points will go down. And so that's one subgenre of this that I think people should be aware of.

But broadly, where there are things in the world that are caused by scarcity, alleviating that scarcity is, I think, a priori in the interests of people with a leftist political persuasion. And good news — qualified good news — AI is going to alleviate scarcity in some fashions.

Garrison: Yeah. I mean, I think it's about how you use it, right? There's a lot of people on the left, and broadly, who just wanna say no to AI, and I get the impulse. I get not wanting to give the companies your money, your data, your tacit approval through usage. But there are examples like Matt Bruenig, this leftist policy analyst, who is using Claude Code to make NLRB databases that people can search publicly — taking this illegible kind of work from the government and helping make it available to people so they can use it to better organize their workplace. And there are examples of people making civic tech to help people fight their landlords when they're behaving badly, or navigate these government bureaucracies.

And it's not to say that it's just a matter of that. We need to also have regulation and industrial policy.

State capacity and regulating what you understand

Garrison: I think technology, in the broad scope of history, has often led to good things. But since the '70s, we've just not been regulating software, and now AI, and these are industries where there is just a lot of externalized harm. There's this book, Regulating Digital Industries by Mark MacCarthy, where he talks about how, for almost 150 years, when there was a new industry that sprang up, the government was like, "We should use an expert regulatory agency to manage this." So railroads, radio, TV, drugs — all these examples. And software happened to come around in the '70s, when there's this deregulatory push starting under Carter and continuing under Reagan through the neoliberal period, and AI and social media came up also in that.

And so we look at enshittification, we look at these things that we use all the time that are making people tons of money, that are just an inevitable part of modern life, and then we're like, "Why does the experience of it feel so bad?" Or, "Why does it have all these harms?" And I think it's just because we've not used the tools that we've historically used to address those harms. And so there is a good version of this. It just requires state capacity and thoughtful regulation.

Patrick: I will say, on the part of the tech industry, that it has frequently felt to me that partially the lack of the regulation has been perceived as a feature by people who desire the power the tech industry has — to say, "If it were regulation, we'd have to make our preferences very explicit and write them into law and similar, and if we can simply jawbone our way into getting them, then we don't have to own up to having done things that we did." And I'm subtweeting some behavior from at least one administration here — at least two at this point, on both sides of the aisle. But broadly, as someone who believes in the American political experiment, I would prefer our preferences to be legible and enacted in laws and similar, versus carried out under cover of darkness through extra-legal means like pressure on the tech companies.

And without necessarily endorsing the 1970s regulatory status quo versus the deregulation push, et cetera, et cetera — I think one thing it would behoove people to do is just understand this better. You mentioned expert regulatory agencies. We largely did not put together railroad regulation on the basis of people who could not articulate how a locomotive works, and could not understand the economics of freight transportation and how the network effects mattered.

But we've had such a decline in state capacity over the course of the last few decades, perhaps, that it is not obvious that Washington — even the parts of Washington that are pro-regulation — actually understand, or are willing to come to understand, the thing that they want to regulate. Which I think is a just fundamental stumbling block for the project.

And to the extent that people believe "AI is likely to create a lot of harms" and, simultaneously, "I don't believe AI is a thing, or worth the intellectual effort to understand — certainly I won't go to their conferences, because that would be cavorting with the enemy": this is not a coherent preference set. If you want to be able to regulate the thing, and to advocate for principled regulation of the thing, you really need to be able to understand the thing, and to talk with the people that make it, et cetera, et cetera. And so we'll fly the radical centrist flag for: "Hey, there's this place called Lighthaven. A bunch of the geeks hang out here. Come on down. The water's fine."

Garrison: I mean, yeah, I guess I would push back a bit, in the sense that politicians have never been experts in all the things they legislate on.

Patrick: Mm-hmm.

Garrison: But there's this process of calling in experts and hearing their testimony and commissioning reports and whatever. And I don't think that the politicians who regulated nuclear power understood fission — the physical process — that well. And we could say that they over-regulated it, or they made mistakes in some way, but it's very safe. It's this very complex thing that nevertheless the government figured out rules that managed to make it safe, without having that expertise in their heads.

Patrick: I think nuclear power itself — one can make some arguments about the NRC, but I think the NRC doesn't fundamentally think that it is opposed to nuclear power being a part of the American experiment. Although, again, one can make some arguments about the NRC. [Patrick notes: I’ll let a guest articulate the industry’s position in a future episode, but it has been repeatedly waggishly observed that one would not have to change too much about the day-to-day activities of the NRC for about 40 years if the actual intent was a nuclear freeze.] 

But I think that the people who want to regulate, for example, social media misinformation are opposed to social media existing in the world, full stop, and they will tell you that explicitly. And a lot of the intellectual effort that goes into regulating tech broadly, or AI specifically, is by people who think: tech and AI are my enemies, and I am searching for hammers to hit them with. And child exploitation, for example, is a hammer that I can hit them with — versus: there is some number X of the amount of child exploitation in the world, and if we had better regulations, I would be able to cut X into 0.2X, and then we would have less child exploitation, which is obviously a good thing, all right-thinking people understand this, and also get the benefits of the social media existing and the tech companies existing and AI existing, et cetera, et cetera.

And so — personally not opposed to prudent regulation in principle, but bluntly, the experience of the last couple of years has not increased my credence that people are abundantly capable of doing prudent regulation.

Tobacco, nuclear, and AlphaFold: strangle it or steer it?

Garrison: It is a hard time for that. I guess I'm kind of like — is social media worth it? You know? I think that there's gonna be something like it, but not every industry that exists right now should necessarily continue to exist, right? Like, tobacco and fossil fuels have all these harms that are inherent in the product itself.

Patrick: Mm-hmm.

Garrison: And I don't think it's crazy to be like: as a society, we should either regulate or more or less prohibit this industry from existing.

Patrick: I honestly would welcome a debate on a societal level of whether tobacco should exist, for example. It seems to be one where we had the historical precedent that we had. The historical precedent is horrific, and possibly it's an important thing to spend political capital on to just kill it. There is no reform of tobacco that makes tobacco not tobacco, and so that is what it is, and we should be honest about what it is.

[Patrick notes: It is considered to be a respectable position in some circles that society should probably cease combusting fossil fuels in the same fashion that a smoker should cease combusting tobacco. Depending on the exact advocate and venue, you might get told “Oh but do it gradually, energy is necessary for human flourishing” or “No, actually, the ‘degrowth’ is the point.”

I am broadly skeptical of claims that humanity should decrease aggregate energy use and think that the arguments against are so dispositive as to be borderline impolite to state outright. For example, if developing nations are constrained to use less energy per capita than the 1900-era US, they will continue to be abjectly poor for most citizens, and that poverty will predictably kill people.] 

I feel less sanguine about — well, not putting these words into your mouth, but because I've heard these words from other people about AI: "My first preferred policy is killing AI. I don't think I can get that, so my second preferred policy is strangling it, and I have levers available to strangle it." It's like — well, it seems to be very, very useful, objectively, and I would hate to strangle it in that fashion. And then there's yet another group of people who might say, "Well, agreed, it's useful, but one of the harms I'm worried about is literally the extinction of the human race. So given that that is on the table, I will pick strangle versus exploit."

Garrison: Yeah. I mean, I think the type of AI matters a lot, right? And so, in the book, what I call the obsoleting project — the AGI industry — I think we should stop that, because it's dangerous and democratically illegitimate. And in the future, if the industry wants to make its case and affirmatively get buy-in from the public, and then demonstrate that it's safe enough to build the universal labor-replacing machine, then they should have to do that and make their case — rather than ask for forgiveness, not permission.

And then things like AlphaFold — these incredibly powerful AI tools that solve these long-standing social and medical and scientific problems for us — we should have more of that. And I think that the people who hate AI, if they know what AlphaFold is, or it was explained to them, they'd be like, "No, that's good." And I even hear this from people like a host of Chapo Trap House, being like, AI-for-science — those are good examples of that. And so I think it'd be hard to find somebody who's so, so truly against everything, where in a conversation you couldn't find examples of the tech that they were in favor of.

But they don't see these options available, because we're in this world where it's kind of "go fast" or "stop," and those are the two levers that they have to pull. And so they just choose to stop.

Patrick: And I think, again, reinvigorating state capacity, and a respect for expertise, and actually having expertise are all good things regardless of where one is on the political spectrum.

And for the specific example of AlphaFold: it might not be obvious to people that don't have a chemistry or biology background that AlphaFold and proliferation of biological weapons risk are the same picture. Because if you have the machine that can tell you what proteins do when you fold them together, and then you combine that with things like the ability to order proteins from labs on the internet, that leads to really bad outcomes in a lot of scenarios where the lab is not actually thinking through what the protein does at the point where they mass-produce it for you.

[Patrick notes: For a more informed view, see e.g. Dual use of artificial-intelligence-powered drug discovery.]

And so we should be aware that — one, obviously, chemistry in the pursuit of pharmaceutical advances has produced outstanding value for humanity, will in the future, we should want to support that, comma, however, we should probably not make Hezbollah, or any actor capable of getting thousands of dollars of technology, capable of creating, you know, COVID in their bedroom. [Patrick notes: Not for the first time I note that Japanese-speaking readers of security debates have a very different prior on how likely it is that a non-state actor would be able to successfully build a chemical weapons laboratory as e.g. a side hustle to a religious organization.]

And so these goals are not directly in tension with each other, but they're part of the same broad conversation on how one regulates and sensibly manages an industry that will have capabilities that humanity has not had a few short years before.

Garrison: Yeah.

Patrick: And scoping out from just how to do proteins: AI is going to create any number of domains in which humans have capabilities that they have never had before. Some of that will be for wondrous uses, and some of that has potential downsides. And I would love a sort of line-by-line discussion on uses, to be conducted by people who have an appreciation that the future is coming, and an appreciation that there are some outcomes in it which reasonable people of many persuasions would be very unhappy about.

And I think "just turn it off" — I don't know if that's necessarily possible at this point. But there would be downsides to "just turn it off," too. Again, we wouldn't abandon medicine just because of thalidomide, and I think we rounded to abandoning nuclear energy just because of a small number of accidents, and that's been greatly negative for humanity, unfortunately. But we probably shouldn't close our eyes on this just because there are many potential — and very real potential — downsides to it.

Garrison: Yeah.

Where to find Garrison and the book

Patrick: And looking at the normally powered human intelligence version of this conversation that's gone on for a while — and while AIs can read the transcript forever, humans can't — can you tell people where we can follow you on the Internet, and what more should we know about this book coming out?

Garrison: Yeah. I'm on Twitter @GarrisonLovely, and I also publish a Substack called Obsolete — obsolete.pub. The book is called Obsolete as well. You can find it at obsoletebook.org, or just Google it. It's coming out September 15th. If you pre-order through my publisher, you'll get it in August.

And if you are somebody who's in AI safety world, or you're an insider of some kind, and you're thinking, "This book is not for me — this is for people who are not already in this world," I would actually say: no, it is for you as well. I'm aiming to meet people where they are and guide somebody who's never read about this before. But then also, it's introducing a new paradigm of understanding AI. There's a critique of the alignment problem — the alignment polycrisis is my alternative — and AI Reform is my bid to kind of unify the AI debates. And I think that there's something in this for anybody, and I think it is a different perspective than what you've heard before.

Patrick: Yep. I would also say, generally, that people should just be raising the sanity bar in terms of communication skill with regards to these issues. And having gifted communicators who can explain this in a way that is legible to policymakers, to members of civil society, and to the broader public is useful regardless of what one's personal point of view is. And so — not to sell more books for you, but to sell more books for you — just reading how people explain this in ways that normies don't bounce off of is an incredibly useful thing.

But with that, thanks very much for taking the time to chat about this today.

Garrison: Yeah. Thanks for having me.

Patrick: And for the audience, thanks very much for listening to this week of Complex Systems, and we'll be back next week.

Garrison: Thanks.