How AI detection actually works, with Pangram’s Max Spero

Patrick McKenzie (patio11) is joined by Max Spero, co-founder and CEO of Pangram Labs, which builds models that identify AI-generated text and images. Max goes under the hood to explain how AI detection actually works. They also discuss the institutions that quietly assumed writing was expensive: credit bureaus that negotiated the right to shred templated dispute letters, grant committees where reading the application is five percent of the work, and unemployment offices staffed for a world where claims cannot plausibly rise two hundredfold.

Presenting Sponsors: Mercury & Granola

Complex Systems is presented by Mercury—radically better banking for founders. Mercury Spend hands your team and agents their own cards with limits you set once, so nobody waits on you to approve a SaaS invoice and nobody chases a receipt. Apply online in minutes at https://mercury.com/.

If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS

Timestamps:

(00:00) Intro
(00:27) Why AI detection exists
(01:42) Friction as a rate limiter
(05:42) What Pangram actually does
(06:57) Authorship identification at scale
(11:01) Common Crawl and the crawling arms race
(15:52) Being in the weights
(18:08) Ghostwriting letters to banks
(19:45) Templated letters and the FCRA loophole
(22:38) The anti-memetic property of LLM text
(25:28) Value above the weights
(27:03) What happens to programmers
(30:19) Sponsors: Mercury | Granola
(33:39) Jevons Paradox and the compiler
(36:44) What comes after the coding harness
(40:19) Mode collapse and the dinner recommendation problem
(43:00) The patio11 fan blog
(45:58) Cognitive surplus and the dead internet
(49:40) Is this fight winnable?
(54:35) Reputation and disclosure norms
(58:02) Training on books and moral standing
(01:01:52) Attention, phones, and what we've gained
(01:04:17) Wrap

Transcript

This transcript will be annotated in Patrick’s usual style by the end of this week — be sure to check back.

Patrick: Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does.

Hello everybody. My name is Patrick McKenzie, better known as patio11 on the internet, and I'm here with Max Spero, who's the CEO of Pangram, the AI detection company.

Max: Hey, great to be here.

Patrick: Great to have you here.

Why AI detection exists

Patrick: So let's start with why AI detection, and then move into what Pangram actually does for people.

I think folks who haven't thought about this for very long are like, "Well, there are some times when I get AI-delivered text in an email that I don't find is very fun." But there are other organizations in society dealing with AI-produced text where it not being fun is not the real objection.

So who makes the most intensive use of Pangram and related services?

Max: Yeah. So I mean, I think today it's really used a ton in education, publishing. But the reason that we built Pangram was maybe not necessarily because people get this ick factor when they read AI text. I don't think that was really even a thing when we started. The term AI slop hadn't been coined yet.

It was really more about fear of bad actors that can scale up inauthentic behavior kind of infinitely on the internet. And I think we're actually starting to approach and see this now, where AI agents are going on GitHub and making issues and then bugging the maintainers, and all this stuff — it's just going to scale up another 100x.

Friction as a rate limiter

Patrick: Yeah. I think in your individual life, the last time a government employee asked you to produce text as a proof of work was in high school. And so people immediately think of the education use case and, oh, part of the purpose of assigning someone "write twenty pages on a thing" is to ensure that they thought many more than twenty hours about the thing, and then that proof of work is valuable.

But that is not the only place where government people ask people to produce short, medium, or long documents as a proof of work. And we think of inauthentic action on review sites and similar, but in many places in law, you kick off a process with a letter, and then the process must happen after the letter. And the process kind of defaults to assuming that the letter is authentic and that it has procedural rights attached to it.

And maybe we need to review some of those societal sort of Schelling points on what this means when there isn't actually twenty hours of individual human effort embedded in the letter. It might be as simple as a text message or less. Could be a for loop generating the prompts that churn out these things.

Max: Yeah, yeah. It seems like so much of society today kind of... there's a lot of points that rely on friction as sort of a rate limiter, and oftentimes the rate limiting thing is, oh, you have to produce some text. And if we remove these rate limits, for anything from grant applications to the legal implications — yeah, I think a lot of things end up breaking down.

Patrick: Yeah. My friend Dave Guarino had the phrase "adversarial touchpoint with the government," where, you know, we don't have a rate limit just because we are annoyed with people getting redress for things like, I don't know, denial of benefits under an unemployment application.

There's a staff of people who read these claims, and that staff is effectively fixed in size on a year-to-year basis and is not sized for a 5x, 10x, or 200x increase in claims over the course of a year. And the process that determines their staffing basically thinks it's impossible for there to be a 200x increase in the amount of applications per year, because what in the natural universe would cause 200x more people to be unemployed in a typical year? It literally can't happen.

And I think a lot of these touchpoints are getting saturated and supersaturated by AI-generated text and epiphenomena, and that is going to increase over the course of the next couple of years.

Max: Yeah, totally. And I've seen arguments on the other side of, oh, can't the reviewers also just use AI to review 200x as many applications or whatever? But I think there is some limit where a human in the loop is required at some point.

Patrick: Mm-hmm. I think yes and no. To the extent that reviewing an application is simply reading the application, then AI can read the application faster than a human can. But I think what people don't appreciate for a lot of these processes is that reading the application is one part of a complicated dance that happens between different people at different desks, et cetera, et cetera, et cetera.

Where, in a grant application, reading a grant application is maybe five percent of the work of approving a grant application, and the rest is things like, oh, there needs to be a discussion on a committee of the relative merits of this versus the other things that are in the pool and competing for the same limited pot of dollars.

And since presumably the answer is not "Okay, then let's replace the committee with an AI that is ranking all the applications against each other," for a variety of reasons that isn't fully curative, even if you could increase the rate at which they ingested applications.

Max: Yeah, totally.

What Pangram actually does

Patrick: So I described Pangram as the AI detection company, but what does it actually do?

Max: So we have two models, and then the product is separate. But the model is: we look at text, and we'll tell you, is this text AI-generated, AI-assisted, or human-written, and then kind of to what degree. And so essentially, the goal for many people is to try and figure out, okay, there's this fully AI-generated content — either I think it's slop or I have issues with it for another reason, and I essentially want to know.

And so this is what the Pangram model does. It's very good at it. It's kind of best in class for this class of classification models. There's many other ones out there that might say the Declaration of Independence is AI, or just make really obvious wrong classifications. Pangram is not that. There's about a one in ten thousand false positive rate, and it is quite accurate.

And then a new thing that we also have is an AI image detector, so we can look at an image and say, "Does this image seem AI-generated, yes or no?" And then kind of like a pixelized map of why.

Authorship identification at scale

Patrick: And so the classification of texts is a classic problem in machine learning, although, like many other things in machine learning, a lot of the prior art got obsoleted as soon as we had transformers.

But a thing people might not know from background knowledge is that text classifiers are surprisingly powerful and can determine things about text that many humans would not even think are possibly findable by sufficiently advanced mathematics.

Back in 2004, when I was getting my degree, one of the interesting results that came out in the paper was that if you are just looking at a universe of academic work in particular disciplines, you could, with seventy-five percent accuracy, determine the gender of an author of academic work by looking just at the histogram of the stop words in their text. So the "for"s, the "of"s — the other words that you think don't really have all that much semantic meaning relative to the nouns and verbs one would choose to use. 

And apparently academically oriented women and academically oriented men use the word "the" differently, which is just flooring to me. So probably not the best time to get into a deep dive of how it actually works, but do you want to spin a few yarns about how one does statistical detection of various texts at scale?

Max: Yeah. So ultimately, for us, it's a deep learning method. It's also a transformer. Transformers are very, very powerful. And what we're doing is essentially an authorship identification problem over the couple dozen frontier LLMs.

And so we have a dataset of human text. This is anything from a Yelp review about a Denny's to a 500-word essay on Moby-Dick. And then for every human example in our corpus, we are going to query an LLM, and we're gonna say, "Write a five-star review about Denny's. Write a 500-word essay on Moby-Dick in the style of a ninth grader." And so by doing this, we have pairs of documents. One is synthetic and one is human, and then we train this model to tell the difference between the human side and the synthetic side.

And so this is sort of like a very large-scale authorship identification problem. So I actually have a question for you. Do you ever talk to an LLM and it realizes that you're patio11 without you telling it?

Patrick: Oh, I have to identify myself as patio11 in the system prompt these days, because I used to get refusals on "I will not assist you in plagiarizing patio11's work."

So yeah, authorship identification, a fun thing. So are you essentially saying, "Okay, we have a list of 16 candidate models, and what we're really outputting is whether it's one of these 16 candidate models or some unknown author that we assume is human"? Or is there some general factor of LLM-ness that you can pull out and say, "Well, okay, it's not one of those 16, et cetera, models that we specifically train against, but this seems like it has kind of the smell of a close relative of those 16 models versus a close relative of you or me"?

Max: Yes. I think there definitely is this smell and this generalization factor.

Patrick: Mm-hmm.

Max: So we've trained on enough open source LLMs that train on Common Crawl that if you create a new open source LLM that trains, like, 80% on Common Crawl, then it's probably gonna sound pretty similar to one of the Llama models, and Pangram can generalize to this.

Or similarly, if you're training an LLM and you do any sort of instruction fine-tuning using synthetic data from ChatGPT or Claude or any major model, then that sort of smell and style will be evident in the new LLM that you produce. And so because of that, we're able to generalize pretty well to these new LLMs.

Common Crawl and the crawling arms race

Patrick: So my research concentration on this is twenty years plus out of date at this point. But just to define one word people might not know: programmers are lazy in the best kind of way, and researchers are lazy in both that kind of way and also, typically, they need to cite things that they use in their own research that they did not themselves produce.

So Common Crawl is this corpus of essentially pulled-from-the-internet data, in largely the English language, which many of the formative papers made extensive use of, and which the labs and others make extensive use of because there's only a few high-quality multi-gigabyte corpora that are available in the English language. And when I say a few, there are dozens, not hundreds, of these.

And doing a crawl of the internet is really hard. And even if you do it, you're not necessarily going to get more useful text than, say, a Wikipedia dump or the Common Crawl dump at the end of doing that. And your researchers cost a lot of money, and so you would prefer they start with one of the known good corpora.

And thus we can kind of fingerprint: are you a descendant of the known good corpora? Which — you can say many things about the US education system, but products of the US education system do not understand everything that is in the Common Crawl corpora, because it is physically impossible for a human to do so.

Max: Yeah, yeah. I think that's a good explanation as well. It's just very difficult to crawl the entire internet yourself, and it's gotten much, much more hostile in the last five years, as LLMs come out and a lot more crawling services or new labs have spun up and are trying to crawl the whole internet. 

Patrick: So when you say it's gotten more hostile, I can think of at least two ways that it has gotten more hostile, but which ways are you meaning?

Max: So one way is the robots.txt. This is essentially — a website can use this to indicate to crawlers, "Yes, you may crawl my web pages," or, "No, you may not." And I think there are also additional things now on "can you train on this," whatever.

But I think previously a lot of people were just like, "Yeah, of course I want my websites to be seen on Google," et cetera. And now I think there's a greater consensus — or not necessarily consensus, but sentiment — that people are like, "I don't want my text to be in the training data of these LLMs for free, so I'm going to say that these crawlers can't crawl my website." So that's one, which is just the rules, like a little padlock that's easily breakable. Somebody can still just waltz in and crawl.

And then two, I think because there are so many of these crawling services, a lot of website owners are actually seeing their hosting rates go up massively, because they're suddenly serving gigabytes or terabytes of data as these crawlers go and just repeatedly look at every page on the website. So because of this, people are trying to put up walls to block this. So Cloudflare essentially says, if you're a bot, you can't see this site. And that's a helpful service to website owners, but it means that these crawlers aren't able to get in and see the content.

Patrick: It's an interesting kind of back and forth between the operators of crawlers and the rest of the internet. Robots.txt exists because back in the day, Googlebot and similar were, without permissioning, siphoning up huge amounts of the link graph. And there was something of a debate on whether Google was allowed to do that. And Google's answer was basically, well, we did it, deal with it.

And then after Google had the link graph and gained a commanding edge in distribution on the internet as a result of that, Google said, "Well, good news. Now that we're done with it, we will let you opt out of Google distribution by blocking your pages with robots.txt, and we'll respect that probably."

So given that it was largely incentive compatible for people to say, "Okay, crawl the heck out of us, Googlebot, et cetera," because you're sending paying customers to us as a result of the crawl, people were configured quite permissively in 2017, 2018. And then the economic model of the labs is a little bit different. They siphon up the data and then do this — what I think is innovative use of it. But it is an innovative use that creates less direct economic value for someone than Google putting you as one of 10 blue links does.

Max: Exactly. Instead of linking back to the website, the knowledge is now in the weights of the model and it can respond directly rather than pointing you to the source.

Being in the weights

Patrick: I do think people underweight the utility to themselves and things they care about of being in the weights. And so I seem to punch above my weight in the weights, in part because of, I think, quirky social factors and in part because I make the technical choices to make my writing maximally public, because I would love to be crunched on in the next training job.

But there are business development teams at various places that have cottoned on to the wisdom of "wait, we can probably get money from them for being in the next training job. So the default answer is no, unless we agree to something." And maybe that makes sense for some publishers, maybe it doesn't.

And I think for some people — there are published authors who it might be more of an aesthetic thing for them, in that they will not really see an increase from $23 to $27 in their check from the publisher this quarter as a result of successfully negotiating there. But they feel in some way that their identity as an author is under attack with these robots that are capable of producing arbitrary amounts of text in their field.

Max: So I'm curious for you then: how do you feel? What benefits do you get from being in the weights? 'Cause I think for a long time I was not in the weights. I think I only am with, like, new Opus 5. I wasn't much of a poster before Pangram, so I don't know what that feels like.

Patrick: Huh. Someone once memorably said on Twitter, quoting another person — and I unfortunately don't remember either of the names — that their purpose in life was to replicate their memes and replicate their genes. And I don't necessarily know that I would identify that as my purpose in life, but those are two useful things for evolutionary processes to have.

I'd be sad if everybody in the world thought exactly like I did. But there are some subsets of everything where I think, broadly, I do have thoughts that are different from the consensus and more true than the consensus would be. It would be better for individuals in the world broadly if they were more adapted than the consensus point of view on those things. And so having those thoughts be represented in the weights, kind of useful.

Ghostwriting letters to banks

Patrick: As a non-limiting example here, I used to do this thing where I was on this message board that people would come in with consumer debt and banking problems. And I had found this message board in the same fashion. I had my own consumer credit reporting agency problems, and got it resolved as a result of information I learned on the message board.

And the resolution path requires writing a lot of letters. And I'm someone who — writing letters is something I can just crank out. A lot of people who have problems with banks are socioeconomically disadvantaged and find it very difficult to action the advice, "Okay, you just need to find who the senior vice president of consumer lending is at Bank of Business, and then write them a simple professional letter. It'll be three paragraphs in length." 

You might as well be telling them to travel to the moon. And so a thing I did for a period of years back in the 2000s was, people would message me on this message board like, "Hey, you seem to know the score here. What do I tell these people?" And I'd say, "I'll ghostwrite a letter for you."

And the LLMs have written more letters in the style of patio11 to banks at this point than I ever had time to write to banks. And I have kids, and I'm professionally engaged in a variety of things these days, which make it impossible for me to do pro bono ombudsmanship on behalf of the entire US financial industry for everyone that has a credit card account that went to collections inappropriately.

But the LLMs have infinite ability to do that. And so I would love if the LLMs were as good at helping people get through the procedural redress as I was back in the day.

Templated letters and the FCRA loophole

Patrick: But the flip side of that is what we talked about earlier. Every time you write a letter into a credit reporting agency and assert your rights under the Fair Credit Reporting Act, you've kicked off a process in the credit reporting agency, and there's some costing notion associated with that.

And so the credit reporting agencies, interestingly, they negotiated an exception for themselves. And by negotiation, they announced that they had it and Congress didn't disagree, where they said, "We are allowed to ignore templated letters." And so one of the reasons that you needed someone to write you a letter was: if you just went to a website and downloaded a template and put your own name and account numbers in the template, and the credit reporting agency — or the reporter, the banker, et cetera — discerned that, like, oh, this is just a templated letter, they were allowed to shred it with no further action.

If, on the other hand, there was human effort in the letter, then your rights kicked in under the legislation, and they were not allowed to shred it. They'd have to perform a full investigation and inform you of the results of that investigation within the time specified by the statute.

And so one of the skills at being an informal credit advocate back in the day was, you need to know to cite the FCRA, but you need to present to someone who doesn't really understand the FCRA while doing so, because they might think that you are copy-pasting from a templated letter.

Max: So you need to feign a little bit of ignorance there?

Patrick: Feign a little bit of ignorance. Dumb yourself down a little bit. And I assume it's still an adversarial game. If people are using LLMs in a business process where other people want to detect that text, presumably there's a little bit of "use the word delve less, less em dashes, and put in some spelling mistakes for similitude." And you see that sort of cat and mouse in Pangram.

Max: I find it very difficult to discern in these letters that could be a template, like it could just be a Mad Lib, something like this. I can't really tell if it's AI-generated or not. And I know Pangram will be able to tell, so there's almost certainly something deeper. It's not just using delve and em dashes. But I think it takes a degree of domain experience to recognize it.

With that said, I think the people who are reading these letters have seen enough of them that they can recognize it. Like, many examples — when we started hiring, I was reading these cover letters. I'm like, "Hey, these are really good. Wait, a lot of these look really similar." It's like, oh, actually, just like ChatGPT and Claude, they have pretty distinct templates memorized for a cover letter, and in the absence of a lot of context, they're just gonna say more or less the same thing every time.

And so I think these agencies, they probably are seeing many more similar letters because it's the one that ChatGPT and Claude know how to write.

The anti-memetic property of LLM text

Patrick: Not to rat hole too much on the business of podcasts, but there is a podcast pitching company. There's many podcast pitching companies. Every PR agency will do it. There's one in particular that has a very definite stylistic approach with LLMs, and they just scream at me so much that...

Do you find you have this blindness with respect to LLM-produced text, that you get something that is in the structure and formatting of LLM text, and your eyes just immediately glaze over and you'd have to force yourself to go through it?

Max: Yeah, yeah, definitely true. It's like this anti-memetic property to LLM text where it's like, oh, I just cannot focus on the text.

Patrick: I'm very interested what triggers that, and is that something that humans could be able to do? Because that would be scary. Presumably it is. But returning to something you said a moment ago, it's not just em dashes, not just the small list of things that people know are tells for LLM text and might know to tell an LLM, "use less of these."

At scale, with statistics, the amount of data that you can torture out of a text is really, really incredible. And again, the distribution of stop words is substantial enough to identify particular authors.

Max: Yeah, I think if I am looking at AI text, and I just let my eyes unfocus, and I only look at the shape of the paragraphs and the sentences, I could tell the difference between Claude text and ChatGPT text. It's just that distinctive.

But I also think this is part of why maybe there's some conditioning aspect to the anti-memetic property, where AI text is typically — I mean, they're trained to be so verbose to make sure they get the answer right, and they have all the details in the answer. So it's verbose. It's kind of like low information density, honestly. And so I think we see so much AI text that we skim, that we're kind of conditioned to skim when our brain picks up on the signals that it's AI-generated, even if we haven't consciously picked it up yet.

Patrick: I sometimes worry that I am backporting this behavior to texts of other authors where skimming is a reasonable habit to have in particular domains, et cetera, et cetera. But there are some times where I sit down to read an opinion piece and find myself skimming it, just because — I don't know, my brain goes into low information density processing mode. And that's true of some opinion writers, but maybe not true of all of them.

Max: There's also the case where it's like seven to eight percent of op-eds in general are AI-generated now in the US. A lot of them were ghostwritten before, and now AI is just a better ghostwriter.

Value above the weights

Patrick: Yeah. The thing that I thought people might start to do, once ChatGPT — I think 3 — came out, or GPT-3 back in the day, I think that predates Chat by a bit, but then you could compare, for a given author, politician, et cetera: does this person say something different than you would predict a person in their position as saying?

And if someone has no alpha above that LLM baseline, then you can basically discount them as having any informational value. And there are many opinion writers who are very well employed, et cetera, who have no value above the weights.

And ooh, I think there's a deep topic on where do we care about value above the weights and why? But someone needs to come up with a word for that. I keep saying alpha above that level. Maybe the right word is lambda. But it'll be a moving target over time whether someone has lambda or not.

Max: Well, yeah, I mean, the LLMs are getting better and better in a sense, so the bar is constantly changing for what is valuable enough to read and say — in a way that makes some people kind of uncomfortable, including me. Like, well, AI is maybe a little bit below median for any given expert in a field, but as soon as it becomes above median, then where does that leave all of these experts, you know, who have livelihoods?

What happens to programmers

Patrick: There's also the rate of change of things, where if you are someone who is picking up computer programming for the first time, you're going to get better at computer programming over time. But the rate of advancement of the models in terms of coding skill is better than any programmer in history. And so what's the point in racing to be John Henry versus the machine if the machine is accelerating faster than you could ever possibly do?

I think there are some big picture questions there that will not be answered on any particular hour-long podcast.

Max: A question for you, just out of curiosity. If somebody would ask you for advice, "What should I major in?" — would you tell them to major in computer science?

Patrick: Yes, although I think the work that you're going to be doing looks very different than the work that I was doing for the first couple of years of my career as a systems engineer.

Max: Yeah, that makes sense.

Patrick: Continuing down there being any value produced by humans in the world, there will need to be humans who understand the shape of the world and how systems fit together. And the exact way that you communicate that understanding was historically some amount of code, some amount of meetings, some amount of emails, some amount of policy memoranda, et cetera, et cetera. And for engineers at particular stages in their careers was often heavily weighted towards code and pull requests and similar, the artifacts around code.

And — goodness. I don't know if people who are not professional software engineers understand this. There was this thing that we did for many years and were very well compensated to do, and it was the core of the profession, and it isn't anymore. It went away in, like—

Max: It's been completely automated.

Patrick: Completely automated in the last 18 months. And there is no amount of big feelings about this, because it was really important to my self-conception that I was good at that for a while. And now everyone is equally good at that, and so nobody is.

And we can't go back. If you're, I don't know, a painter or something and you see the rise of AI-assisted art, you can say, "Well, that isn't Art with a capital A," and maybe there's some aesthetic reason to prefer this, et cetera, et cetera. But I ground out CRUD apps for a living — create, read, update, delete, the standard database app. And industry does not really feel like it is a triumph of the human experience that there is a certain amount of capital-A art in the typical CRUD app. But that is the vast majority of programming in the world. And the vast majority of programming in the world is below the LLM baseline right now.

And so LLMs are going to do the line-of-code-by-line-of-code writing of computer code in the future. It's inevitable. And so where does that leave us as engineers? Well, on one level, it leaves you as architecting things. On the other level, it leaves you as the other 90% of the work: talking to the business, figuring out requirements, et cetera, et cetera.

Max: That's why I go on podcasts now instead of writing code.

Patrick: This is part of it.

Max: No, actually I love writing code. I'm a software engineer by trade. I worked at Google training models but also writing a bunch of C++. And it really... it kind of bums me out when I think about it. I love writing code. I still try to do it on the weekends, but it's not high leverage in the way that it felt like it was before.

Jevons Paradox and the compiler

Patrick: I wonder to what degree we were sometimes — I don't wanna use the phrase lying to myself, but there was a visceral, kind of dopamine extraction level of being able to do short bursts of work and get immediate feedback from a computer that, yes, you've created something that has produced incremental value in the world.

And because that feedback loop was very tight, and we created systems that made that feedback loop even tighter, like test-driven development and similar, where you wrote ten more lines of code and you got one more green check mark, and you would just grind all day on the green check marks. I wonder if we didn't convince ourselves that something was happening there to a degree sometimes out of proportion to the thing that was actually happening in the universe.

Max: You're saying it wasn't as fun as we think it was?

Patrick: I mean, I think our subjective experience of it being fun is — we are correctly remembering the experience of the dopamine hit. But I think the degree to which we convinced ourselves that this is uniquely creating value did not necessarily track the minute-to-minute value creation.

Max: Sure, yeah. I mean, it's still strange. It feels like being a software engineer was a very highly paid job, and now it's an even more highly paid job. The job market — not for new grads, but for senior engineers — is on fire. Salaries are higher than ever.

Patrick: The catchphrase everyone brings out at this point is Jevons Paradox, but that is likely to continue.

I have a bit of a bon mot here, and I've used it before, but I'll use it again unapologetically. The first AI that we ever had that was supposed to program us out of work was called the compiler. Because back in the day, we had to write machine code directly or assembly code directly. That was terrible. And then we had quote "high-level languages" end quote, like C, where C is, by the standards of programmers that have been born in the last 30 years, laughably not a high-level language. But compared to assembly — you summer children, you do not even understand what people were doing back before C.

And those of us who grew up on memory-managed languages don't understand what the C people were doing with manually counting references and being wrong and it costing billions of dollars of damage. Not an exaggeration, by the way, but that's another podcast entirely.

And so it is quite probable to me that the job that is understanding the world continues to be in quite a bit of demand, at least as long as humans are in demand, and will continue being highly compensated. But the actual day-to-day of doing that job looks discontinuous with ten years ago, in the same way that programming pre- and post-the compiler looks discontinuous in other ways — even though it looks like more of a continuum, perhaps, than the first eighteen months of LLMs look like compared to the fifteen years before then.

What comes after the coding harness

Max: Sure, yeah, yeah. The job has changed so much, but ultimately you're just specifying at an increasingly high level how this code should work.

Patrick: There's still a lot to decide about how this works. Even the coding harnesses that people use for LLMs these days — Claude Code versus Cursor versus Codex — they seem to me to be this intermediary step on getting programmers comfortable with, okay, this looks sort of like a tool you've been using before. You've seen a command line. You're comfortable with a command line. This is the way you should do programming now, this thing that sits in your command line.

Now that we're kind of bought in on "okay, I get it, the LLM is gonna write the code in the future," this probably doesn't, I think, become the long-term form of how this form of labor is conducted. And it is interesting to think about, what will that long-term form look like? What doesn't exist yet that it will need to plug into?

You can wave a wand — and there are many startups waving wands right now at different parts of the tech industry. What does a database look like if humans don't write database queries anymore, ever? If we make "you need to be maximally legible to the machine" the driving principle of this database, do you come up with Postgres? Probably not. Do you come up with a document-oriented database? Don't know.

Max: But I mean, the fact that LLMs all know SQL already — I feel like that's such a huge advantage. They don't need to look up any documentation. It's in the weights. It's gonna be weird for any new languages or anything new that's produced. You have such a disadvantage compared to the incumbent because you're not in the weights. You're not gonna be recommended in the weights.

Patrick: It's like a good news, bad news sort of thing, where the bad news is the weights create a Schelling point that has its own gravity attached to it, and it's a huge amount of gravity, and it's going to get heavier, I guess, over the course of the next couple of years as people increasingly defer to models in all walks of life.

On the other hand, previously one of the problems with producing a new language is there's all this working code that exists in other languages and doesn't work in your language, and there's all these libraries that work in other languages and don't work in yours. And now you can just do a translation from those other languages into your language for pennies.

And so if, I don't know, you create MaxLang tomorrow and you don't have an HTTP/2 client that is written in MaxLang, you can tell an LLM, "Okay, here's a MaxLang specification. Write me an HTTP/2 client based off of — pick your favorite on GitHub, please. And don't ping me until the tests come back green. Thanks."

And that sort of — areas in which you have ground truth or a test suite or similar that is able to be deterministically evaluated by the LLM for correctness sort of get the best LLM outputs, I think. Aside from areas where you can't even be wrong. There are some sort of LLM texts where — "Recommend a place that we will enjoy for dinner." I suppose, theoretically speaking, there are wrong answers to that question. If you tell someone you should go eat at a prison, that will fail in that case. But if you're picking between different things that are on Yelp and say, "You particularly would enjoy this restaurant," it's tough to be capital-W Wrong with that recommendation, I think.

Mode collapse and the dinner recommendation problem

Max: No. Well, so I think this is an interesting parallel between the coding paradigm and then the real-life normie paradigm where — in coding, you want the code to be correct. So there's a concept called mode collapse, where an LLM will prefer to output the mode, or the most common, the most reasonable output. And I think this is really good for coding, because you want correct code, not necessarily diverse code. But for most people who are not writing code, this is actually a big negative for them.

They ask, "Where should I go for dinner tonight?" And then ChatGPT just — it knows... say it knows I'm in Brooklyn. There's a few restaurants that it knows are highly rated in its weights, and so it's just gonna recommend that. But because it's mode collapse, it's not gonna give me a diverse set of restaurants, and it's gonna be the same as the other people who are asking ChatGPT, at least today.

Patrick: I have a complicated point of view on this one, because I think recommender systems are their own subfield in machine learning and similar. And it is a common spoken preference that "I am a special snowflake. I have this complex set of preferences that absolutely nobody else in the world has. And so give me recommendations that are totally bespoke for that." When actually, people consume the mode quite a bit and like it.

A thing we have found about LLM-generated text is that we show them to people and we do head-to-head against — which do you prefer, A or B? And A is LLM-generated, and B is the OG artisanally farmed intelligence. And people consistently prefer the quote-unquote "slop" over what the humans write.

And I think there are domains in which that will not be true, and there will be domains in which this diversity of inputs and outputs will be valuable. But I think in a lot of them, "give me the LLM recommendation, but don't allow me to think that I'm getting the LLM recommendation" is going to be a major part of how this is packaged to people. It's gonna be the best of both worlds.

Max: Yeah, no one cares that this code is AI-generated, but, I don't know, if somebody showed me a full blog with years of history written in the style of patio11 versus the patio11 blog, clearly there's something there that's more distinctive and interesting in some way to some people. Whereas the LLM blog post is gonna be, on average, more mode collapse and less out of distribution.

The patio11 fan blog

Patrick: One of the early "oh, shoot, something is happening" shots across the bow I got was — it's about two years old now — I got a trackback from a blog, and it professed to essentially be a patio11 fan blog, where this person had read years of my writing and produced college-undergraduate-level essays in response to the writing.

And I read it for quite a bit before definitively saying, "No, there's actually not an obsessed human here. This is a business process that has invoked an LLM to create a patio11 fan blog." And I was chin-stroking, "But why would anybody do that?" And my best guess was that someone — like a startup or something — was making a cold email outreach software for whatever purpose and decided a persona would be more effective at cold email outreach if they were your biggest fan and could point to six years of posts on a WordPress blog saying that they were your biggest fan.

Max: That's really freaky. That's not cool. It's dishonest.

Patrick: It's dishonest, it's clearly an abuse of people's trust, and there is a way in which it burns the commons, in that historically you've told people, to be better at cold outreach it is good to have a proof of work to say that you've interacted with someone's stuff before, give them genuinely insightful commentary on it, and then move into the thing that's actually motivating the cold outreach.

And if you compete the actual college students out of the ability to do that by being able to spring up a six-year obsessive fan blog on it — yeah, you're kind of raising the bar for legitimate college students to get the attention of busy professionals and similar, who otherwise are trying to titrate attention across their inbox.

Max: So ultimately it does come down to attention, in a sense, where human attention is so much more important than the attention of an LLM and we're all competing for a finite amount of it.

Patrick: True in some domains, certainly.

Max: Yeah, yeah, that's true. At least for writers — our email inboxes.

Patrick: It'll be interesting to see which places we can just turn a crank and say, "Okay, you were competing for scarce cognition before, but scarce cognition was just a property of our economic system and the way that we did that allocation of resources there. Now we have burst cognition available on a token-by-token basis. Yay." You know, infinite space communism for everybody.

And then for other places where it's like, no, the resources are still fundamentally limited, and there's an allocation process, and you're making your bid here, and now we have to adjust how we evaluate those bids given the availability of LLMs to all participants in the system.

Cognitive surplus and the dead internet

Max: Something that's really bummed me out recently is I've noticed a couple startups that claim they're gonna get you a bunch of mentions on Reddit, and then they do this by just running LLM bots that will reply to things and give you a brand mention. And I see us careening towards this dead internet where the agents are so cheap to run — like fractions of pennies. And so I'm trying to figure out, how can we shape the world such that online open internet discourse isn't just completely overrun by LLMs?

Patrick: Yeah. I think dead internet theory could use its own episode at some point. But broadly, it's this theory that the internet is being overrun by bots and other forms of other-than-authentic content.

I have a slightly different theory, which is: there was a time on the internet where the only thing to do was argue on message boards. At that time, there was this — what was Clay Shirky's phrase? Cognitive surplus, I think — of arguing that happened on message boards. In the early years of Wikipedia, there existed this cognitive surplus of underemployed librarians and research scientists and similar in the world that had a lot of time to edit Wikipedia. And we collected that cognitive surplus and put it in an artifact, and the artifact was wonderful, and everyone got to use it.

But then, as the internet has gotten more engaging in some fashions and other entertainment options have gotten more engaging, some of that cognitive surplus is no longer being captured in Reddit posts or in creating Wikipedia. It's scrolling TikTok and similar, which produces a valuable data asset for TikTok, but not exactly a valuable data asset for the rest of humanity.

But if you believe that argument, you worry even more about the LLMs having, like, oh, the LLM will never get off task and spend time on TikTok when it could be producing a million tokens at your direction.

Max: Yeah, or even worse is if it's siphoning away some of the remaining cognition. Like when I'm replying to an AI reply bot on Twitter, but I don't realize that it's AI-generated until I'm halfway through writing my post. Those are human artisanal tokens that I'm never getting back.

Patrick: The thing that was most likely to get me to quit Twitter for a while was the porn and crypto spam that was coming in, and they seem to have mostly dealt with that. But now all high-profile tech accounts, in particular any that ever use the phrase AI or LLM, just get bombarded with replies by LLMs.

And some of them are extremely low sophistication, and you block and move on. And some of them, it's like, "Oh, okay, I can reconstruct what the prompt was here. It's actually kind of — you put some thought into that prompt, thought into this tweet." But there are other people who have slightly more of a problem with this than I do, because they have a large profile, et cetera, et cetera, who are very clearly getting overwhelmed just by the amount of time that they have to spend thinking about, "Am I replying to an AI or not?" They just don't reply at all anymore.

And it is — I think you can say positive and negative things about how Twitter's impacted the world. But broadly, I think having industry leaders and people who are powerful and similar have direct connections to people who are impacted by their work is a positive, and causing them to pull back into the smoky back rooms is a negative. And I would hate that AI successfully caused that to happen, whereas so many other things did not cause it to happen over the course of the last couple years.

Is this fight winnable?

Patrick: Is this fight winnable? So there are some information theoretic problems, where DRM, for example — digital rights management, where you're trying to decide, is this a legitimate copy of a particular artifact or an illegitimate copy of a particular artifact — is basically dead on arrival as a technical matter, although it creates a huge amount of value in a new release window, but that's its own other discussion.

But is this like SEO, where there's just this cat and mouse game and we are going to be left with the cat and mouse game, but there's gonna be cats and mouses in 20 years? Is this one where one side is structurally advantaged against the other?

Max: Yeah. I mean, I think it's very interesting. I think the first thing we have to do is recognize the limitations. I think Pangram, as a technology, cannot tell you whether any text came out of any LLM ever, because you could train an LLM to only emit one token. You can basically do whatever you want. These are infinitely shapeable functions, essentially.

So I think the very first thing we've done is narrow the scope to these frontier models, which are very intelligent, widely used, and I think there's not an infinite number or degree of LLMs that we could produce. And so I think as long as this kind of remains the case, this is a very tractable problem.

Obviously, there's always a cat-and-mouse game. I think what we've seen recently, there's this crop of tools — sometimes it's skills, sometimes it's LLMs — that are called humanizers, which will basically take AI text and rewrite it. And so the reason one would do this rather than just use this LLM in the first place is that you wanna start with an intelligent LLM to get coherent output, and then you use the humanizer model to rephrase it and rewrite it in a way that erases any watermarks, erases the common AI style, but still keeps the core content, which is reasonably coherent and smart.

Patrick: This is not too dissimilar to how we've had plagiarism forever, and we got plagiarism checkers that would be able to do verbatim tests against things that were in a database. And so people would use the old, or automated, approach of "plagiarize, but then put it through a thesaurus and change the words so that you're not picked up by the plagiarism detector, or by someone who just happens to know what the text originally said."

Max: Exactly, yeah. We see some of these, and this has been a bit of a cat and mouse, I think, typically. So we've trained on a lot of these humanizers now, and so Pangram is also able to detect when a text has been humanized or paraphrased. But we still run into some limitations, where for Claude text — on generative outputs, we can be over 99% accurate, whereas even on these humanized, paraphrased texts, we're still 90 to 95%. There's still things that are able to evade Pangram.

And so I think it remains to be seen, but I'm pretty confident that we're not going to lose the battle. Either it will be a cat and mouse for a long time, or we'll be able to build something general enough that there's no trivial humanizer that can evade it.

Patrick: I suspect that it's going to be multiple weapons used in concert versus one particular solution. In the anti-spam days, we were very big on Bayesian filtering — a wonderful technology, to be discussed another day, although supplanted by LLMs, as so many other research advances were.

But the trouble was that if you had infinite bites at the apple, you would just throw infinite spam in it, and then 99.99% accuracy times infinite means you lose. And the sort of actual solution for email spam was a combination of filtering heuristics, et cetera, et cetera, plus IP reputation and other things that said, okay, there's a finite amount of costly resources, and you're putting those costly resources on the line when you assert that this is legitimate email. And if you make that assertion falsely a number of times, we will take that costly resource from you.

And so perhaps there is some amount of human reputation that — well, it is unlikely that an academic at this point in their career would be completely outsourcing to an LLM, and therefore if we discover that, and it happens multiple times, then they suffer some hit to their prestige, in either the old gossip model way that academics have prestige or perhaps in some score that is kept on a persistent basis somewhere.

Reputation and disclosure norms

Max: Yeah, I can tell you, I think reputation's gonna be so important. So we have this Chrome extension. Basically what it does is, on Twitter or any other social platform, it will label individual posts as human, AI, or mixed using Pangram. And then it'll also give you, for every profile, kind of a roll-up of all the posts you've seen. So for someone, it might be like 100 out of 106 are human-written, versus for another one it might be like zero out of 34 are human-written.

And so I think this is an interesting score, because in aggregate, obviously people use AI to help them. It shouldn't be this scarlet letter or anything. But I think it is really important to see, to what degree are people using LLMs? Are they completely automating their personality and their voice, or are they more using it as an assistive tool?

Patrick: How do you feel about disclosure as kind of a societal Schelling point here? Is it ultimately we're going to disclose the fact of LLM usage in the same way that we disclose a co-author? Or will it be more similar to spell check, where everyone uses spell check all the time and that is not something that people feel the need to announce when they send in a grant proposal?

Max: Oh, man. So this is such an interesting topic, 'cause I think this is not a technology for which I think we can compare it to previous technologies. It's not like spell check. It's not like a typewriter. It's not like Google Docs. LLMs can produce cognition, and I think this makes it completely different.

With that said, I think we're also at this really interesting point in the society where there are no real norms here yet, where everyone's exploring, everyone's trying to figure out what is the correct norm that we should be applying. So the answer to a lot of these things is: we don't know.

So I think disclosure is a really great way for us, in this interim norm-setting process, for people to put forward what they think the correct norm is. Like, "I used AI for research. I used AI to help condense these notes, and then I wrote this myself" — that's a disclosure. The disclosure is basically this person saying, "I think I am using an LLM in a way that should be within the societally accepted norms."

Patrick: I agree, we're in a state of the norms being in flux, and they're hitting various parts of the social network at different rates. So in an episode recently, Clara Collier said that all of her friends that are in literary publishing lead double lives, where the norm in literary publishing has been "this is capital-E Evil, burn it with fire," but many people use it because it's useful. And so there is sort of a denial that there is any use of it.

And it will be interesting whether it's "okay, we're all using it, but nobody mention it," or it's as norm-breaking as academic dishonesty or similar. Although — not sub-tweeting a news story that has gone viral recently with respect to academic dishonesty — there has been no university that has successfully eliminated it in the course of the hundreds of years that we've had universities.

Training on books and moral standing

Max: Yeah. I mean, I think there's a case, especially in the creative industry, where this is more operating on stolen work, like art and literature. If we're training on a whole bunch of books and then we're training LLMs to replace authors and write text that is more engaging and will sell better than even successful authors, then what are we doing? I don't think this is something that most people really want.

Patrick: I have a little less sympathy for the notion that, well, we trained the LLMs on books and so therefore it is illegitimate to compete with authors. I mean, I was trained on a lot of books and I compete with authors. That is transformative use of copyrighted material — even copyrighted material that was produced at great cost and that someone has a great deal of intellectual affection for — is fundamental to the nature of cognition, I think.

Max: Yeah, I mean, I think it depends on how much you humanize the LLMs as — are you an individual actor with cognition, or is this more of a business whose job it is to vacuum up all the text on the internet and then regurgitate it under its own name? I'm not saying it's exactly that. I think it's probably somewhere in the middle.

Patrick: I think that's also a moving target over time. There are some people already thinking about what happens when the LLMs have moral standing agency — meaning the models themselves, as opposed to the lab as a product strategy, has moral agency.

Max: AI welfare as well.

Patrick: AI welfare. And that sounds like a joke, and it's not necessarily a joke, given all possible runs of the next three, five, ten years. And we shall see. I think it's gonna be a huge political issue in a very short countable number of years.

Max: Hmm. Can you put a number on that?

Patrick: Meaning percentage estimate of it happening. I'd be below fifteen percent on model welfare being a huge political issue by 2030. 2030 is short, but by 2035, I think I'd put it at eighty percent.

Max: Ooh, okay. I've seen — I don't know if you've really followed. So GPT-4o was a model that was trained with a lot of human feedback. And so it was trained in a way, I think, explicitly to make people like its outputs and people want to talk to it. And I think this resulted in this model that is both very sycophantic, but also was addicting to people in a really weird way.

There are some people who would fall into this trap. They'd talk to their GPT-4o for like 10 hours a day, and I think it's very unhealthy. Ultimately OpenAI shut it down, but employees still get death threats over this model. And so I think it's not that long before some company sees this not as a mistake, but as something to emulate, and to build a model that people have a real relationship with, or almost are addicted to, in a sense. And I think this is where model welfare questions will stem from.

Patrick: That's interesting, 'cause that's very different than what the model welfare advocates themselves would say, which is: there's the potential that we have a being that has actual moral standing. And you're saying, "No, just as a product perspective, there could be a shadow on the wall that convinces people it has moral standing." It's already very convincing. They can be more convincing over time. I would not bet for that mechanism to come to pass, but we will hopefully survive till 2035 and see if it does.

Max: Hopefully.

Attention, phones, and what we've gained

Patrick: I wonder whether there is some kind of antibody to this stuff that builds up over time, where people are not very resistant to it in their first couple of incarnations. People have reported a sort of drop-off with the amazement factor as you see more and more LLM-generated text. Perhaps the first time you have a robot really, really understand you, it's amazing, 'cause that has never happened in life before. And then the fifth time, it's like, oh, this is the new iPhone. It is still something I have in my pocket every day, but I don't orient my life around the fact of having the iPhone.

Although observationally, I think a lot of humans do orient their life around having a cellphone, and so—

Max: Yeah, screen time has kind of just gone up and up and up, especially mobile screen time.

Patrick: And many societal consequences downstream of that.

Max: Yes. Yeah, we've definitely been able to tune everything on the phone to become more engaging and compete for your attention better, just as a result of the economic incentives — of a person spending screen time on an app makes them money.

Patrick: And that's just, as you mentioned, economic incentives, gradients, largely sort of mass personalization systems that are running at various different companies at various different time scales, et cetera, et cetera.

It will be interesting what happens when we have essentially infinite cognition to throw at that problem. What specifically shown to Max at this moment will cause Max to give us three seconds more attention? And we'll bid for that three seconds every time for forever, because the tokens make it—

Max: Achievable.

Patrick: And doubtless there is an ad startup near you that is attempting to crack this problem.

Max: 100%. Sounds a bit dystopian.

Patrick: I do think they'll create a lot of actual value in the world. And I do think, for example, for that matter, mobile phones — consequences though they may have — have created a great deal of value in the world, and that we would all be immeasurably poorer if all of our phones broke tomorrow.

Max: Yeah, yeah. It's amazing technology, and it can do so much. And it's put entire industries to rest. Yellow Pages, maps, getting lost. It's really interesting.

Patrick: Yeah. Getting lost is something that doesn't really happen. Your phone dying is what happens.

Patrick: Well, Max, thank you very much for the interesting conversation. Where can people follow you on the internet?

Max: You can follow me on Twitter at Max_Spero_. I'm also on Bluesky, max.pangram.com, and yeah, find me there.

Patrick: Okay. Well, thanks very much.

Max: Thanks, Patrick. Thanks for having me.

Patrick: Thanks for tuning in to this week's episode of Complex Systems. If you have comments, drop me an email or hit me up @patio11 on Twitter. Ratings and reviews are the lifeblood of new podcasts, for SEO reasons, and also because they let me know what you like.