LLMs and the economics of Evil, Inc., with Manish Goregaokar
Patrick McKenzie (patio11) is joined by Manish Goregaokar, a senior software engineer and member of the Rust security response team, to discuss what LLMs do to the cost side of running a sophisticated con. They walk through a scam built for programmers: a plausible company, a few rounds of interviews, and an NDA signed via "sign in with email" that quietly hands an attacker the identity account everything else resets against. A version of it hit at least five people in the Rust community within days of Manish publishing on the subject, and Patrick describes being targeted himself by someone posing as a Bloomberg reporter who wanted him on a Zoom call with a codec problem. They also cover money mules and the three stages of laundering, the $25 million Hong Kong transfer authorized on a fully deepfaked video call, and why the only durable individual defense is refusing to trust any communication you did not initiate.
Presenting Sponsors: Mercury, MongoDB & Granola
Complex Systems is presented by Mercury—radically better banking for founders. Mercury's new feature Command brings an LLM directly into your banking interface, so checking balances, finding invoices, or sending a wire is as easy as asking. Apply online in minutes at https://mercury.com/.
If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS
What's the point of building faster with AI if your database can't keep up? MongoDB's native data model mirrors the language LLMs already speak. Ship at the speed of AI while staying ACID compliant at Fortune 500 scale. Start building at https://mongodb.com/ai.
Timestamps:
(00:00) Intro
(00:27) The future of the con is already here
(01:40) Scams are run like businesses
(03:30) A scam aimed at programmers
(05:12) Why single sign-on is the one ring to rule them all
(06:50) Persistent threats against individuals
(08:58) What LLMs change about cost
(11:40) Manual scams and business email compromise
(14:30) How money movement constrains scammers
(17:39) The industry that helps you pay ransoms
(19:32) Bespoke attacks on individual targets
(21:41) The Rust community gets targeted
(23:02) Supply chain attacks and long-lived impersonation
(24:58) Money mules and the three stages of laundering
(28:35) Why law enforcement struggles with internet crime
(30:01) Sponsors: Mercury │ Granola
(33:03) Patriotic hackers and the post-Soviet talent pool
(35:27) What individuals can do
(36:27) Only trust communication you initiate
(38:21) Wire fraud in real estate closings
(41:07) Why legitimate institutions look like scams
(45:22) Who eats the loss
(47:21) Sponsor: MongoDB
(48:12) Why banks can't just block new payees
(50:45) Verification protocols and family passwords
(53:07) Urgency and secrecy
(54:44) Systemic fixes and who pays
(57:55) Open weights and the coming wave
(01:02:17) Wrap
Transcript
This transcript will be annotated in Patrick’s usual style by the end of this week — be sure to check back.
Patrick: Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does.
Patrick: Hideho everybody. My name is Patrick McKenzie, better known as patio11 on the internet, and I'm here with my buddy Manish, who worked with me at VaccinateCA back in the day and is also a senior software engineer.
The future of the con is already here
We both have something of a hobby interest in fraud, and Manish, you've written a post recently about, I guess, the future and/or present of fraud.
Do you want to briefly talk people through that, and then we'll talk about it from the attacker angle, the defender angle, and perhaps the target angle?
Manish: Yeah. I'm happy to be here. I recently wrote a post called "The Future of the Con Is Already Here, It's Just Not Evenly Distributed." And the general idea is that LLMs are attacking the cost side of doing sophisticated frauds and cons, and we should start expecting more sophisticated cons against ourselves.
Previously, a lot of people have been kind of safe in knowing that, "Hey, I'm smart enough to not get hit by the Nigerian scam. I'm probably fine." The Nigerian prince scam. And now I think that assumption is going to start falling apart as it becomes cheaper and cheaper to do more sophisticated cons, because LLMs are just good at automating all of the scut work.
Scams are run like businesses
Patrick: Mm-hmm. I think a key realization here, which you would hope that everyone professionally relevant already has, and which is not true of everyone professionally relevant, is that scams and other forms of property crime committed at scale are committed by businesses. They might not be formally incorporated, although oddly enough, some of them are.
But they are operated like businesses. They have targets, they have training, they have a supply chain, and are not just informally managed. And those businesses go to where the opportunities are. And so for the last couple of years, they're generally called Lazarus Group, a subunit of the North Korean military, has spent a lot of their time and attention working against crypto exchanges and others in the crypto economy, because that is the easiest way to ring the register and exfiltrate hundreds of millions to billions of dollars in ill-gotten gains.
But they're not morally committed to only hitting the crypto industry. They will go anywhere they can get hard currency to fund the North Korean state. And if it was easy to extract million-dollar checks out of individuals' retirement accounts, they'd be doing that. But since they have finite resources in the status quo, they've only used the fullness of embedding covert operatives, et cetera, into crypto enterprises.
But that could change, from them or from other actors, as we are no longer blocked on the amount of cognition available given this LLM era. So you've sketched a particular sort of scenario for an attack. Do you want to tell people what the scenario for that attack is? And then we'll talk about various ways in which this plays out both in the status quo and in the future.
A scam aimed at programmers
Manish: Yeah. So I basically tried to put together a scenario that would hit most of my readers the closest. This is by no means the only thing that's possible. This is just the interesting one that when I tell my programmer friends, they go, "Oh, damn, that's happening. That could happen to me. I would fall for that."
And the general scam that I put forth was: hey, you're a programmer, you're looking for work, or maybe you're not looking for work, and you get a really interesting opportunity. Looks like it comes from a real company. They've got a website. They've got everything. You get reached out to on LinkedIn.
You schedule some interviews. They have you sign an NDA, and you go through all those interviews, and then in the end you don't get the job. This is something that's happened to many of us, just the regular process of being a person with a job and trying to find jobs. And nothing in this is particularly suspicious. This is just a normal process.
And then months later you realize that somewhere in the process they got your email account or your Apple login or whatever you use for SSO everywhere. And what happened was, when they were making you sign the NDA, you did a sign-in with email and they phished you there.
Your guard was completely down. You're like, "Well, this is a real company." When you get sent to the email account login instead of the "You're already logged in" page, you just think, "Oh, yeah, sometimes that happens," because that does sometimes happen. It happens to me every now and then. I get logged out. You don't think too much about it, and you log in.
Why single sign-on is the one ring to rule them all
Patrick: Just to give some context for those of the audience that are not programmers, SSO stands for single sign-on, and this was a bit of a trade-off. In the mists of prehistory, every website and application had their own list of emails and their own list of passwords, and we realized it was basically impossible to get users to action the advice, "Do not reuse the same password everywhere."
And so, given that people were reusing the same password everywhere incorrigibly, and that this caused a problem where the worst-secured site on the internet — someone's forum for discussing baking Christmas cookies — would get compromised by the attacker, the attacker would get a list of emails and passwords, and then they would use those passwords against high-value accounts like at brokerages and banks and email accounts and similar.
And so we've moved more, both for the user experience reason and the security reason, towards having certain kinds of identity provider accounts. Sometimes there's other accounts in a corporate context, or sign in with Facebook, et cetera, where that identity is kind of the one ring to rule them all.
However, given that it is now the one ring to rule them all, there are things that are directly tied to your identity account — such as, for example, your email — or indirectly tied to your identity account, where the possession of the email will allow someone to reset credentials on, for example, a bank or brokerage, such that a compromise of the identity account is equivalent to compromising many of the high-value accounts that are in a given person's ecosystem.
Persistent threats against individuals
Manish: And then once they had this access, they can kind of even just sit on it. The attack was described as: they're sitting on it, and they use that to slowly gather information on you, slowly expand that access to other places with that login, maybe even use that to get access to some of your financial accounts by resetting passwords and deleting the "Hey, your password was reset" email, and just generally monitoring and being really smart about how to do the attack.
You don't even know you're compromised, and they're able to just sit on this for a while and then strike when they think is the best time to strike for the most amount of money, or whatever they're looking for.
Patrick: Right. In the case where they have an email account, they have your email archives going back for years, so they can build a profile of you.
Where do you do banking? Where do you do brokerages? Et cetera, et cetera. They could potentially even — if you had an unlimited amount of time, if you were an intelligence agency looking into the life of a person — you could predict, when are they going to have a lot of money in their account? When will they sell their house next? When will they have the next IPO? Et cetera, et cetera, and then hit them after that.
This is not what people historically do when they get credentials. Historically, this credential is sold onto the black market very quickly, and the credentials are known to expire rather quickly as people move to new credit cards, et cetera, et cetera.
And so the quote-unquote "best practice" for Bad Guy Inc. has been to extract as much value as quickly as possible from purloined credentials. When that changes, when these sort of — the industry jargon is persistent threats — start to be something that is not just affecting intelligence agencies on intelligence agencies, but actually affecting regular individuals at the top of the socioeconomic ladder if they're programmers, and elsewhere in the socioeconomic ladder because this scales to the moon, that's going to cause something of a rethink in how we deal with financial security and how we deal with securing accounts that can move money.
What LLMs change about cost
Manish: Yep. And it's worth highlighting: nothing that I listed so far needs an LLM. You could do this scam against someone, it's just really expensive. Going through everyone's email is really expensive. Mocking an interview is expensive. Making a website is expensive. Except now it kind of isn't.
You can make websites for free-ish. You can monitor someone's email for very cheap. You can deep fake calls. And I'm a little skeptical of how well you can deep fake calls right now, but there is an attack that happened, I believe, last year in Hong Kong, where $25 million was stolen because some financial company had an employee pulled into calls with who they thought was their CEO and other important people, and the CEO presumably told them, "Hey, press the button that moves the money from point A to point B."
And that was entirely deep faked.
Patrick: Mm-hmm. I think the state of the art is changing rapidly, and I think the most useful thing to understand is that what is available today is the worst it is ever going to be. But state of the art is that you generally need quite a bit of training data on an individual person if you want to fake their voice.
There are many publicly exposed people that have lots of training data out there. You know, I produced a hundred-plus episodes of a podcast. If your voice API provider of choice doesn't do a lot of work to prevent that sort of thing, you've got all the training data on me that you want. But for a randomly chosen CEO in America, if you don't have thirty-plus minutes of them speaking in a particular style, it might be difficult.
However, just targeting generically, like, a middle American accent is absolutely trivial. And given that that is sufficient in many cases, because the person that you need to convince with the deep fake isn't necessarily a bosom buddy of the person you are faking — it's just, "Oh, I'm aware that the CEO of this company speaks with a New England accent, and is a man in his 40s." Man in his 40s with a New England accent, good enough for government work, is absolutely trivial and can be produced for pennies, regardless of what your own face looks like or your own accent sounds like.
And there are good uses of that, too. Accent modification is a thing that people have trained on at quite a bit of cost for many, many years, and there are folks that love the ability of turning on something during the middle of a Zoom call and sounding more natural to their coworkers or counterparties or people they are trying to sell things to.
But the bad uses of this are going to be quite common in the next couple of years.
Manual scams and business email compromise
Manish: And I'd also want to highlight that cost wasn't necessarily prohibitive for these kinds of attacks as well, if the number is good enough. Because previously — actually, Patrick, you yourself have written about the shoebox scam in a different post, and that was presumably an entirely manual scam where they just called the person over the phone a bunch and convinced her that she should give away $50,000.
Patrick: So this is a reference to an article in New York Magazine about — you've already hit the highlights, but a wealthy individual was scammed for $50,000, which she withdrew from a bank, put it in a shoebox, and gave to a courier. And I don't want to speak too much about the internals of that scam because I've written on that topic before, but broadly speaking, you can imagine a business process which involves large amounts of manual labor at various steps in the business process, because that is what most business processes in the real world look like.
And that is very viable when you are targeting crypto exchange amounts of money, or five-figure or six-figure compromises. One of the most common historical compromises here, which hits people for quite a bit of money, is called, quote, "BEC," business email compromise. And it largely involves getting one person within an organization to download malware, give up an email account or similar, and then use things that are in that email account to successfully impersonate the CEO, CFO, treasurer, other similar person with authority to move money, to a lower-level peon who has the physical custody of money.
So one thing that we've historically done in capitalism to avoid scams and many other misuses of money is to separate the authority to move it from the actual control and custody. And so they spoof the authority to the custodian, and in cases where the custodian is in another entity, sometimes that involves the loss of money, but not the loss of the target's money.
In the cases where they're in the same entity — the CEO and the treasurer work for the same company — then it often involves the loss of the target's money. And that has involved individualized work to make an email sound like the CEO and come up with an excuse that sounds plausible, somewhat.
And one of the countermeasures against this has been teaching treasurers and bookkeepers and similar, "Look, the CEO is never gonna send you a text message asking you to buy $50,000 of iPhone gift cards" — Apple gift cards, rather — because someone asking for a payment in $50,000 of Apple gift cards is essentially attempting to get exfiltration of value in a way that is easy to launder that money.
How money movement constrains scammers
Patrick: I think that is useful to segue briefly away from the LLM side and the attacker's view of the world into the defender side, and what the ecology of money movement does to scams and, hopefully, to our ability to defend against them.
So as people know, I've written about this a million times in Bits About Money, but broadly speaking, banks and other regulated financial institutions have a responsibility to block all illegal use of their rails, and there is a large team of intelligence analysts that is hired within the bank's compliance department. Its job is to successfully identify illegal money movement, which can involve the proceeds of a scam, for example.
And they don't necessarily catch 100% of it. Catching 100% is not necessarily the goal. It's just making it increasingly difficult to do, and, contingent on that happening, forcing the bad guy to identify themselves or a co-conspirator to the process, to make prosecution and recovery of funds easier.
So since it is relatively difficult to do this within the regulated financial ecosystem, the bad guys often choose to use financial providers which are straddling the edge of the regulated financial ecosystem. So one of these, unfortunately, is gift cards, and another is the crypto economy. I've written an essay previously, which I will link in the show notes, about how gift cards keep getting misused for this. And this is unfortunately by design.
So gift cards are used for just outright gifts in the middle class, but they're also used as payment instrument substitutes for a variety of people that are, generally speaking, lower on the socioeconomic ladder, might have difficulty proving their identity to a bank to the satisfaction of the bank's AML and KYC policies.
And we want these people to be able to use gift cards, because we want them to be able to spend money to buy groceries or to buy Apple products, for example, which are used by people at all points of the socioeconomic ladder these days. The trouble with that is that if you make a product which is available to move $500 of money around arbitrarily, it is also capable of moving $50,000 by just taking the same thing that moves $500 and repeating it a hundred times.
One could imagine that the gift card companies might be able to detect when someone is attempting to do this sequentially five hundred times in a row. In actual practice, they are very bad at detecting that in the status quo.
The other major thing that is used in the sort of scam flavor text is to tell someone to walk over to an ATM located at some convenience store and buy a bunch of Bitcoin. And indeed, if you have Bitcoin ATMs in your neighborhood, a lot of them will have review-bombing it, saying, "Don't put money in. It's a scam," et cetera, et cetera. Because, bluntly, the only reason to use an ATM to buy Bitcoin at a ridiculous premium to the price of Bitcoin when Coinbase exists is because you have been instructed to do so by a scammer.
The industry that helps you pay ransoms
Patrick: And I don't know that those businesses understand the way their bread is buttered, but there are sort of cottage industries which are organized around facilitating scams. It feels almost unfair to say. I'm going to drop some links in the show notes to one particular cottage industry which I recently became aware existed and kind of blows my mind.
So in the case where you're doing a ransomware payment, the bad guys are located in, without loss of generality, Russia, and they would like to receive a million dollars, or they will wipe all the data associated with your hospital. And they can't take a wire transfer, because geopolitics is one complication, and then the usual AML, KYC is one complication, and also international wire transfers just suck for everybody. And so they will direct the hospital to buy a million dollars of Bitcoin and send it to our Bitcoin address, and then we will release your computers back to you.
And a hospital can't simply phone up Coinbase and say, "Hey, can we open a Coinbase account to buy a million dollars for a ransom payment?" Partially because that will take days that they don't have, and partially because I suspect, without specific knowledge, that Coinbase would not like to be used for large ransom payments.
So there are brokers that specialize, and will say on their homepage, like, "We are the people that you use to pay ransom payments, and we will allow you to pay your ransom payment in a legally compliant fashion." I have no position on whether they are factually legally compliant or not, but they certainly claim on their homepage that they will help you avoid sending money to people on the OFAC sanction list or similar.
And so there is an onshore, nominally regulated, nominally aboveboard industry which exists to, bluntly, facilitate criminal proceeds, which is a wild place to be in the world.
Bespoke attacks on individual targets
Patrick: But again — ransomware is an instructive case, because a huge amount of the victimization is of organizations that have a lot of money, have a lot of societal importance, like hospitals, but don't have a corresponding amount of IT countermeasures.
There are viruses that go around and try to lock random computers and extract a $500 bounty payment from them. But those are, in the past, totally automated viruses, and if only one of a thousand people that is infected actually pays, that works out because the cost of the virus copying itself to another computer rounds to zero.
When actually convincing someone to install software on a system also rounds to zero, you can imagine an automated, semi-automated, or orchestrated attack which involves a substantial amount of bespoke work to identify a target, to identify what series of words will work on them, to even build a relationship with the target before you have the ask.
That is the dangerous one for the target. I'll say this publicly: I was targeted by one of these myself recently. Someone who was not affiliated with Bloomberg claimed to be a Bloomberg reporter, claimed to be interested in a piece I had written, and then attempted to get me to join a Zoom call.
And the typical way that this works is that the Zoom call will, in the flavor text, have some codec verification issue. Can you just install a new thing on your computer? And then you've lost your computer and everything connected to it. And presumably, the business that was targeting me for exploitation there was thinking, "Hmm, probably doesn't have a billion dollars in his back pocket, but probably rich tech professional."
And in early 2026, "rich tech professional" is enough to justify some amount of human and/or automated attention. In six to 12 months, if you're capable of buying a $500 gift card, I think you are likewise going to fall below the marginal line of where they can justify exploitation.
The Rust community gets targeted
Manish: So actually, this reminds me of a thing that I learned about that was happening as I was finalizing my post. Like, literally as I — and one of the pieces of background is, one of the things I do for the Rust programming language is I'm on the security response team. I get reports of security breaches and things like that.
And I believe two days after publishing the post, which is about a couple weeks after I'd written my first draft and started sending it out for review, we get an email from someone saying that they got asked for an interview for a Rust programming job. And they went through the interview, and part of the interview process was, like, a coding exercise, some kind of coding exercise, and they had to download something.
And fortunately, they fed the thing that they downloaded to Claude first, and Claude was like, "That's malware." I don't remember exactly what it was. It was a library or something. So it was just code with an obfuscated thing somewhere that did something and basically would retain access to their computer.
And this attack — I don't know if it was done to people not in the Rust community, but there are, I believe, at least five reports of major Rust community people being targeted by this. I do not know if anyone was actually compromised by this, but this was a thing that happened that week.
Supply chain attacks and long-lived impersonation
Patrick: My first thought is that this is probably a setup for a supply chain attack, and this is where the bad guys compromise someone who, generally speaking, writes or has the capability of writing software which is used by a large portion of a particular relevant community.
And so Rust maintainers write software that is relied upon by hundreds of thousands or millions of businesses, including some businesses which have money for the usual reason businesses have money, and other businesses which are professionally in the business of moving money. If you compromise a maintainer of a well-beloved open source program, you can then publish the next version of that program and include an attack in that, and the users of that program will download it in the ordinary course, and then you can compromise their businesses as well.
And this is also something which we need to become increasingly aware of on the email compromise side. Historically, when people compromise an email, they don't make multiple hops — aside from maybe they compromise your email and then ask your treasurer to send out that organization's own money.
But now there's capability to impersonate you successfully over the course of multiple months and multiple emails. You could imagine someone compromising an email for no reason other than "let's see who they talk to," waiting for many months, and, for some percentage of those compromised emails, using outgoing messages which are then deleted from the inbox to compromise some other individual that trusts the originally targeted individual, with the goal of either getting their money or getting a credential that they have.
You know, I don't write any open source that is actually relied on by businesses, but I send emails to people who do. So perhaps you use, quote-unquote, my email address to compromise them. You use them to compromise the open source software. You use the open source software to compromise the bank, and then you use the bank to move money.
Money mules and the three stages of laundering
Manish: And this kind of tiering up of compromised individuals is already a thing in scams as well. Sometimes you might compromise some less sophisticated people who are not, like, super tech-savvy and whatever, but also may not have that many things to compromise. But you compromise them because they are a person.
They're a person with a bank account that has been KYC'd and AML'd, and you compromise them so that you have another person that is known to various things, and then over time you can take a group of these people and use them to scam someone else, either by doing things through them or using their bank accounts as, like, hopping points to turn a wire transfer into cash that is untraceable to you. And things like that.
Patrick: The bad guys are quite creative and extremely evil with regards to how they do this. One thing that they will do is go to people that they have scammed previously and represent themselves as, "Hey, I'm the agent investigating your case. I'm so sorry you were scammed. You know, I found a work-from-home opportunity recently, and so since you're out $4,000, why don't you use this work-from-home opportunity?"
And the work-from-home opportunity is being a money mule, which is using your own KYC'd accounts to move around money on behalf of the bad guys. The typical flavor text — and there's variance of it — is, we're going to have people who, for whatever reason, can't move money and, due to some flavor text, send money to you. You will deduct 10%, for example, as your own compensation, and then forward the rest to another account that we nominate. And that might be another money mule. And then a few steps down the line, it's a Western Union transfer or similar, to get it out of jurisdictions that will reverse the transfer.
That, by the way — in anti-money laundering parlance, there are classically three stages of money laundering: placement, layering, and integration. Placement, moving cash into the regulated financial system. Layering being introducing hops to delay the jobs of investigators in tracing where the money went. And then integration being taking money from the last layered location to buy some asset in the real economy.
And so that usage of money mules is typically at the layering step.
Manish: Every time I hear "layering" in a laundering context, I get a flashback to the Breaking Bad scene where Saul Goodman is explaining money laundering to Walter White.
Patrick: Jesse Pinkman. He's explaining—
Manish: Oh, he's explaining to Jesse Pinkman. Yeah.
Patrick: Yes. And this is such a good scene that this is literally used in financial industry trainings for money laundering, because in about three minutes and 30 seconds it tells you more than the typical two-hour training done in a federal law enforcement agency.
But the brief version of his suggestion to Jesse Pinkman is, "You are making a lot of money by selling meth. You should buy a nail salon. The nail salon is a cash-based business. You place the physical dollars earned by selling meth into the nail salon. You report those as income from the nail salon, and then you draw it out as the owner of the business and then buy rental properties or similar, and that's the integration step.
In the event that you are looked at by the IRS, this is all explicable. You've paid taxes on it, et cetera, et cetera. It won't throw off nearly as many red flags as, you are a young man with no apparent source of income and a very low number on your tax return, and then you are out and about town flashing a lot of cash and buying properties with money that was not easily explained."
Why law enforcement struggles with internet crime
Patrick: And, you know, it doesn't just happen in Breaking Bad. That is indeed how quite a number of federal criminal investigations start. IRS flags someone as, "Hey, we see this person buying all these expensive things, and yet we don't believe that they earn that income legitimately. What gives?" It is incentive compatible in that the IRS's first guess is always, "Well, they're probably simply not reporting their income to us. We would like that to stop, please." But the other federal agencies that are drawn in often say, "Oh, the reason they're not reporting their income is massive criminal enterprise. Well, we can stop that for all sorts of reasons."
Notably, while I do say this in the general case, law enforcement has historically struggled to do meaningful amounts of abatement of internet-based crime. This is for a variety of reasons. One is that they're sort of culturally optimized to target things where all parties to the transaction are within their jurisdiction. They have higher quality options for arresting people locally than hoping a country that we have a very complicated relationship with will arrest them.
Also, not to put too fine a point on it, a lot of the professionalized scamming operations are either state-sponsored or state-adjacent.
Patriotic hackers and the post-Soviet talent pool
Patrick: One of the things that is well known in the financial industry is that a lot of the post-Soviet Union scamming gangs are staffed by people who are formerly — but formally — members of the state security services in prior nations, and so formerly a member of the FSB or similar.
And they got downsized as the FSB downsized, but they retain relationships with colleagues, and there is some usage of them as — the phrase thrown around sometimes in Russia is "the patriotic hackers," where you're not currently an intelligence asset, but in the case of, for example, a war, we might do cyber attacks against the place we are invading.
This actually happened, of course. And in that case, the criminal gangs are kind of a reserve army of hackers for us. And since you are a reserve army of hackers, as long as you don't hack people that are living in Russia, and as long as you don't target the Russian state, we will sort of give you freedom to operate.
And this is kind of the status quo for the course of the last twenty years or so. I'd hoped for a number of years that one of the things that would decrease scam victimization worldwide was the normalization of relations between the West and Russia, followed by the large tech employers going to Russia to scoop up all the juicy tech talent, and then outcompete the gangs for the people that the gangs rely on to do their activities.
Sadly, that seems less likely to happen given geopolitical events over the course of the last decade — at least less likely to happen in the near term.
But returning to the LLM capabilities frontier: so it's good news for the attackers, who need to come up with scalable communications with potential victims. They need to be able to investigate a large number of potential victims, maintain dossiers on them, et cetera, et cetera. Again, the degree of customization in coming attacks is going to bedazzle us in the worst way possible, I think. It also might be good for the defenders, although I think that will take longer to bake on the defender side of things.
What individuals can do
Patrick: So until industry and until the labs and until widespread societal adoption of LLMs helps us take care of this from the defender side, what can individuals do to make themselves, their organizations, and their families more resilient?
Manish: A lot of good security best practice is still good security best practice. Using 2FA — two-factor authentication — like YubiKeys and things like that, is good.
One important thing is really understanding what is and isn't spoofable. The thing that surprises most people is caller ID — and including the phone number, not just the name on caller ID, the phone number on caller ID — as well as the "from" address for an email that's sent to you. That is completely spoofable.
It is possible for me to write an email or make a call that looks like it comes from you. And some spam detection may detect that, but that isn't guaranteed.
Only trust communication you initiate
Patrick: The thing I would tell people, just to make it simple and be a rule that is easily rememberable, is: you can't trust communication that comes to you. You can only trust communication that you initiate to well-known endpoints.
And so if there is a phone call from a bank — and banks, despite broadly, in some parts of the organization, knowing that there is this ecosystem, will still call people and ask them to do arbitrarily security sensitive things over the phone with no authentication other than "hi, this is your bank," or "hi, this is your bank, I know four digits of your credit card number" that many people have seen before — you should call your bank's 1-800 number on the address that is listed on your debit card or credit card or on their website.
And unfortunately, one attack that has happened many times is that people successfully buy search engine ads for the name of the target organization, inclusive of banks, and then redirect searches for it through that. And of course, many users find it difficult to differentiate search engine ads from the organic results. So, call your bank versus trusting something that quote-unquote your bank or anyone tells you on the cell phone.
A good word to know if you're someone who has to do this frequently: banks typically have internal extensions, or what are sometimes called desk names. A person calling you from a bank and saying, "Hi, this is Frank at Bank of Bigness" might not actually be Frank in the real world, because the bank is aware that Frank has to talk to criminals as part of his job and they don't want Frank to be followed home by those criminals. So "Frank" is his desk name.
So if you ask for an extension and desk name and then call back into the bank, it should be pretty routine for the bank to forward you. "Please forward me to extension 1234," or "please forward me to Frank in the fraud department, or his desk name was Frank." And good news: if it doesn't work, it wasn't that important anyway.
Wire fraud in real estate closings
Patrick: You're at particular risk if you have a large transaction in flight. The most common one historically has been selling houses. And so, the amount of IT security that the real estate industry can afford is not all that much. And real estate agents are not exactly at the top of the educational attainment distribution within the United States, et cetera.
And so a thing that the bad guys will do is pop the email account of a real estate agent, a real estate lawyer, or even a closing company, trust company or similar. And then they will tell the buyer, okay, the account that you should wire the money to is this one — that is controlled by us, or we are using a money mule or similar — and replace the legitimate seller's account. And it is very, very difficult to reverse wire transfers.
What the bad guys will attempt to do is to move it from, again, the original bank that gets the money to another place, to another place, and try to get it outside the perimeter that US banks and law enforcement can usefully try to claw the money back from.
There's been some improvements in the last couple of years. For example, when I bought a house a few years ago, they said, "Don't trust any communications which are sent by email with regards to the closing of this transaction." And I said, "Great. So how am I supposed to get the wire instructions?" "You have to call the title company."
Okay, sounds great. I searched the name of the title company, I found them, and then I called the title company, and the title company says, "Great, we'll send you an email right now with the web application to log into." I'm like, "Okay," already walking back the security guarantees we made, but neither here nor there.
And then they send me to a decently designed web application, which does some security theater and says, "All right, the number that you can wire to is this." And, you know, a couple of years ago — okay, the fact that you were able to do this instantaneously after the call, and you sent me to a well-designed web application, that is evidence in favor of "this is the actual correct number here."
However, if the bad guy were hypothetically in possession of my email account, at that time they might have been able to do the switcheroo to a well-designed web application, because new AIs are capable of many things, and creating well-designed web applications on the fly that have never been seen before and are maximally tailored to this transaction is something that they can do all day long.
So that is unfortunately gonna get worse.
Why legitimate institutions look like scams
Manish: Yeah. And they don't even have to do it instantaneously, because if someone sent me a link to some fintech web application and then I clicked the link and it said, "System is down. Log in in an hour," I would be like, "Yeah, that sounds like a fintech web application."
My general experience with fintech, or just banks and everything, is that they're kinda terrible, and they're janky. And because they're not good at making things work well, things not working well isn't any kind of signal. If anything, it's a positive signal, because maybe the LLM spoofing site will work better than bankofamerica.com. Who knows?
Patrick: This is unfortunate too, because since many banks, inclusive of the large banks, are not internally competent at technology, they rely on a wide variety of partners. And then they break some security guarantees like, you know, "always check that we are using HTTPS and the domain name of the bank."
Well, that's great, but during real estate transactions, you will redirect me to someone whose URL is intentionally not bank branded, and it will be like yourrealestatetransaction.com — which, in the legitimate case, that's designed to not worry any users. But the illegitimate case has the same goal as the legitimate case: of not worrying any users, and having a minimum of signal that yes, this is the authorized portal application for applying for a mortgage, or for signing mortgage documents, or for getting closing details or similar.
Manish: A couple months ago, I had an email — this is not a bank, but governments and banks have the same problem here. I got a call from the LAPD about something. And what I learned when I was trying to verify if it was from the LAPD — I didn't need to verify during the call, but I needed to later — the email that the officer had given me was their badge number at the website lapd.online, which is the fakest website you've ever heard of.
And if you open lapd.online, it doesn't work. The LAPD website is not lapd.online. Their emails just go from there. I ended up calling up their front desk and being like, "Hey, what do your emails look like?" And they said, "Yes, it's @lapd.online." And I'm like, "Wow, this is terrible." But also, at least this is not a scam.
Patrick: I had a similar case once where someone in my family was applying for various immigration things, and so a paper envelope comes in the mail, and this is — due to the routing of this particular envelope, it's gone through at least two national governments at this point.
So I think I trust the contents of the paper envelope, to the extent that one trusts those particular two governments. We open the envelope, and it says, "You need to access the following two documents under penalty of law." It's like, okay, let's access those two documents. And those two documents are linked on the Twitter URL shortener.
Like, I don't trust the Twitter URL shortener as much as I trust the two governments that are most relevant to our lives. But they're telling me under penalty of law I need to do this. So I click on it, and it unfurls the URL, and it unfurls the URL to a 32-or-so-character .com, which is almost optimized to be a phishing test example.
And phishing tests are an anti-practice in companies, by the way, where the company will send a fake phishing email to a large portion of their employees. Some portion of those employees will successfully get phished by the fake phishing email, and then the company will say, "Ha, you were phished. You shouldn't be phished in the future." But the capability of being phished is a bug, and we should solve that. Neither here nor there.
So this thing that was legitimately the way that a large national government chose to publish things looked maximally insecure at multiple steps during the interaction.
Who eats the loss
Patrick: And I think we often blame the user here — like, oh, if the user was just capable of looking at the URL and deciding if it was secure or not, or if the user just didn't fall for it, it would be okay. That has not historically been effective for us. The most effective historical methods have been either interventions directly against the bad guys and against the economic model of the bad guys, or interventions at the structural layer, for example, moving money.
And so, the reason there is quite a bit of credit card fraud every year — tens of billions of dollars — the reason it doesn't eat the middle and working class alive, is that there's a defined waterfall for the incidence of that credit card fraud. And so in the case where your payment method is used illegitimately in the United States, under Regulation E, less than $50 deductible, it's not your problem, it's the bank's problem, and the bank will typically, via the action of contracts and other things, externalize that cost to the business that accepted the credit card from the person who was using it fraudulently.
And even the $50 gets waived almost universally in the United States as a marketing decision. So that sort of societal understanding that there is a relatively weaker participant in the credit card transaction, the end user, and a relatively stronger participant, the bank — and so say, "Okay, credit cards are enormously lucrative. Figure it out, and if you don't, you're the first point of recourse" — has successfully decreased the amount of societal losses with respect to credit cards, even while not decreasing the amount of credit card fraud to a number that is anything like zero.
And I suppose I have to name-check my famous essay, "The Optimal Amount of Fraud Is Not Zero," here. But that observation continues to be true. I'm not saying that the optimal amount of fraud is $20 billion a year, but we would be sad in a world where there was zero credit card fraud in most cases.
Why banks can't just block new payees
Patrick: So I'll ask you the question then. Why don't we just make it impossible at the bank layer to pay anyone that you've never paid before?
Manish: Because I wanna pay people that I've never paid before.
Patrick: Yeah, this is the straightforward answer. Yeah.
Manish: I do that all the time, usually via credit cards, which work better and have that entire system, but also I have done transfers to people I've never transferred money to before multiple times.
Especially if you're like, "Oh, I've got to fix my car," or, "I've got to loan money to a friend," or whatever. There's just lots of different places where you'll do a large transfer for various reasons. And sometimes when you do that, the bank will take you into the room that they have for this, and then they will quiz you about this and make sure that you know what you're doing. But it really depends, and you might still give the right answers while being scammed. It's hard to say.
Patrick: Yeah. Unfortunately, one of the things that the bad guys will definitely do as a business practice is coach you on what are the right answers to get past the bank's scam screening. And I don't wanna say outright that many victims are complicit in their own victimization. That's not quite the right moral calculus. Many victims are, at the exact moment of their victimization, cooperating with the person that is victimizing them. The core definition of fraud is: you tell a lie to gain trust, and then use that trust to extract money from someone. And so at the point of cooperating, the victim has believed the lie, for whatever reason in the flavor text, fair or foul, and they are willing to help defeat some of the things that are put in place to protect them.
Also, the day-to-day operation of those things that are put in place to protect people are often not experienced as happy events by the people in the moment. You know, why is the bank hassling me about my private financial details, et cetera, et cetera. Well, some combination of A, they have to, and B, they're trying to prevent you from getting stolen from.
But there's a reason banks don't always say, "Hey, we're just asking you this question so that you don't get stolen from." Many people who are doing legitimate transactions will say, "I don't feel like being second-guessed. I'm not an idiot. Are you discriminating against me because I'm older? I still have my faculties," et cetera, et cetera. Some people will say that contemporaneously with dealing with some of the challenges of aging. And it is just really, really a ball of wax.
Verification protocols and family passwords
Patrick: But you did mention in your post one novel protocol for, if you have a novel transaction, what is a way you can gain more credence about that transaction than people did historically? And I think it's a well-advised protocol. Do you wanna tell people about that?
Manish: Yeah. The protocol is basically: reach out to the person and ask them to hop on a video call, and try to reference something that is definitely not online or well-known about the two of you. If it's a family member, there's probably events in the past that are not recorded that you can reference.
There's almost always something you can find. It's especially something mundane that you'd never talk about. Just the thing we talked about first on the first time we had a video call that is memorable. Just try to talk about that.
With your family, you can even set up passwords, especially if you've got older family who might appear to get a call from you, and they may not understand that it's not from you. A lot of people I know have told their family members just flat out, "If you get a call from me about anything urgent or money related, know it's not me. Hang up, maybe call back, and if it is me, yeah, we have the password."
Patrick: I've gotten this warning from some family members and think that it is good advice, particularly for people who might feel themselves particularly exposed, either due to the nature of the work you have, due to your financial resources, et cetera, et cetera.
Tell the people who are important in your life that you will never have an urgent call about money with them, and that in the case that there is an urgent call about money or safety, call the police and let them deal with it. The fake kidnappings are one of the scams that goes around, and fake ransoms, and you can fake the voice of someone in the course of a fake ransom call.
And you don't want your grandparents or similar making decisions in a position of stress when there is someone who appears to be you or your child crying on the line. You tell them, "Hey, always call the cops, always call the cops, always call the cops." The cops are gonna default to doing nothing. But, good news, nothing is the right thing to do in most of those situations. And in the case where something is the right thing to do, it will be something decided by someone who was called in by the cops, the specialists in that.
Urgency and secrecy
Manish: And this kind of ties into a very common skeleton of scams that I don't think is actually changing in the world of LLMs, which is: a lot of scams involve — not all of them, but a lot of them — a sense of urgency, and maybe also a sense of secrecy.
And that's one of the ways, as you were saying earlier, people end up working against themselves. If you have this sense of urgency and a sense of secrecy, you might be willing to do all the things that are needed to dupe the bank into letting you just do the thing, or do all the things needed to just get past all of the other societal checks, because now you're kind of on their side. The urgency and the secrecy are putting you in this mental state where you just need this to be done and you're just gonna push through everything that's there to protect you.
Patrick: This is, unfortunately, one of the trade-offs on novel payment methods that can move money instantaneously. Historically, as you go quicker to real-time settlement of things and to non-revocability of transfers, the risk of fraud goes up. And so you try to push someone into a place where they're not able to make good decisions over a period of a few minutes or a few hours, get them to make a transfer, and then exfiltrate the money immediately.
Whereas if they use the previous things like the ACH — ACH has a multi-day period in which you can recall it — and they wake up the morning after and they're like, "Wait, I don't actually think that you're a prince," then they might be able to involve their bank and successfully recoup the money, or some of the money.
Systemic fixes and who pays
Patrick: I ultimately don't think that this is going to get solved on an individual-by-individual level. I think we're going to need sort of systemic interventions at a number of choke points within the financial industry, at the labs, and probably in any number of places.
I think at some point we are likely going to have an LLM or LLM-like thing living in our inboxes to intervene against scams before the flavor text reaches the user. And indeed, your email provider of choice does that currently. They just do it against scams that were quite common ten years ago, versus the probable-scam notification.
Neither here nor there, I wish we could convince the telecommunications industry, "Hey, you have to be at least as good as the email industry in — if it is probably a scam, the ring should not get as far as the handset." That is an absurdity to expose most users of handsets to.
Manish: There are now standards that I think give a better confidence in caller ID. And I believe Android phones already deploy them. I don't know about iPhones. In my post, I linked to Android's announcement about, like, yeah, now we can detect spoofed caller ID using this.
I don't know how perfect it is. Go look at the post. I haven't looked into it. But yeah, there's stuff like that, and I'm sure banks are gonna start doing more heuristics.
In the US, typically, if someone gets your login or gets your credentials and spends your money, the bank is gonna — because of Regulation E, and because of the, like, marketing expansion of Regulation E that you described — you're gonna get that money back. The bank is just gonna give it back to you, even if that means they take a loss. But if you get convinced to send your own money elsewhere, the bank isn't liable. It's just, you tell the police, and maybe the police can get it back for you. They're not gonna get it back for you, but maybe they can.
Patrick: That one's a complicated story, and it's on a case-by-case, method-by-method basis. But to avoid going into a deep dive, I'll say the one that causes the most trouble is Zelle, for a variety of reasons that I should probably cover in Bits About Money at some point. But long story short, if your device is compromised or your login is compromised, the bank will typically take your side very quickly.
And if you are convinced to Zelle money to someone, the bank will typically be very standoffish — even though I think they're in dereliction of their legal duties if they're standoffish there, but I've covered that elsewhere. But you were mentioning about the UK?
Manish: Yeah, the UK since 2024 requires banks to make you whole even when you have been defrauded in that way. I don't know the details of where the limits are, but I think basically if you can convince the bank, "Hey, I got tricked into sending this money," they are supposed to give you the money back. And then if they want to recoup that loss, they have to deal with the police, and probably they're just gonna eat it on their balance sheet.
Open weights and the coming wave
Patrick: I know it's a sobering talk today. It probably isn't the only sobering talk that needs to happen over the course of the next six to 12 months, because the LLMs are getting good at a lot of things. They're not going anywhere. And right behind the frontier by three to six months or so are the open source models.
And broadly, I'm in favor of open source. I think the open source models are a complicated thing to explain to the world, because the novel capabilities of LLMs are something which needs to be controlled in some fashion. And the technical reality of open source models is that if there are controls built into the models to, for example, refuse to do hacking, then you can, quote, "abliterate" those models — to essentially reach into their mind and rip out the circuitry that doesn't wanna do hacking.
And that is relatively successful. And so as a consequence of this, the bad guys will be able to do nasty things with the open source models at a capability which is equivalent to the market-leading capabilities as of today, in only a few months. And I think the bad guys have, like much of capitalism, underinvested in LLMs in the last year, but that will not continue being the case forever.
These things are just ridiculously capable in a wide variety of business contexts. And, you know, clueful operators are going to increasingly do capital allocation to have experiments on that, find that the experiment works very well, and then scale it up. And so we are going to just get hit by a tidal wave.
And "we" being the financial system, organizations that control a lot of resources, individuals that control a lot of resources, and ultimately individuals broadly. I think it's broadly under-understood in the public that groups in society which are known to be vulnerable to scams, such as the elderly, have dealt with a very, very bad security environment for the last 10 to 20 years, and increasing over that interval.
The stories you will hear from your elder relatives at the moment, and particularly from people in their social group that were scammed, are just horrific. And if you happen to take one's cell phone for a few minutes and just see what their inbox looks like, it looks nothing like your inbox. Much, much worse.
And I predict fairly confidently that we're all going to be facing a threat environment that looks like that, and need to deal with it in our individual capacities and as sort of responsible decision-makers for infrastructure companies and other companies that can either choose to get scammed or choose to not get scammed.
Manish: Yeah. And the thing about the frontier models, and the thing about the capabilities present — for a lot of these things, you don't need the Claude that can solve the Jacobian conjecture to do scams. You need much older — like, older capabilities are fine.
The open source capabilities for a lot of this are already there. I am not an expert in exactly where the capabilities are, but the way people have been talking about them, they seem pretty good to do a lot of this. And a lot of any shortcomings can often be worked around by more just design work.
Patrick: We're years past obsoleting some of the old security advice, where the old security advice might say, "If there are misspellings in email or strange syntax for a native English speaker, that's probably a scammer."
And GPT-3 had produced very fluent English sentences on behalf of many people of varying levels of skill in the English language. And again, for good and for ill. It's a good thing to allow people to be able to send emails to their colleagues and customers that are spelled correctly and fluent.
But we shall see where this is as the technology develops and as their countermeasures develop. But thanks very much for taking the time to speak with me today, Manish. And where can people follow you on the internet?
Manish: I'm ManishEarth on Twitter and on most platforms, I guess.
Patrick: And I will put a link to your post and blog in the show notes.
Thanks for tuning in to this week's episode of Complex Systems. If you have comments, drop me an email or hit me up @patio11 on Twitter. Ratings and reviews are the lifeblood of new podcasts, for SEO reasons, and also because they let me know what you like.
