Secrets designed to be divulged and other payment oddities

Patrick McKenzie (patio11) deconstructs the "original sin" of payments: building a global financial substrate on shared secrets that were distributed promiscuously to function. He examines the multi-decade game of Whack-a-Mole played by the industry to balance the "optimal amount of fraud" against the catastrophic conversion hit of high-friction security. From the physical failure of terminal buttons to the smartphone finally solving the lifecycle problem of cryptographic tokens, Patrick explores the technical and social reasons why we’ve moved from "something you know" to the "continuity of access" provided by the device in your pocket.

Presenting Sponsors: Mercury &  Granola

If you have more interesting hobbies than managing your money, Mercury Personal is built for you. It allows you to automate movement between accounts—allocating paychecks and tax prep the moment they hit—with a sensible permissions model for partners or accountants. It works the way tech people expect banking to work. Go to mercury.com/personal to experience banking built by the same folks Patrick trusts for his business.

If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS

Timestamps

(00:00) Intro
(01:32) Publishing the shared secret… again
(03:39) Manufacturing shared secrets at scale
(07:51) Something you own, take one
(10:10) Sponsors: Mercury | Granola
(13:48) Something you own, take two
(18:26) Something you own, take three
(21:24) One other semi-successful method: positive pay
(24:45) Wrap

Transcript

Hideho, everybody. My name is Patrick McKenzie, better known as patio11 on the Internet. I’ve got a bit of a wonky one today: secondary authentication mechanisms in payments. It says a bit of everything: widely distributed infrastructure you’ll touch dozens of times this year, curious historical decisions we’ve paid the price for for decades, a balancing act between security and ease-of-use, and a bit of a policy angle. And so here are my off-the-cuff thoughts, which I’m working into a Bits About Money issue for publication later this month.

Authentication and authorization ("auth/auth" for the geeks) classically are demonstrated by some combination of “something you have, something you know, and something you are”. The original sin of auth/auth in payments systems is, to maximize for ease of adoption, they used shared secrets (something you know) between users and their payment providers, but then widely distributed the information which was theoretically secret, such as e.g. bank account numbers and credit card numbers.

We have played a fun multi-decade game of Whack-a-Mole since then, attempting to use so-called “secondary” authentication to defang abuse of payments systems while compromising as little as possible on the ease of use and range of deployment which caused them to be adopted pervasively. Much of it is (intentionally) somewhat opaque to users of the systems, and so I thought I’d make a brief survey of it.

Publishing the shared secret… again

Many credit cards have an extra special super duper secret written on the back of them (or, for Amex cards, on the front, presumably not because someone foresaw the development of the Internet and wanted to complicate a hundred thousand web pages trying to direct a customer to the correct location on their card). This is called a Card Verification Value (CVV) code. The industry calls the longer number on the card the Primary Account Number (PAN).

Your PAN is distributed very promiscuously in the standard operation of your credit card, back in the day via embossing it onto paper, for decades via reading it over the telephone, and on the Internet via typing it into web pages fair and foul. Importantly, it is also maintained relatively promiscuously within card-accepting businesses. Compromises of internal infrastructure were probably a larger mechanism for PAN compromises than scammers convincing customers to tell them the number. No matter how good your criminal gang is at scaling operations, Target is better, so you target Target  for one-stop-shopping for all your illegitimate needs.

Thus the CVV. It’s a double-plus secret: secret like the PAN is, where you maintain its secrecy by always communicating it to businesses who you want to pay. Double-plus secret, though, because the business is told to forget it immediately. This is a charming feature of the Payment Card Industry Data Security Standard (PCI-DSS), and has been slightly more effective than King Canute ordering the tide to not come in. One of the relatively few value adds of a PCI DSS audit, which is required for largely businesses and largely handwaved away for smaller ones, is that someone will actually check whether the CVV gets saved to a database record or written to a log file.

In a recurring theme for secondary authentication mechanisms, CVV is not actually required to charge a credit card. There is a tradeoff between conversion rate (driven by ease of transacting) and fraud rate (driven by ease of transacting); the optimal amount of fraud is not zero, and broadly businesses are allowed to pick from a menu of options the financial industry makes available. Much like your restaurant of choice, many menu items have prices next to them . CVV is a trivial database lookup but that lookup costs someone money (generally a fraction of a penny).

Manufacturing shared secrets at scale

When the payments industry attempts to improve security it runs into a slight problem: there is an installed base of hundreds of millions of users and millions of businesses, and it is broadly a non-starter to make breaking changes to the operations of that installed base. And so early attempts to find a secret so secret we wouldn’t just put it on a piece of plastic handed to a waiter could not, for example, coordinate a global campaign to have banks get in touch with all customers and tell them a new randomly generated code.

So the payments industry put on the thinking cap and asked what information a bank would certainly know about a customer, which a customer would certainly know about themselves, where that was less likely to be known by an attacker in a scaled fashion. They concluded that, because of Know Your Customer (KYC) requirements and the operational exigencies of mailing credit cards and statements, the bank would certainly know the customer’s address. And so, how about we have businesses wanting to charge cards ask the user for their address, and we’ll have a computer system do a real-time comparison of whether that address is correct, and then refuse transactions where the address is incorrect? And because the payments industry likes its three-letter acronyms to avoid needing to say painfully generic product names, we’ll call it the Address Verification Service (AVS). 

AVS almost universally ditched the global scope, because prior to convincing technologists to adopt a universal schema for addresses one has to do so for names, and that is futile no matter how many requirements documents think otherwise. See my classic essay on this topic: "Falsehoods Programmers Believe About Names". Even scoped within a particular nation (or, as in North America, a pair of nations where cross-border commerce is quite common but where socio-legal substrates have substantial variance), AVS was quite difficult to make work.

One reason is, KYC requirements aside, very frequently banks will not actually have the correct address for a user on file.

Frequently “billing address” means “the address on file for this account, as opposed to the address you use for most purposes this year.” There are periodic attempts at various financial institutions to refresh billing addresses on portions of their portfolios, and those are stressful for all concerned. The financial institution usually does not want to actually close accounts over infelicities with the information, gaining the cooperation of hundreds of thousands of people is deeply non-trivial particularly in the absence of a scalable contact method, and most users do not perceive their information not being up-to-date as the sort of prompt emergency that e.g. an account deficiency is. This is in major part because, unlike an account deficiency, one’s information being stale doesn’t block one’s use of the card.

But even in the case where a financial institution and customer agree where the customer lives, it is non-trivial to have a non-involved computer system successfully compare addresses. Many addresses are written differently but semantically equivalent (123 Main Street and 123 MAIN ST). Many addresses are semantically different but routinely not treated as such.

Post offices are welcome to their opinions as to how national addressing schemes work, but financial institutions live in the real world, and so regardless of what a detail-oriented postal professional might say about the matter, both “Apt 2” and “#2” are acceptable ways to identify an apartment. Similarly, business users routinely label private mailboxes with the pound sign (#) (to avoid calling attention to the fact that it is a private mailbox) which the Post Office believes is contrary to the purpose of private mailboxes. They're welcome to their opinion. I'll keep using the # sign.

And so AVS does not provide a binary match/doesn’t match determination for addresses. It provides a Russian doll of nested statuses. Some combination of businesses relying on AVS and the financial infrastructure they’re touching can either use those statuses in an advisory fashion or fail the transaction without a status at least as good as some threshold. Generally speaking, most businesses do not get sufficient additional improvement on fraud after matching a five-digit zip code to justify how many legitimate transactions they’d decline over e.g. a user failing to “exactly match” the spelling they used on an account application several years ago. 

But AVS gives you some options in the middle, such as just requiring someone get the numeric portion of the street right, on the theory that this is the least likely part of the address (after zip code) to have a user give different businesses different representations of it.

Something you own, take one

Prior to ubiquitously available mobile phones, there was brief experimentation with giving single-purpose electronic devices to payments customers. One mechanism these devices used was time-based one-time passwords (TOTPs), where essentially there is a secret embedded physically in the device, used with the time of operation to generate a short pseudorandom number. Your financial institution knows which secret is on your device and knows what time it is, and so they can predict what your device should display right now, and require you to know that number to transact.

TOTPs via single-use devices have largely fallen out of favor. One of the reasons is that the logistics of device management are terrible. This is sometimes referred to, in industry, as the “lifecycle” problem. Conveying someone a TOTP device is not harder than conveying them the original credit card, though an early fly in the ointment is that you’ll generally send them in separate envelopes to avoid interception by casual bad guys, and your customer will not understand this and wonder why their brand new card/bank account/etc doesn’t work.

But the loss of the devices is really, really annoying for the bank operationally, and happens frequently. They’re modestly more expensive at industrial scale than cards, but unlike cards the bank doesn't have a finely-tuned process for taking reports of loss and immediately couriering out a replacement to anywhere on earth. (Many banks will happily ship a replacement card to any hotel in the non-sanctioned world. Security professionals are occasionally askance at this, not really understanding that card program managers consider “don’t ever annoy the user on holiday” to be more important than “zero fraud losses.”)

A non-electronic version of this was giving the user a password grid. It would be a sheet with e.g. 12 rows and columns of random numbers on it. When attempting to authenticate a transaction, you’d ask the user for e.g. the 7th row, 3rd column and 2nd row, 5th column. Since you would rarely re-use particular combinations, the theory was that an attacker would not successfully discover enough of the grid to authorize their own transactions in the future, even if they successfully compromised some transactions in flight or convinced the user to divulge a few grid spaces to them.

We don’t see these much anymore, because users hated these password grids and frequently did not succeed with using them. A lot of people who control financially important transactions cannot successfully retain a piece of paper indefinitely or will not successfully navigate the instruction “find the 7th row and 3rd column on your secret card.”

Something you own, take two

The industry put its heads together and really, really wanted to put the secret on the card itself, but in some fashion that a nosy waiter could not simply compromise it by looking and that an inattentive business would not copy to their records in the ordinary course of transacting. Embracing public key cryptography, which was proving extremely useful in securing Internet transactions, the solution was conceptually simple and a massive technological and operational lift: put a chip on the cards with the secret on the chip, and have the card sign transactions with its secret, rather than conveying that secret to others.

The standard was written in the 1990s, but actually rolling out enhanced cards would have been useless without also rolling out enhanced terminals capable of interacting with the cards. This also required effectively constantly-on Internet connections, which younger readers may not understand were something of a novelty at the time, and operationally difficult for many businesses to procure.

Many businesses had, for decades, operated in a model where terminals probabilistically approved transactions without phoning into credit card infrastructure, then processed the transactions in a batch at, for example, the close of the business day. This didn’t require keeping a telephone line in constant use. The security guarantees a terminal can make without talking to the network are very minimal: “Well, that certainly looks like a credit card number,” yet  there is a check some algorithm called the Luhn check, which, uses most digits of the credit card number to identify your account, and then the final digit basically checks to make sure that the proceeding digits “look like a credit card number”. A Luhn check was once considered a bit of secret information, but of course we had to tell the entire world how to perform this algorithm, and there is a Wikipedia page where a high school student can easily create things that pass that first pass filter as being plausible credit card numbers. Clerks were a weak secondary form of authentication: “Well, that piece of plastic certainly looks like a credit card.” This was weak because cloning a credit card onto a blank magnetic stripe was technically trivial, and if you didn’t know how to get a blank magnetic stripe, buying a magnet and then using on any legitimate card was reasonably effective. This made the bad guys not even have to do supply chain attacks, though they also did supply chain attacks to get plausibly-looks-like-an-issuer-marketing-department-designed-this plastic.

The chip was designed to defeat two attacks: enduring compromise of the PAN secret and also physical theft of the card. It did this by allowing a user to set a PIN, which would be programmed onto the card, and the chip would refuse to sign transactions unless the terminal it was connected to provided the proper PIN (presumably after asking the user to key it in). 

This doesn’t sound like it necessarily requires a quantum leap in payment terminals, but it factually did, and some regions prioritized the rollout more than others did. “Chip and PIN” was ubiquitous in the UK by the mid-2000s. In the U.S. there was a legal shift in fraud liability in October 2015, but adoption lagged even that, as businesses which were not routinely fraud targets didn’t prioritize upgrading their terminals. (At larger businesses, especially, this required upgrading a fleet, often upgrading the software at the same time, perhaps retraining staff, and similar.)

This also exposed businesses to a predicted but handwaved away operational problem: wear and tear on the terminals. Previously, the physical point of contact for terminals had been a card magstripe, and if it was abraded the user could easily get the card reissued; the magnetic reader broke very infrequently as it generally did not have a critical part exposed to friction.

The chip readers, on the other hand, had the same reliability problems that the NES generation remembers at the physical interface between the chip and the reader. That was not the biggest problem, though. It was those darn buttons. Fingers are, at scale, disgusting. Buttons are physically actuated. Buttons which are physically actuated tens of thousands of times sometimes break or become unreliable.

And so, to this day, the equilibrium that prevails in the US is “chip and signature,” often accepting a simple swipe as a backup. There is a bit of risk transfer and pricing incidents to this, but most businesses do not prioritize caring about it.

More to the point: chips are difficult to read in “card not present” (CNP) transactions, such as those conducted over the Internet. They accounted for a huge growth of volume and were a rich target for fraudsters. The industry, in the throes of experiencing the disinterest of professionals at businesses in upgrading their hardware to decrease fraud, was sure that asking users to buy a chip-reader to plug into their PC was a non-starter. (Japan actually tried this, and one of the salaryman’s collection of technical oddities is a peripheral chip reader, useful for e.g. submitting one’s taxes electronically while in possession of a chip-bearing identification card. I've never seen a fax machine with an embedded chip reader, but it seems like the sort of thing that came up in a meeting at least once.)

Something you own, take three

Worldwide deployment of specialized hardware capable of public key cryptography is very hard, and needed a forcing function to make happen. As it turns out, that forcing function is called "a smartphone" and we give them to basically everyone these days.

The payments industry has, as smartphone adoption increased, deprecated many of its prior art on distributing tokens and upgrading terminals, in favor of moving the transaction to the user’s device. For various reasons, the combination of device and phone number is less amenable to compromise at scale than the combination of PC and email address, and it is easier to demonstrate reliably that a particular device is indeed the same one the user has used before.

There are a variety of ways you can use the fact that the user has a smartphone to perform secondary authentication of transactions. One wrinkle is biometrics: something you are. Your phone can scan your face or read your thumb to authorize transactions. The payments industry considers that amusing flavor text but doesn't actually think the biometrics is the interesting part of that story. The interesting part is that the biometrics unlocks the phone, which can demonstrate continuity of access to the device, and the bad guys have difficulty replicating that at scale.

Also, in terms of lifecycle management: it’s difficult to get users to care about their password cards. Their phones, on the other hand, basically never leave their hands, and get replaced quickly when lost or stolen, at somebody else’s urgency and expense. Enrolling the new phone as a trusted device sometimes requires a bit of operational work from the company, but it is operational work amenable to facilitation by a scaled call center, which doesn’t require touching atoms under someone else’s physical control or waiting for a package to reach the user.

Some jurisdictions have experimented with mandatory two factor authentication built around mobile phones. For example, in Europe, Strong Customer Authentication (SCA) has for the last few years required businesses taking payments to require a second factor. The dominant one first was SMS-based one-time-passwords, but this has mostly been supplanted by push notifications to banking apps. This is to minimize the impact on conversion. Though, worth noting: conversion rates for legitimate transactions declined measurably across the board after SCA was mandated.

This is an interesting policy choice, and different polities made it differently. In the US, businesses largely are liable for fraudulent use of payments at them, following a waterfall established by Reg E and contracts in the payments industry. Businesses with low margins and high fraud rates are incentivized to adopt toothier secondary screening of transactions; other businesses are more trusting-by-default and achieve better conversion rates.

In Europe, customers bear losses much more frequently, and that was sufficiently intolerable that businesses were not given the discretion to choose their level of fraud countermeasures, as they mostly have in the US. This essentially socializes some of the burden of fraud from the worst targeted businesses / communities to broader ecosystem, in the form of across-the-board impacts to conversion rates.

It is worth noting that the SCA regulations have a thicket of carveouts, but explaining the interaction of risk scores and subscription systems would get us even further on a tangent.

One other semi-successful method: positive pay

The taxonomy of payment methods is often first split by push versus pull: whether a user paying a business initiates the payment (including choosing amount and timing), or whether the business initiates the payment. Credit cards are, to specialists, the paradigmatic “pull” payment method; although you present your card, it is historically at the business’ discretion when and how much to charge.

[Patrick notes: One reason credit cards took over commerce on the Internet was because the pull is relatively easy to code even with somewhat primitive technologies and poor Internet connectivity, but still returns a result fast enough to plausibly cover with a single HTTP request/response cycle. This is very easy for programmers to reason about

When Stripe, where I used to work (and which doesn’t necessarily endorse what I say in my own spaces), introduced PaymentIntents to make the API robust enough to accommodate push payment methods and credit card payments which had a potential off-device interaction (such as a 3DS secondary verification), this ballooned complexity of the typical integration. I sympathize with users who were frustrated with it, particularly those who didn’t understand why payment methods their customers didn’t use or decisions of governments they never voted for would change their future integration pathway. (The answer was broadly “The world is moving away from a card-centric model even in the U.S., and future U.S. payment volumes will flow over payments that do not map well to the single synchronous pull abstraction. For example, Apple Pay looks like a credit card charge to the user but doesn’t look like one to an integrating developer.”)] 

Many payment methods, such as bank transfers, support both modes: you can push an ACH payment from your banking app, or you can give a business your routing and account numbers and have them pull payment from you. (Confusingly, the paradigmatic push payment, a check written, is actually a pull payment thanks to electronic check presentment. The shared secret between you and the bank authorizing the pull is… written in plain text on every check and we spent extra effort on making it machine readable. This decision cost at least a few tens of billions of dollars, which seems like a large number until you divide by how many checks have been presented over the intervening decades.)

To defang fraud in pull systems, someone had the bright idea of requiring a pull to have a matching push: both parties to the transaction agree on the details or no transaction takes place. The most prominent deployment of this is in positive pay in large U.S. enterprises. The enterprise receives invoices from various parties it owes money to, and a specialist team in the Accounts Payable department communicates to its bank which payments it anticipates making. They then send out checks in the mail. Those checks are presented at various banks, and the bank they are drawn on matches them against pre-authorized checks, rejecting any which are not known to be pre-authorized.

This prevents the quite common attack on checks, where any computer printer is capable of producing a facially valid check given the account and routing number, which—I repeat—we promiscuously distribute to the entire world. At scaled enterprises making large numbers of payments per year, absent Positive Pay, a bank might not successfully guess which checks are authorized or not.

 There's also, as an aside, a common attack where someone intercepts a legitimate check and then simply writes different numbers on the check than were printed before. And so even if you have security features in a physical check such as watermarking or similar, the good use of whiteout or other household chemicals, can allow someone to substitute their own numbers and their own payee onto to that check. And then the bank doesn't successfully detect that before paying out the fraudster.

So why didn’t we do this for consumers? Mostly, because the financial industry meets people where they are. Most consumers do not have a full-time staff managing their household finances. The economy depends on them being able to consent to transactions in advance before knowing the exact timing or amount of them; the electricity bill, for example, probably comes on a predictable cadence but in an unpredictable dollar amount. Neither the bank nor the utility nor the consumer want to negotiate a three-way conversation about that $90 bill, and so we simply accept that some small portion of $90 payments are fraudulent, in the interest of lubricating the interactions of businesses and households of all sizes.

You can think of real-time authorization loops with phones as not-quite-positive pay, capable of being understood by non-specialists, with a far more modest conversion hit than preidentification would have required. And they’re still nascent and painful for businesses.

And so that is several decades of iterative improvements on transactional security. They’re not the entirety of the discipline; much of what is happening happens under the hood and is not directly legible to consumers. For examples, see the previous episode with Emily Sands on how the financial industry deploys machine learning models to sift fraud from legitimate use at scale, without requiring the user do anything special. But this should help you understand the secrets in your wallet a bit better than most people.

Thanks very much for listening and see you next week on Complex Systems.