Fixing government technology, with Mikey Dickerson
This week, I'm joined by Mikey Dickerson, who is probably best known for leading the healthcare.gov rescue effort. After that he founded the U.S. Digital Service, and most recently co-runs a consultancy for organizations that find themselves with just-short-of-intractable problems. We talked about his experience in government, my experiences with VaccinateCA, why government has quirky preferences in buying software and how this drives poor end-user experiences, why failure is generally a systems issue rather than narrowly a workforce competence issue (in his view), and how political considerations sometimes sabotage service delivery (including unintentionally).
There are, unfortunately, no magic bullets here, but would-be reformers and the broader body politic would benefit from knowing why things are broken, and why well-intentioned past efforts have (in some cases) made matters worse, not better.
Sponsor: Mercury
This episode is brought to you by Mercury, the fintech trusted by 200K+ companies — from first milestones to running complex systems. Mercury offers banking that truly understands startups and scales with them. Start today at Mercury.com
Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group, Column N.A., and Evolve Bank & Trust; Members FDIC.
Timestamps
(00:00) Intro
(00:24) Government software procurement
(06:02) Fighter planes and requirements
(08:37) Software development cycles
(11:37) Deadline challenges
(12:18) California vaccine scheduling
(16:15) Pandemic priorities
(17:27) Sponsor: Mercury
(18:40) Government employee competence
(22:30) Government pay scales
(25:56) IRS modernization reports
(27:48) System modernization plans
(34:33) Healthcare.gov lessons
(40:29) Feedback loops in civil service
(44:09) Legislative expertise
(46:49) Applied mathematics
(47:57) Loss of knowledge
(49:28) Tour of duty recommendation
(53:06) Wrap
Transcript
Patrick: Hideho everybody. My name is Patrick McKenzie, better known as patio11 on the Internet. And I'm here with Mikey Dickerson, who is best known as the architect of the healthcare.gov rescue effort. He then founded the United States Digital Service and is currently a founding principle at Layer Aleph, which is a consultancy which does crisis engineering and also complex systems engineering. And they have a book coming out about crisis engineering next year, right?
Mikey Dickerson: That's right. We expect some 2026, but it's really up to the publishers, so you know. We'll see.
Patrick: I'm honored to have you join us today. I've been a big fan of your work from afar and wanted to talk maybe a couple of war stories and then spend most of the time on just the systemic issues of why do we continue seeing these sorts of issues in places like the healthcare.gov rollout. You also did a lot of work on the state of California's unemployment insurance scheme. So I'd love to dig into some more stories, but also the more systems level view of why we keep seeing these commonalities in things like the healthcare.gov rollout, the California state unemployment insurance scheme back in 2020-2021, and the VaccinateCA experience with regards to helping the government find the vaccine that it had almost miraculously caused to be developed in about a year.
[Patrick notes: For readers just joining us, VaccinateCA was a non-profit I lead in 2021 to help ameliorate the vaccine discovery problem. It’s best described in The Story of VaccianteCA at Works in Progress. The experience was a bit of an eyeopener, as we were shipping software (and doing a complicated not-quite-software-but-aided-by-it operational effort) in collaboration with partners in government and private industry, on an issue of immense social concern.]
Government software procurement
So starting with the thing that I have sort of come to believe is one of the central bits of the problem: Can you explain how the government procures software projects?
Mikey Dickerson: Yes. Thanks. I'm really glad to be here. Thanks for inviting me. I wouldn't have known you even knew who I was. [Patrick notes: I thought that was my line!]
I've followed your work on the VaccinateCA rollout too, since our company was inside the California state government for pieces of that, so we saw it from the other side. It's cool to be on your podcast. You've had some, to me, pretty impressive people. Like I saw you talked to Dan Davies, Dave Guarino and stuff. So I'm glad to be here. I'll try to be interesting.
Now your question was how does government procurement work and boy, have there been a lot of books written on that subject? Some of them are next to me right here, and since it's convenient, I'll let your listeners know if they want to know someone who really knows something about this, then read James Wilson's book called Bureaucracy, which I think came out in the eighties. That's a pretty in-depth look at about a bunch of those issues. Or you could watch Pentagon Wars. It's a comedy. It is entertainment, but it is entertainment kind of the same way Office Space is entertainment. Like if I told you it was documentary on how Pentagon procurement works, you wouldn't be misled. [Patrick notes: I saw it back in the day, and would second the recommendation. A related piece of searing-indictment-of-government camouflaged as summer entertainment was the 2016 Shin-godzilla film, which is absolutely not about the 2011 earthquake/tsunami and subsequent disaster response, however would you have gotten that idea.]
Let me see if I can give you a Cliff's notes to a Cliff's Notes answer to how government procurement works. I think the thing that's most under—it's a process. It has 80 zillion steps. Everybody knows that it has a whole bunch of documents and artifacts that get created. It takes forever and ever and ever, like all your favorite government processes do.
But I think the essence of it that would be both most understandable and most frustrating, or the core problem as it would look to anybody whose entire life is not government procurement, is that the government doesn't buy things with anything like the thought process that you or I would buy things with.
I am not the richest person in the world last I checked. So when I interact with a market of any kind, probably when you interact with a market of any kind, I'm interacting with an institution that's hundreds of times bigger than me, millions of times bigger than me. I'm gonna buy a car, and the Ford Motor Corporation and Toyota and Honda, whatever, they have all decided what are the products that they're gonna offer. And my choice comes down to which one do I want to trade money for.
And so I'm gonna go about it in a way which is like, there's a lot of things that I kind of like about cars, but I'm gonna have to look at what's available and decide what trade offs I'm gonna make and whether it's worth it to me to spend this much more for this feature or that feature, or this brand, or that brand. But I'm basically doing the best I can to allocate whatever resources I've got against things that are available.
Governments do not think that way at all because so often the government is buying something that no one else is gonna buy. [Patrick notes: I agree that this is a special case and source of complexity in procurement, but the government also does some really… interesting things when buying things that everyone in capitalism buys.]
Fighter planes and requirements
So if someone like the military decides they want a single fighter plane that can do many missions—they don't go to like the Best Buy for fighter planes and see what's available and make reasonable decisions about what features are worth paying for, which have already been worked out in a market by a big corporation that's trying to make money.
It's just not at all like that. They're buying a plane that no one else has ever bought before and no one is going to buy except them. So instead of having any concept of trade-offs, you start by just writing a list of what you want the plane to do. And if you do that, and this is governments, that means that it's not like even one smart person gets to write that list. It's gonna be a committee of people whose smartness level varies. And then there's gonna be another committee that reviews that, another committee that reviews that.
So if you can imagine by the time you've had 70 different people write on a list every single characteristic that they want—let's go back to the car example—that they want their car to have. The result is gonna be totally incoherent. It's gonna be a car that can drive up a 45 degree slope and can accelerate from zero to 60 in 2.5 seconds. And it weighs less than 2,000 pounds and it weighs more than 8,000 pounds, like all at the same time.
[Patrick notes: The more common case is when there are 2+ requirements where satisfying one requirement makes satisfying another requirement more difficult, but there are, in fact, bid documents which have requirements which are strictly impossible to satisfy simultaneously. I regret, but not too much, that I cannot point you to an example.]
Patrick: And it needs to be submersible too, just in case. [Patrick notes: A long time ago, in a place far far away, a young engineer pulling months of consecutive death march on an over-schedule over-budget project for a university was informed by his superior that, to give the customer a win, we’d now make the site fully translated, for the benefit of approximately four of several thousand users. “How hard can it be? You speak the language at issue.” I almost had a breakdown on the spot.]
Mikey Dickerson: Right? And it needs to be amphibious, and it needs to have a winch and it needs to have trail lights, and whatever—like any niche feature you can imagine that someone might want their car to have is gonna be on the list.
And then they're gonna post that list basically and say, who would like to bid to build this car? The point I'm trying to make is that there's no back pressure on any of these requirements. There's no point in this process, even the people that are bidding on the car. And it's gonna end up being, obviously it's gonna be Lockheed or it's gonna be Boeing, or it's gonna be something like that. They have very, very little opportunity in the process and they have no incentive at all in the process to look at this list of requirements the way like a product manager might do at Toyota and say like, sure, I understand you want an amphibious car and you also want a sports car, but they're not the same thing. Like you can have one or the other, but if I try to make this car both, it's gonna suck for every possible job.
Patrick: And as a contractor, if you say, you know, point number 13 is going to drive up your costs by 20%, I really suggest that you don't do that. One, your employer will be furious at you for costing them 20% of the deal. But two, the person you're talking to both probably doesn't care and probably can't do anything with that information anyhow, because they are not scored on the cost of it. They have been given a document by the committee and their only job is to get you or some other vendor over the line against the spec that they have been given.
Mikey Dickerson: You got it. So you understand, and it's even more that than anything else. Absolutely. So whatever—I didn't plan out this example, so all of my details are dumb, but like my imaginary car required a winch and I'm the person bidding on the car. I might know perfectly well that winch is gonna be used three times out of the 40,000 units we're gonna build and it's gonna add 25%, as you just said, it's gonna add 25% to the cost of the car. You're gonna pay $15,000 in order for me to design the frame around that winch.
I have no incentive to say that. I'm gonna get punished if I do, because as you said, it's just gonna bring the cost down, which nobody wants. The builder doesn't want that. The builder's generally paid cost plus, so more is more. And the government doesn't want that either, 'cause the government doesn't care.
That's another part of this process that is like, the trade off is utterly missing. When I go to a store and look at what's available, sometimes I have an exact budget, which is how much I can afford to spend on kitchen cabinets, but more often I have a kind of elastic budget. Like if I tell you $21,000, but there's a much, much better option for $23,000. Okay, I'll take that. But if the amount I'm gonna spend is somewhat elastic versus what I can get for the money, that's another part of the government process that's utterly backwards.
Before we start, we make that list of requirements and we decide how much we're gonna spend. And for eight zillion reasons, I don't have any incentive to make that number small. And I have many incentives to make that number big. So if I say, you wanna buy this plane and I also have $35 million to spend on this plane, then absolutely that $35 million is gonna get spent. And it doesn't matter if any of it makes any sense.
Software development cycles
Patrick: Right? And there's just so many layers of broken feedback loops here. One of them is that the procurement process for software works essentially the same as it does for airplanes, essentially the same as it does for buying hammers, essentially the same as it does for procuring services work. [Patrick notes: See generally Recoding America or try reading a bid document for a government software product sometime.]
But the development cycle for software does not necessarily look like the development cycle for steel and concrete structures. In that, in industry we expect to spend 90% plus of the effort over the lifecycle of software after launch date. But the way contracts are written, and then get copy pasted down through the generations at various places in government that want software are procured, they want 90% of the contrary failure to be in the design and build phase and only 10% for ongoing operation maintenance, et cetera. And I owe a shout out to Dave Gino for making this point. And Jennifer Paulka, I believe also in her book Recoding America, has addressed it as well.
Mikey Dickerson: Yes, you are right. And I belabored the story about buying a car because it's much easier to—that's the maximally sympathetic version of trying to tell the government story. Like I don't know a better way to buy a fighter plane because I don't buy fighter planes.
But the reason I belabored the story about the fighter plane or the car is because it's easier to understand. And then if you follow that somewhat, then believe me when I say that, when the government wants to buy an email server, it does the same thing, and then it gets phenomenally stupid. Like you get stupid enough results when we're talking about really bespoke products that no one else will buy. But you get a whole order of magnitude more stupidity when it's something that really does just sit on a shelf.
And there's absolutely no need for the Department of Commerce to have its own email server. But unless it's forced to—I mean, its own email server as in a new piece of software that was written for the Department of Commerce only—no business anywhere would do this. But unless something comes along that forces them to do otherwise, it will go exactly the same way. We'll start by asking Congress for some number, which will be in the billions of how much money we're gonna spend on this. Then we will start by putting together the 70 person committees to make lists of features we want the piece of software to have, and we're off to the races.
And it will just like when we're buying the fighter plane, it'll probably be 5, 7, 10, 15 years before we have anything to show for the effort, even if it went to plan, which it never does.
Deadline challenges
Patrick: This is particularly acute in cases where there is an emergency or other reason for having a tight deadline on things. In the case of healthcare.gov, the tight deadline was largely caused by political exigencies, it being a centerpiece of an administration's agenda. In the case of the general pandemic-oriented software ecosystem, there was a prompt pandemic that we didn't have a lot of software written for prior to. We did the best we can with repurposing some that was off the shelf, but also wrote some net new stuff.
And just the rate at which we can make decisions over the course of even a year, which is an awful lot of cycles in a tech company: I have to say, that was a little bit frustrating. [Patrick notes: Not exclusive to software. Many organizations did not have their protocol for e.g. administering the vaccine developed until early 2021, but that didn’t need to block on the vials being in their hands. The single most enraging anecdote I heard was a prison was delivered the vaccine, a refrigerator failed, and then—lacking a plan which they believed would allow them to safely administer to prisoners in less than 12 hours before the vaccine expired uselessly—they instead distributed all of it to the staff.]
And then after the decisions were made, it being so difficult to cause the government to reflect upon decisions or update in response to stimuli that they were getting from the outside world, made it difficult to unmake the decisions that were not working.
California vaccine scheduling
One thing we saw during the VaccinateCA experience you mentioned—the state of California had a website, MyTurn, which would allow people to see appointment availability and eventually book an appointment to get vaccinated. The state of California engaged the state of California's procurement process to get that for themselves and the county health departments.
But there were several county health departments that felt that the statewide solution would not exactly meet their needs. [Patrick notes: As mentioned in the conversation with Dave Kasten, this isn’t entirely without merit. California counties have populations between a few tens of thousand and about 10 million. One size very much does not fit all. But for historical reasons the U.S. is divided into counties and the national government and several states figured counties were the natural level of abstraction to target.]
So they independently engaged their own procurement processes. And so the team of 60 people, not talking to the other 60 team of people apparently, because why would you do that, came up with long list of requirements, which had many, many things that they asked for.
One thing that they did not ask for was make sure that we have data flow between us and the statewide system. That seems like it would be very important. And so for the couple of counties that had their own county health department scheduling portals, those could not sync information with MyTurn, because that was not in either the MyTurn's bid document or their bid document.
And then after it was delivered in that fashion, the state of California says, well, we have the thing that will work for your county. They're like, no, no, no, no, no. We already had this bid out. We're contractually forbidden and/or it makes us look stupid to turn this off and go with MyTurn. So we'll just have your thing hanging out there and our thing hanging out here, and they'll be non synchronized and occasionally present conflicting data. And pharmacists won't know which one to point people to.
[Patrick notes: Pharmacists got their information about the state’s COVID policies from the same place you did. There was no privileged information channel. As the state and counties routinely changed policies rapidly, sometimes multiple times over the course of a single day, VaccinateCA spent a non-trivial number of cycles informing medical providers that the policy which they believed was the policy, which might have been communicated to them by e.g. chainwide management or from a regulatory affairs circular published elsewhere in their provider, was no longer the policy as of e.g. three days ago, so please consider changing what they enforced at the counter.
We briefly considered spinning up an effort to courier deliver newspapers to pharmacists but it turned out most of them were convinced by a script like “You can Google this on your phone if you need confirmation but as of yesterday the state says that…”
You’re welcome to your estimate of how many of the hundreds or thousands of pharmacists we said that to actually found an authoritative source and how many simply trusted the representations of the first person to talk to them and confidently announce the state of California’s most recently policy. Which, I agree, is terrifying, but it’s so far down the list of system failures that year that it barely even rates this aside.]
And that's just what we're gonna roll with. And indeed, several of the largest counties of California did roll with that.
Mikey Dickerson: Yes. You're pointing there at a pattern that I've seen in government and I've also seen the same thing in the Democratic Party. And I'm sure it's probably the same in the Republican party. I just don't know. So I don't try to speak to it.
The pattern being that we have, in effect, these—like the society, the nation has collapsed into one entity for a lot of purposes. There's just one election going on in any given year, it's the presidential election. I mean this is not literally true. I'm saying this is what has become like spiritually the case. The housing policy is a macro scale thing that's gonna affect all of California and probably gonna affect the whole country at once, but we don't have structures of control that were built to cope with that kind of scale.
We have this massively decentralized, disempowered Governor's office that can do a couple of things, most of them PR oriented.
[Patrick notes: One of the most disquieting things I get from reading Dominic Cummings was the sense that senior British leaders are almost entirely driven by the news cycle. I wish I could say that I am confident this is a peculiarly British affliction.
But I have abundant private evidence that it isn’t. I cannot share the anecdotes, because sometimes the price of sitting down at the table is keeping secret what is said at it.]
Then we have a whole bunch of disparate agencies. California's especially hilarious because it has like four or five different independently elected officials. Like a California insurance Commissioner, if I'm not mistaken, is a separate job, which is elected by a different process. Like they're not accountable to the governor. So you, this is like having co-branches of the government that don't answer to each other. Attorney general, same thing. Then we got county governments, then we got city governments, and I'm probably forgetting even others.
But that leads to this massive duplication of effort, like you said, 'cause none of these groups has really any incentive to collaborate with any of the others. And it leads to very bad outcomes a lot of the time that are really hard to do anything about. And that's certainly true when you're talking about building a website for getting vaccine information.
And it's also true when you're talking about California's going through a fight right now over—I don't even live in California by the way, but I lived there for a while. And its policy fights reverberate everywhere. It's housing. Housing unaffordability is a problem that everybody agrees is a problem. And there's much too much devolved local control and much too much ability of any particular tiny group like neighborhood association in San Francisco. It's literally this way, like a tiny, little neighborhood association of like two or three blocks has been empowered to stop any project in its tracks indefinitely.
Pandemic priorities
Patrick: When I was in San Francisco in 2021 to run the VaccinateCA project, I got a letter in the mail from the city of San Francisco saying someone two blocks away from me wanted to build an apartment building, and if I wanted to register my objection, I should show up at 6:00 PM on a particular day.
And this is in the middle of a pandemic where I was being shooed out of Chipotle for attempting to eat my food at a table. I did not understand that was illegal, because I had just arrived off the plane. But you know, California had its priorities. Churches, closed. Schools, closed. Chipotle, takeout mode only. But for objecting to housing, like, oh, no, we will definitely tell you show up at six. We're still in business.
Mikey Dickerson: Yep. Absolutely. That's because—you mentioned Jen Pahlka, and she tends to describe it as there's way too much stop energy. [Patrick notes: By coincidence, a colleague at VaccinateCA taught me that term the same year. They described me as occasionally exuding it to a degree which wasn’t helpful, during discussions about prioritization.]
So the government has some amount of go energy in the tank, has some amount of power to go if you push the gas pedal and has some amount of power to stop if you push the brake pedal.
We have built these things. We have built all of our distributed system of government in such a way that the brake pedal is many times more powerful than the gas pedal and we're pushing them both all the time. So, you know, we're just creating a lot of noise and a lot of heat and very little useful work.
Government employee competence
Patrick: Yep. So something which is advanced frequently is the notion that government can't fix the state of affairs because government employees are incompetent. And I've come to believe that the situation is a lot more complicated than that. But you've had the opportunity to work directly with people. To what degree is it a competence issue? To what degree is it a cultural issue? To what degree is this more of a systems issue than either A or B?
Mikey Dickerson: Oh, that's—I really should get better at doing this kind of interview 'cause that also is like, should be a multi-volume book answer and I'll do my best.
I've been all over the map myself in the last 12 years of, what is the answer to—let's take just a competence angle for a second. Is the problem that government employees just suck at everything and they're terrible at their jobs?
I went from this kind of standard assumption, certainly the kind of assumption that you get baked into if you work in California and Silicon Valley, that yes, that is the problem. I'm even being more sympathetic than a lot of people who dislike the government would be by saying that you're not gonna attract the best people with the working conditions that you're offering. So of course it's mediocrity all the way through.
So I started with that assumption. Worked in the government, found that there's exceptions to that. It's not, certainly not always that way. There are people who are trying really hard. There are good people, there are people who want to have a career in public service for reasons that make sense to them, even if they're not the same factors that the people working at Y Combinator are optimizing for.
And where I'm at now is—all those things are true at once because we're just talking about law of large numbers here. The government is so goddamn big. It's three something million civilian employees in the federal government alone. [Patrick notes: Approximately 1.8 million civilian employees and 1.3 million active duty servicemembers.]
I don't have any idea what that number becomes if you include state governments and county governments and city governments, and then quasi government things like the animal control agency and the housing finance agency and all these things that are like, not really government, but kind of effectively are. [Patrick notes: I would also mention that there is a vast contractor class and, as noted widely, a large expanded cinematic universe of non-governmental organizations which are providing a service to the government’s specifications on the government’s dime.]
You include all that and you're talking about like a third of the population, a third of the economy, something that's in the ballpark. [Patrick notes: I’d ballpark at about 25-30%, too. This is low by the standards of peer nations, which go up to about 40% for our closest peers and then crack 50% in more troubled European economies.] I don't know what the number exactly is, but it's a massive fraction of a really big number, which is 350, 400 million people in the country, and it's not possible to have a group that big that doesn't span the entire gamut of you have wildly incompetent people, you have mediocre people, and you have some really smart people. They're just all there.
So that question just needs to be set aside, like it's not helpful. You might as well ask like are Americans incompetent compared to Canadians or Germans or some other similarly sized enormous population. The statistics say that can't have an answer because when you're talking about something that big, the population mean is only gonna be different by some tiny fractions of a percent. [Patrick notes: I would politely disagree with this assessment; that isn’t what we see on e.g. PISA scores. But I can understand why we didn’t go down that rabbit hole in real time.]
So on competence, I think the answer is just unask the question. Like sometimes they're good, sometimes they're bad. Don't think about it. The question means just as much as if you were to ask whether American private sector employees are incompetent. To that question, almost everybody, including people who live entirely in Silicon Valley, will immediately see that the answer is, obviously some of them are and some of them are not. There is a whole spectrum and everybody I went to high school with is doing something somewhere right now, for the most part. And as I remember my high school class, it spanned the whole gamut from really smart to really less so.
Government pay scales
Patrick: I think the more sophisticated version people bring to the table is one, the sort of incentives rule, everything around this point. The GS-13 pay scale versus what Google will pay a software summer intern is pretty painful to look at.
For people who don't speak government-ese, GS-13 is particularly like occupational rung on the government pay scale letter, which is what you would do for, you know, a senior scientist or a manager of scientists. It's a very high rung. It's 13 outta 15 on the federal scale. And slightly less than what Google pays summer interns cash wise. Last time I checked the numbers.
I do think that private sector has aspirationally, and I think everyone who is in private sector agrees that this is aspirational, a culture of rigorous performance management where the government has some places where it does generally have rigorous performance management. [Patrick notes: Off the top of my head, the most salient example to me is elite military units.] It is large, it contains multitudes, and then it has other places where performance management is illegal by law and/or under like a state constitution. [Patrick notes: I refer you to the city of your choice’s most recent series on how difficult it is to remove underperforming teachers. If you don’t have a city in mind, try New York.]
Mikey Dickerson: Yeah. And as someone who was middle management at Google for six, seven years, my knee jerk reaction to—you're right. It is absolutely true. I do believe that incentives set up kind of energy gradients, which drive whole populations in one way or another, even though there's individual variation and can't at this kind of large scale.
And I would grudgingly with a gun to my head have to say that Google did performance management better than the government. But the difference isn't as big as most people think. We were not great at performance management either, in that company or in any—I don't know any manager at any private company anywhere that thinks that they're handling that well.
I hear very similar stories from a friend who manages a government unit about the person who is an endless sandbag tied to their legs that they can't do anything about and can't fire. And like when I managed at Google, I had people that were sandbags tied to my legs that I couldn't do anything about and couldn't fire. So I'm not saying they're the same, but the difference isn't as big as people think.
And I do think much more nowadays, I very much more tend to think in terms of—the individual people in large groups have certain characteristics that are predictable. We're not gonna get magically better at performance management next year. We're not gonna get magically better at it at Google. We're not gonna get magically better at it in the IRS.
So they're just a part of the system. They're an inextricable part of the system with the technology parts of the system. And it is the interactions among those components that lead to the complexity and it's the emergent behavior of the whole system overall that is what we're trying to improve.
So attacking—and you talk about complex systems a lot on this podcast. So I don't know all the aspects of the faith around here, so I might get them not quite the same when I talk about it—but the competence, performance management structures, incentives, pay scales, all that, all those matter. And they matter in ways that may or may not be relevant, that may or may not be significant when you compare to all the other systems things that are going on.
Because I could set that problem aside and think for a second about how much better would things get if I stopped using a mainframe to process taxes. We happen to be recording this on tax day, and I happen to know that the master tax file is mainframe based. It has been forever. So should I worry about managing my people better, or should I worry about getting them or re-implementing the current COBOL code in something else? Which one would matter more? Like, it's the emergent behavior of the system that I'm actually trying to optimize for. And sometimes that's really unpredictable in like butterfly effect kind of ways.
IRS modernization reports
Patrick: Yep. I will drop some links in show notes for people, in honor of tax day to various—I forget the formal name of the report, but there's been an IRS committee on modernization for at least about 20 years or so now. And every few years they publish a report on how the IT modernization is going. As a technologist, I don't think there is any document that describes more tales of Lovecraftian horror than that document does.
I could quote parts of it verbatim. One year—and I'll find a citation for this—but they said that the service is overly reliant on the laptops of individual employees being used as the authoritative source of record for documents submitted by taxpayers. [Patrick notes: OpenAI’s o3 is preternaturally good at going from this to this [PDF link].]
Mikey Dickerson: I have no doubt. I haven't read that one, but I am utterly unsurprised if we've evolved to a state where like the individual laptop carrying home on the metro by some IRS employee is a load bearing component in that. That doesn't surprise me in the slightest.
It's more opinion than it is science. I don't think I could prove this. But for us, us being LaLF, the people that I work with nowadays, modernization is a running joke. I was just saying how you, what you actually want to optimize is the emergent behavior or complex system, and that's a tricky problem and you could go about it a whole lot of ways.
System modernization plans
That being said, I don't think projects that are called modernization tend to amount to much because they're kind of just replacing technology for the sake of replacing technology. That's usually what that word decodes to when I go find that 20 year plan, which someone has filed.
And it's kind of a running joke for us that we get hired to try to untangle problems that are perceived as urgent. And every time we do that, in every agency or company, we're there to try and see what we can do in the near to medium term to solve some kind of acute problem. And all the while in the background, we just budget for it in the meeting schedule, we're gonna have to sit through the like three or four or five hours of meeting so that we can be brought up to speed on the modernization plan that the agency has been, you know, the three year modernization plan that's currently in year 12.
That is, if anything now an obstacle to making any changes that are needed because it's this humongous behemoth that's taking a lot of oxygen, will cause people to say things like, "well, we shouldn't spend time on fixing such and so right now, because it'll be addressed in the modernization plan."
Patrick: Yeah. To the limited extent, I think modernization plans have been a useful force for change, it's when people use them as sort of like, okay, we're going to make a new system. We won't completely deprecate the old system 'cause we like some parts about the old system, but we'll reduce the scope for the new system. We just want it to do one thing and we want it to do that one thing pretty well and all the other stuff from the old system, we'll get it to the new system eventually.
But then use that to protect the new system from the bureaucratic, political, et cetera realities that cause the old system to have like 15,000 pages of supporting requirements. And then you hope that you can do the two-step where, well, we like the new system so much better because it does better on like the core mission, that we will direct more and more traffic to it until someone makes a decision to sunset the old system eventually.
Aspirationally, that's what happens.
Mikey Dickerson: If you have success stories of that nature, by all means tell them and I will learn something. [Patrick notes: … Ouch.]
We work in brown fields, we kind of chose that for ourselves, so it's not any great shock that I just know story after story of that process not working.
And I can tell you about how—jumping back to a topic here from a minute ago—California and its counties, like all states got mandated at one point to maintain a statewide immunization registry. So it's got your name in it, it's got your date of birth, and it's got what date you had your MMR vaccine, if any.
Patrick: I happen to know, it does not have my name in it because my name not being in it caused me to be unable to return to Japan one day. Oh the irony.
Mikey Dickerson: That sounds inconvenient. And here's one possible reason why that happened, is all states were mandated to build one of these, and California could not agree on what it was doing. So California has three of them to this day. And they service different parts of the state, kind of, sort of, except for they also kind of overlap in some places, kind of, sort of.
And the outcome that happens much more often with a modernization or second system or system replacement plan—and Fred Brooks wrote about this like 50 years ago, like in the sixties, in the seventies, nothing has changed—the much more common outcome is you get partway through it. You get more bogged down and more bogged down and more bogged down as that list of requirements gets longer and not shorter as the years go by and the goal gets farther away and not closer.
And eventually you cut it off and declare victory with like 40% of the features implemented and you launch it because it seems like if you don't, you'll literally never have anything to show for your trouble, which is also a common enough outcome.
So you launch it, but the old thing never goes away. So literally right now you mentioned California EDD, that's the California Employment Development division. Maybe it's California's unemployment insurance agency. And the way they do this right now is exactly the way I've seen it done in private healthcare companies, that do data integration type stuff is they sit at a desk that has two computers and two monitors on it. One of 'em has the old system on it, and one of 'em has the new system on it. And when you need to get information from both places, you look at both monitors. They call it swivel chair integration. And like you will just stop right there. That's, that will be that way in 15 years. I can almost promise you that.
Patrick: Ironically was one of the surprising things for people from the tech industry to think back in 2021 and VaccinateCA was that we did a lot as a nation of “shipping the org chart.” The county Health Department wants to have a source of information about the availability of vaccines. The state wants to have a source of information. Each individual healthcare system, the Department of Veterans Affairs, et cetera, et cetera. The information gets sharded out into the universe.
And then many actors, in the course of developing, say government websites and similar said, okay, we would like, you technologists to now find it in all the places in the universe and develop an API or a database integration or similar to get all this information onto one pane of glass so that we can present it to users or present it to decision makers.
[Patrick notes: Allocators of the vaccine, mostly in government, had a week-to-week optimization game to play of where to ship newly produced batches of vaccine. They had complex preferences with regards to how to allocate them, to put it mildly, and very little visibility into where the vaccine actually was. You would, aspirationally, not want to continue shipping vaccine to a freezer which already had unused vaccine in it.]
And VaccinateCA, which is mostly staffed by people with tech industry backgrounds [Patrick notes: this became far less true of our volunteer base as time went on, FWIW], said that seems intractably difficult as of day one. But there is ultimately an interface available at every site that administers the vaccine. It is, you can call them up and ask them if they're administering the vaccine right now. They speak one API—English—and one transfer method—the telephone. [Patrick notes: VaccinateCA would have been a very, very different operation if we had had reliable LLMs available.]
And so let's just do that and then construct a completely new data source and put all the curated information that we can find via the telephone API onto that one place. And then we can push that one place out to whoever wants to consume it. But, yeah, swivel chair integration. Definitely a thing in the world.
Mikey Dickerson: Yes. I liked that about your VaccinateCA effort was the, you put together a group of people that were experienced enough or whatever, had the right mindset that the amount of technology here to use is the least that will work if you want to get anything done.
And yes, there already is something called the telephone and we can take advantage of it. And we can do that starting today. And we don't have to wait for the modernization plan that's gonna integrate all the data sources everywhere.
And I worked on the modernization plan that was supposed to integrate all the data sources everywhere and did the best that we could, but it definitely did not get anybody information they could use today about where vaccine dosages were available today.
And that was cool 'cause the stereotype, and you cover this in your writeup, the stereotype here is that the tech people always show up and just wanna do a bunch of high tech stuff that isn't needed. We don't need jet engines, we just need to fix the flat tires on the Honda Accord. That would help a lot more.
And I, and just as a style matter, I lean hard in that direction and do my best, did my best when I had people that reported to me to push them in that direction. But the story the government people have of Tech Solutionism is still deeply embedded and you have to fight against it every time.
Healthcare.gov lessons
Patrick: Yeah. I think the narratives we tell ourselves about these incidents matter quite a bit. 'Sometimes we learn the wrong lessons. And this has been a bit of an eyeopener for me from the VaccinateCA experience.
I'll caveat the following, personal opinion only, not that of the organization or necessarily any partners, funders, et cetera, et cetera, which is how you know I'm about to say the good stuff.
I think the government mostly declared success as a result of its efforts and, given that the efforts were successful, there's very little need to learn anything about the experience. Which is one reason I've been so vocal about it. But if one looks back at the healthcare.gov rollout and your work on that, a thing that I have perceived from government employees that I've interacted with in Washington and other places was that their takeaway from healthcare.gov, it was primarily, big software projects are hugely risky from like a project risk perspective.
And if you are in a position where you could take a political punch to the nose as a result of a software project, not shipping on time or shipping in a buggy state, that is an untenable place to be. So we, the government and, you know, politicos attached to the government are just disclaiming any sort of responsibility for software. Like we are moving out of the way of that boulder rolling down the hill. And we wanna make it somebody else's problem.
And so in 2021, the solution for some degree of solution when we wanted spin up, vaccines.gov, the one pane of glass government website that was going to direct people to the vaccine near them, was to tap on the Boston Children's Hospital's vaccine finder initiative, which had been doing great work in helping people get their annual influenza vaccination for 10 years.
And that was the institutionally blessed, had the right gravitas, et cetera, group of people that the CDC and similar could say, yep, they probably know what they're doing. Like that is gonna be the national solution to this because the United States federal government has learned that it can't do a simple website with three months of lead time.
And I existed a superposition of, I understand how people could have that takeaway, but that seems to be a deleterious takeaway for a large and powerful nation to believe about its ability to ship software into the world. Does that match your sense of talking to people in these parts of the world, parts of the economy, or do they tell you a different thing about how they perceive their ability to get software shipped?
Mikey Dickerson: The narrative you're describing is a big one and probably a dominant one. Again, just because it was big and touched so many people and nobody could see the whole elephant. Like when I run across someone who was part of that healthcare.gov project 12 years ago now, or even part of USDS and all the stuff we did later in the Obama administration, very often they will tell me how that experience changed their approach to work or changed their later projects or something like that. And the lesson they took away from it is usually wildly unpredictable to me.
But it could be a lot of things. Sometimes it's that we should do projects in smaller pieces. That one I like—that I do think makes you more likely to make progress across a long period of time. But yes, that's again, kind of just baked into the atmosphere.
Feedback loops in civil service
Going back to what we, one of those huge topics that we kind of bounced off of a few minutes ago was, what is going on in the government environment? Is that culture? Is it the system? Is it the people? Is it their individual choices? Is it their individual competence? Blah, blah, blah.
A thing to understand about the civil service, federal civil service environment. And I can't say that I know how it works in every state, but I've never encountered one that's different—this is a work environment where there is no reward, there's no positive reward for doing something exceptional. It's not quite true, but it's very close to true.
You could get a pat on the back. You could get an invitation to the governor to give you a hardy handshake and a trophy of some kind, that that can happen. And it motivates people more than you'd expect, given that it's the only kind of motivation that, the only kind of positive recognition that exists in the world.
It doesn't work as well as seeing your friends that invented Gmail get a Founder's Award at Google that's worth eight, 10, $12 million. That has a bigger motivating effect in my direct observation.
So in the civil service, you don't have that. You have almost no way to get recognized for doing something good. You have almost no way of being punished for conventionally bad performance. If your modernization project is in year 12, when you said it was in year three, that's actually normal in the government environment, nothing's gonna happen to you because of that.
The only thing, the only exception that can cause your career to jump off those rigidly defined seniority level tracks. If you attract attention for doing something not the normal way differently and it goes bad, then you'll be thrown under the bus.
And so if you try to picture a large mass of people that you locked into lab cages and you created an environment where there's no reward for standing out in a positive way, there's no punishment for chronic underperformance. The only thing that can happen is you can be punished for exceptional underperformance. Then you have this herd behavior where the only thing they're gonna learn from something like a healthcare.gov project is never try to do a project like that again. It's a survivor game, it's survivor bias.
The people that are in charge of agencies right now have, by definition, gone through 15 to 30 years of never attracting attention for doing something exceptional. So you roll up on something like the day of the healthcare.gov rollout, and all of the GS 14 and 15 and SES jobs are occupied by people who by definition were survivors to get here. And you're like, we need to try some new, risky things. Like, no, like, absolutely not. You might as well talk to the bricks in the wall. They will listen the same amount.
Patrick: For the benefit of people who haven't worked in that senior branches of the US government, I think the SES is the Senior Executive Service, which is sort of the on-call rotating class of executives that the US gives to, you know, when need to pick someone who has experience in running a budget of a hundred billion dollars. There's a list of people you can go to, and that is the list.
But, yeah, I think there's also just a lack of a feedback loop. And often people in civil service point at the elected officials and say it's all their problem. But I think at least some of the problem is that the elected officials are extremely sensitive to a 24 to 48 hour media cycle, and are very willing to intervene in a, you know, six weeks, six month or six year project for 24 to 48 hours if it becomes something which gets media attention, but are very disinterested in like sitting in meetings in year three of the project.
And also don't understand that they have a different hat on and different staffers writing the procurement rules in one year versus, you know, the emergency appropriation bill for, I don't know, California unemployment insurance in 2020-2021. And they don't seem to exhibit object permanence with the appropriation rules, which we passed a number of years ago, and which said, you know, we are absolutely going to minimize graft and fraud in the system. That is our number one systemic priority and the emergency appropriation bill for EDD—like these interact with each other in very important ways, and a lot of the underperformance we are observing as a result of that interaction. But that doesn't get fed back into like, maybe we should update the appropriation rules or similar.
Mikey Dickerson: Absolutely. All of that that you described. Like I said, I have come the—whatever you would call the journey I've been on, I think mostly about systems nowadays. And to me all that you're describing is kind of, again, a problem of scale.
And to be clear, I don't know the solution. This monologue is not gonna end with me saying that the correct system of government is X, Y, Z, because I have no idea. But I know that what we did was we believed in this democracy where we elect singular people to function as executives.
And there's this—I grew up in a small town in Connecticut. I currently live in a small town in Arizona. And there is a scale at which electing a person and then expecting them to be accountable for whether the government's doing its job kind—it can work sort of.
If the big issue in my town this year is whether or not we're gonna continue to fund the county pool or not, I can reasonably expect the elected county supervisors to understand what the pool does. Like they can walk over there and look at it. They can see how many people are using it. They can look at the bills, and they'll mostly understand the bills. Like, we gotta pay a lifeguard, we gotta pay a maintenance person, we gotta fix the concrete, blah, blah, blah.
Legislative expertise
I can vote for somebody to solve that problem on my behalf as a county resident, and I can sort of hold them accountable for whether they do a good job or not, because I also can sort of understand the pool.
But like you go up even one tier from that. We're talking about the governor of the state of Arizona who administers like benefits programs that they have absolutely no hope of understanding. The complexity is far, far too much for a person, especially when that person has been selected for their ability to win a statewide election, which is mostly a beauty contest, is mostly, you know, it's about PR and messaging, doing the right thing politically.
Patrick: Similar for the legislature. You're asking everyone to be an expert on, you know, every field of human endeavor simultaneously, or at least all those that fall under any sort of oversight. And then ultimately they are shelling out to, you know, 22-year-old legislative aids who are largely bright people, but have no background in, for example, heart surgery. And that's who will write the law about heart surgery.
[Patrick notes: For a further discussion of the role of legislative aides and legislature-attached lawyers, see my discussion with Daniel Golligher.]
Mikey Dickerson: I believe it is just literally an unsolved problem of human civilization. I assume worldwide, because I've done a lot of talking to governments outside the US in the last few years, and mainly it has served to convince me that if I visited every single country in the world, the variation would be—I can tell you what the 95% confidence interval is. There's like one or two standard deviations in any direction on efficiency, or outcomes or whatever. But no one is an order of magnitude better than anyone else at any of this.
It's just an unsolved problem of how to get something as big and as complicated as like Medicare to actually apply the things that we even know. Even the things that we know—there are things that literally no one knows. That we just haven't done science for.
I'm jumping all over the place, but I also worked on organ transplant policy recently. How donated organs get allocated to potential recipients. And there's a lot of trickiness to that problem. Unlike almost everything we work on, like even as a computer scientist, the answers aren't obvious. 'Cause it's a multidimensional optimization and those are notoriously tricky.
So there are things that we just don't know about it, but we just don't have the organizational science or capacity to take the things that we do know and cause 'em to be consistently applied in effective ways.
Best we've got is this idea that we vote for the primate that we like best, how they appear on TV, and hope that they will also be so attractive that they will attract smart people to make the decisions for them and on their behalf, and that somehow they'll be incentivized to make those decisions good. Like that's the system we've got.
It seemed to—we kind of believed in it during the Obama administration. Of course I would say that, 'cause I was an Obama appointee. We kind of don't believe is working very well now, but that's our only hope.
Patrick: I like the point about managerial science because that is an actual thing in the world. There are some forms of science that we just don't have access to yet. And then it seems like we are unde applying things that generally do exist.
One of the things in the writeup of the EDD, I'm gonna quote directly because I just love the sentence: "As long as the inbound rate of claims per day exceeds what the slowest, lowest throughput part of the system can complete in a day, the backlog of undetermined claims will grow unbounded until the new claims rate decreases."
It's bolded in here because it was very important for the decision makers to understand it. And this is like a classic operations research 101 result, which was well understood in the 1950s, but is still surprising administrators in the recent past. [Patrick notes:
Mikey Dickerson: That is right. That was probably a Marina sentence. [Patrick notes: Marina Nitze, formerly CTO of the Veteran’s Administration, is one of Mikey’s co-founders at Project Aleph. She recently wrote a piece for Reason which directly inspired me reaching out to have this discussion happen.]
And it's the kind of thing that you absolutely will find—I'm gesturing to my pile of books that the listeners can't see again, but that has been a thing that has become like a hobby of mine. I'm just fascinated by the amount of organizational and management science that we once took seriously.
That came out of the war. That's what happened. 1950s, 1960s. We had cybernetics. You talked to Dan Davies a couple weeks ago, his book was great. It's The Accountability-Unaccountability Machine, I should say. And we were working on all this stuff. We knew things like that were super obvious. They're in these diagrams that were type set on these IBM electric typewriters that ran corporations in 1962. We just kind of forgot all about it.
Like I went to work for the government. I was appointed by the president to work for the president almost directly, pretty close, and there wasn't anybody anywhere that had any of that information at hand that could say like, what's actually going on with the DOD problem here is explained in this chapter of this book that was written in 1962, which it was, but it's just lost to history. Like the ruling class right now has no awareness of it.
Applied mathematics
Patrick: On the one hand, this is gobsmacking for me to hear. And on the other hand, it mirrors the American private industry's loss of the same technology for a period of decades, immediately after the war where, most famously the American automobile industry completely forgot everything it learned about statistical confidence intervals testing and a bunch of other pedestrian applications of high school math.
And then Japan found this English gentleman [Patrick notes: I misremembered: Deming was American] that really got the message and all but made him a national hero over the course of a few decades. And worked out pretty well for Japan to have a worldwide monopoly in applied mathematics, is how I describe it, for a few decades.
Mikey Dickerson: They certainly won the car market for a good 20 or 30 years before anybody started to find their bearings. That's a really important point that I think technology people, that it's not obvious to them is that both science and technology are not a monotone function where we get better year after year, we also lose stuff.
And it happens all the time. It happened all the time in history. And it so happens that right now we're at a point in history where we have lost things that we, the United States once learned about running large organizations.
Tour of duty recommendation
Patrick: So, wanna be respectful of your time. I know a thing that you've recommended to people over the years is to take a tour of duty after perhaps a career in tech and try to do something in the National Service. Given 10 years to see how this advice works, do you think that's a good use of people's time? Would you still give that advice or are you focused on different levels of the stack at this point?
Mikey Dickerson: I still basically agree with past Mikey. As far as that goes, I still think it is probably a good idea with a major asterisk, which is there needs to be a landing place for you that meets a bunch of conditions.
I will tell you a thing that doesn't work. I've tried it enough times, and I don't recommend anybody try is to try to go help an institution that isn't asking for help. It does not make any difference. [Patrick notes: Enthusiastically co-signed.]
Pick any institution. You've probably got one in mind right now. Any institution that is very obviously failing at its job as you define it, if it does not believe that itself, and is wanting to change itself—the motivation must come from within—and willing to make a space for you to get anything done.
If the institution you wanna help is not already receptive, you cannot make it receptive. If anyone knows how to do that, I would love to learn the trick. 'Cause I do not know any way to make a person or an organization care about something it doesn't care about today.
And I'm harping on that asterisk so much because it has been really tough to find places like that for most of the last 10 years. We had healthcare.gov created kind of a generational opportunity there for a couple of years where the way that story got written. And yes, as you've hinted once or twice, it's as much an exercise in storytelling as it is anything else.
But the story, the way that story got written is that things went really badly and then there was this big rescue and things got better. So things can get better if you would listen to the right people some of the time. And that created a window where I could bring in 200 people and have decent chances of putting them in a room where someone was willing to work with them.
I am out of that racket right now, so I can't say anything about how plentiful those opportunities are in year of our Lord, 2025, with Department of Government Efficiency and Elon Musk and big balls in charge. Not there, I wouldn't know. But if you don't have that kind of opportunity, then you're not gonna have a good experience and I can't recommend it. So the conditions matter a lot. If the opportunity does exist, then it's worthwhile.
Patrick: Well, thank you very much for that guidance. It's something I've been wrestling with in my own career as well, when trying to figure out the next thing to do. So we'll definitely chew on it a little bit.
This has been an excellent conversation. Thanks so much for sharing your time with us. Is there any place that people should follow you online for writings about current and future adventures?
Mikey Dickerson: I'm really disorganized at that. I'm a very bad think influencer, but anything that I do, I tend to at least gesture at on LinkedIn. It's the only social network type thing that I maintain. So you can find me on LinkedIn, and from there, I post, I publish things on Medium sometimes, and my company website is layeraleph.com. And that's what I've got.
Patrick: Thanks very much. And I will drop links to those in the show notes. Alright, well thank you so much for your time today and for the rest of you, we'll see you next week on Complex Systems.
Mikey Dickerson: Great. Thanks very much. Thanks for having me.
