Delve into compliance theatre

Patrick McKenzie (patio11) explains how compliance regimes designed to be viral brought many more firms into the scope of frameworks like SOC 2. This created a market demand for compliance-on-the-cheap by companies like Delve. Delve has been accused in an anonymous bit of investigative journalism as engaging in Potemkin compliance.

Patrick contrasts what real audits look like with what Delve allegedly delivered. He argues that selling compliance theater as compliance is fraud, not the sort of benign rule-breaking celebrated in startup culture.

Presenting Sponsors: Mercury, Meter, Granola & Framer

Complex Systems is presented by Mercury—radically better banking for founders. Mercury offers the best wire experience anywhere: fast, reliable, and free for domestic U.S. wires, so you can stay focused on growing your business. Apply online in minutes at mercury.com.

Networking infrastructure has a way of accumulating technical debt faster than almost anything else in IT. Meter handles the full stack (wired, wireless, and cellular) as a single integrated solution: designed, deployed, and managed end-to-end so there's only one vendor to call when something goes wrong. Visit meter.com/complexsystems to book a demo.

If meetings consistently leave you with hazy action items and lost context, Granola handles the transcription so you can actually participate and gives you searchable notes afterward. Try it free at granola.ai/complexsystems with code COMPLEXSYSTEMS

Building and maintaining marketing websites shouldn’t slow down your engineers. Framer gives design and marketing teams an all-in-one platform to ship landing pages, microsites, or full site redesigns instantly—without engineering bottlenecks. Get 30% off Framer Pro at framer.com/complexsystems.

Timestamps:

(00:00) Intro
(02:14) The taxonomy of compliance
(04:11) Why compliance is viral
(09:08) Defense in depth
(14:19) Accountability and liability
(16:05) The allegations against Delve
(19:53) Sponsors: Mercury | Meter
(22:41) The allegations against Delve (cont'd)
(24:31) The response and evidence
(29:38) Implausible patterns
(38:22) Heuristics for truth
(40:10) Sponsors: Granola | Framer
(42:52) Heuristics for truth (cont'd)
(44:28) Naughtiness vs. fraud
(51:16) A voice in the startup community
(52:04) Advice for the exposed
(55:36) Wrap

Transcript

Hideho, everybody. My name is Patrick McKenzie, better known as patio11 on the internet. I come to you today as that lowest of all creatures, a pundit on breaking news. Earlier this week, there was a startup, Delve, which was accused of fabricating hundreds of audit reports for SOC 2 compliance, a topic we will return to in a moment. This is theoretically a story about one company, but it also shines a broader light on the industry motion that is compliance, both with respect to this particular regime and a larger set of regimes which are intentionally viral.

[Patrick notes: The “DeepDelver” artifact is dated March 18th; podcasts are frequently recorded before being aired and so the relative timeline may be slightly different when you read this.]

I want to talk about why they are designed to be intentionally viral, what good that was supposed to accomplish for industry and society, and where there is actually the possibility of good compliance versus simply Potemkin village compliance. We'll address a little bit why I find the claims that are made in the independent journalism credible, and some of Delve's responses to the claims.

I'd also like to talk a little bit broadly about startup culture and the disputed link between cowboy behavior and a sense of acceptance of naughtiness and fraud. And finally, we're going to wrap up with some opinions as to where this leaves people who might be exposed here.

So I have some mandatory disclaimers, appropriate in an episode about compliance. One, I've been a HIPAA compliance officer in a past life. I sold the company that required me to be that; I was the person signing for personal liability under federal legislation at one point. Throughout my career, I've been subject to many different compliance regimes in at least two nations.

All comments today are in my personal capacity and not as a representative of any other entity. This podcast, Complex Systems, has previously run ads for Vanta. Vanta is a compliance startup and might, theoretically speaking, be a Delve competitor.

As we've discussed previously in an episode about editorial independence, advertisers do not choose what we discuss on this podcast or any of my other surfaces, and they also do not get to choose our editorial line.

The taxonomy of compliance

So let's talk about compliance certification services. But before we talk about compliance certification services, I want to discuss briefly the taxonomy of compliance regimes. Broadly, you can split them in two. In one set of regimes is auditor-certified, where you, as the regulated or not-regulated entity, do a risk analysis informed by guidelines which are promulgated by somebody. You come up with a list of controls which are responsive to the risks in your risk analysis. And then you pass that list of controls, saying "we've done all these things," to your auditor.

And your auditor is supposed to do two things. One, they will look over that list of controls and say, "Yes, it sounds like it is reasonable and sufficient for the risks that you've identified, in our professional opinion." Two, "We're going to actually test that you are actually doing these things that you have attested that you are doing." In the case of SOC compliance, this is over an observation period, but there are different mechanisms under different regimes. SOC 2 is an auditor-certified compliance regime. PCI-DSS is for larger entities that don't do self-certification.

[Patrick notes: PCI-DSS is the payments industry standard meant to discourage businesses from e.g. keeping around large quantities of credit card numbers in a fashion which could be stolen. A mostly forgotten part of the history of my previous employer is that it was once very novel and very difficult to charge a credit card without coming to know what the credit card number was. Stripe dot JS made tokenization the default way to interact with credit card numbers, to the relief of auditors in many places. It certainly was a breath of fresh air to me, as I was a customer/business owner at the time, rather than someone who actually loved payments industry compliance management.]

That is the other kind of regime, which is self-certification. In HIPAA, the Health Insurance Portability and Accountability Act, there is federal legislation which creates these standards under the Privacy and Security rules. But those aren't actually reviewed by anyone in the ordinary course of things. You do not have to get an auditor to sign off on the fact of your HIPAA compliance as a covered entity or a business associate of a covered entity. What you have to do is write down a list of mandatory and addressable controls and then write down how you have achieved or decided not to implement those mandatory and/or addressable controls. And then provide that to other people in the chain who might be relying on you.

Why compliance is viral

This providing of documents to people in the chain is very important for both of these regimes because they are designed to be viral. In the case of HIPAA, by the text of it, it covers so-called "Covered Entities," which are essentially medical institutions that have the primary set of information about patients, PHI (patient health information). But medical institutions will routinely, in the course of business, do business with people who are not medical institutions. And they need to give the data to those people some of the time. This could be anything from as simple as, well, we use telephones here, and so information travels from us onto the telephone network to a customer, and we don't want that information going missing in the middle.

[Patrick notes: The verbal discussion of telephone networks elided the conduit exception, which is that a mere conduit for PHI doesn’t require a full BAA unless they constructively possess the PHI, which is the sort of minutiae that you can only really love if you’ve ever had to write a memo for a future bureaucrat defending your decision on whether Twilio is a conduit or not.]

There could be core parts of patient care which involve non-explicitly medical entities, or ancillary services which aren't explicitly patient care, but which will inevitably include protected health information. 

[Patrick notes: As an example of core patient care which involves non-explicitly medical entities, you could reasonably arrange with a non-ambulance company for transportation of patients from their place of residence to the place care is provided at. That is essential to care, but the drivers are unlikely to be credentialed medical professionals, and their employer is very likely not a medical institution. They will, however, inevitably learn things about patients — where they live, when they have appointments, and potentially their conditions — that constitute PHI. Those facts will be written down. Every device that sees them is now in Privacy and Security rule scope. The driver’s handwritten checklist that has Ms. Smith’s name on it and the time he dropped her off at the cancer clinic is in scope.] 

For example, in the process of billing, you might have any number of companies involved in totaling up the bills and doing collections, sending a physical piece of paper to a large number of people in parallel. And you need to be able to attest that those companies, which know important things on a physical piece of paper, like a cancer diagnosis, don't leak that information in ways that the American polity would find unsettling.

So in the HIPAA case, you go out to your vendors and you say, "I need a business associates agreement, a BAA, from you, as a condition of doing business with us." If you won't sign a BAA, I can't work with you. If you do sign a BAA, congratulations, you're not a doctor, you've never sworn the Hippocratic oath, but you are going to be subject to the restrictions under HIPAA, just like I am subject to those restrictions. And I am responsible for making sure you do not screw up. And you are responsible for all of your subcontractors. You need a BAA from all of them. You are now responsible for them not screwing up, et cetera, et cetera. And this chain develops.

[Patrick notes: The model language of the BAA is very similar to every one I’ve ever personally reviewed, though in principle two contracting entities can use custom language as long at it is a superset of the obligations under the regime. The BAA will, in a very efficient number of column inches, bind you to almost the entirety of the HIPAA apparatus. Consider this paragraph:

Business Associate will use appropriate safeguards to prevent the use or disclosure of PHI other than as provided by the Agreement or this BAA and Business Associate agrees to implement administrative, physical, and technical safeguards that reasonably and appropriately protect the confidentiality, integrity and availability of the Electronic PHI that it creates, receives, maintains or transmits on behalf of Covered Entity. Business Associate agrees to take reasonable steps, including providing adequate training to its employees to ensure compliance with this BAA and to ensure that the actions or omissions of its employees or agents do not cause Business Associate to breach the terms of this BAA.

That paragraph was a few hundred hours of work for me, of which the most hilarious was locking myself in a room and “training” myself on HIPAA responsibilities, then signing an acknowledgement I had done so. I am extremely aware that sounds like theatre. The computer code to exempt HIPAA-flagged clients of the system from having their Rails form submissions get stored verbatim in production.log was not theatre. Nor was the full-disk encryption for the server with the MySQL database on it, getting a dedicated server in preference to a VPS to avoid a neighbor being in a position to snoop on activity within the machine, writing the procedure document for a server reboot requiring the operator to provide the password without which the full-disk encryption would turn the database into an extremely expensive paperweight, etc.]

In SOC 2, the chain starts with, well, there is some enterprise, and the enterprise wants to attest to SOC 2 compliance for its own reasons. And one of the things that they will say as a control is that all of our subcontractors, all of our vendors, are SOC 2 compliant. How do we know that? Because we have asked them, and we've gotten a certification from their auditor that they are compliant.

This is viral within an industry once it happens. Even if there is a small seed set of customers that really care about SOC compliance, they ask everyone who does business with them. Intentionally, it's everyone. One is not generally in the mood to identify the most crucial five vendors and ask them for SOC compliance attestations first. One wants to demonstrate the totality of compliance here, in the same fashion that one want to demonstrate that all of my employees went to the training on this, all of my employees surrendered their laptops at the time of offboarding. And each exception to that is meticulously documented.

So: one goes to all vendors and asks for certification about compliance. In the midst of prehistory, this required there to be a person whose full-time job it was to email questionnaires to people asking for architecture diagrams and similar, and a copy of the report from their auditors, et cetera, et cetera. And no one loved doing that. But we in the tech industry like automating low-value-added work. Compliance is value-added, but receiving documents about compliance is not that value-added. And so what we've done in a lot of places, in say, SaaS applications which are used by procurement, is to automate requesting an attestation of SOC compliance from the vendor.

[Patrick notes: I was most recently asked for Kalzumeus Software, LLC’s SOC 2 compliance documentation several weeks ago. We have no such documentation. One is welcome to one’s guess as to whether any human involved in the work I was hired to do was consciously aware that I would be asked for SOC 2 compliance documentation prior to me asking for an exemption.]

After you grease the wheels in that fashion, after the leading procurement SaaS and accounts payable SaaS, et cetera, et cetera, have an easy check-this-button to get SOC compliance attestations from your vendors as a condition of them receiving money from you, it sort of supercharges the coefficient of virality for this within the ecosystem. And so this is why SOC went from basically a thing in the Fortune 500, but not really seen by many people, to startups being asked for this routinely the time they do their first enterprise deal, where their enterprise deal might not be a true enterprise deal with the likes of Ford or similar, but it might be a $15,000 pilot with another startup.

And then the team gets asked, "By the way, are you SOC 2 compliant? We need everybody to be SOC 2 compliant." The IT security team is going to review your questionnaire. The startup founder says, "Yeah, SOC 2 compliance. I totally understand what that is." And they Google it and find a firm that says that we will be able to deliver SOC 2 compliance quickly and for cheap. And that creates the market need for a firm like, for example, Delve.

Defense in depth

But before we get into that, what are controls anyway? And why do we care about them? There's a saying in the engineering community that “every regulation is written in blood.”

[Patrick notes: I am most familiar with this saying in the context of aviation safety but it has metastasized like any good bon mot, and it was the subject of a lecture on engineering ethics given to computer science students when I got my degree two decades ago. 

The professor made an enduring impression in observing that many people do not understand that computer code can kill people. He then discussed the Patriot missile system timekeeping bug (subdecimal precision in floating point numbers gets more fuzzy as the numbers get larger and so Patriot missiles would get less accurate the longer they went between reboots). Then we went into the real meat of the lesson, which was probably my single most useful sixty minutes of my education: “Write a memo designed to kill a project before it kills a person.”

I bristle when I am told that computer engineers don’t get a well-rounded liberal arts education with courses on ethics like paragons like lawyers, politicians, and health insurance MBAs do.]

And the less evocative version of that saying is that we don't make up controls entirely randomly. We look at failures that have happened in the past and in the postmortem identify what would have prevented that failure. And for failures which are sufficiently widespread, we say, you know, everybody needs to anticipate the postmortem in the future about this cause of failure, and adopt sensible practices upfront.

SOC 2 is the System and Organization Controls. That's what the acronym stands for. And it is intentionally not designed to be a thoroughly personalized, bespoke, boutique analysis of what the particular security considerations are in your firm. A cryptocurrency firm, which is worried about someone grabbing the private keys, versus a SaaS company which has the ability to upload and download arbitrary documents and is worried about people hosting horrific images on their servers, versus a credit card processing company which is worried about people stealing the money, all have very different considerations under the hood.

And SOC 2 says, "Yeah, we know. But there is some level of routine competence in managerial capitalism that has computers involved in the Current Year. And we want everyone to say that they are at least as good as that very low bar for routine competence." 

One example of a thing that we've seen in many, many postmortems over the years suffices. There was private information and it ended up on a hard drive which was sold on eBay to somebody. How did this happen? Well, the proximate cause of it happening is that we issue people a laptop for them to do their jobs. And that laptop will, in the ordinary course, get private information saved to it. And then someone left our employment and kept the laptop and sold it. And so the information ended up on eBay.

[Patrick notes: For a real-world example of this exact failure mode at scale, see Morgan Stanley's data center decommissioning breach, which cost the firm over $160 million in aggregate fines and settlements after a vendor sold unwiped hard drives containing PII for 15 million customers at online auction.]

There is a concept of layered controls here. One, when someone leaves your employment, you should probably get the laptop back. But even if you do not successfully get the laptop back, there should be an automated linkage between your HR processes and your IT processes, such that HR deciding to offboard someone should result in IT deterministically, within a matter of hours, decredentialing them on all systems that they can touch. But that might not be enough. There might still be information saved on that laptop that has left your control. And so we'll have another layer here. We want all of your laptops to have full disk encryption, so that if it is sold on eBay, someone not knowing the password of the previous at-one-time-legitimate user of the laptop will not be able to access anything that's on the disk.

Even though the disk is no longer in your physical control, there is no reasonable expectation of disclosure as a result of that. But that is still not enough. We're going to add another layer under that. We're going to say, even if we have full disk encryption on, and even if we have successfully recovered most of our laptops, and even if we know we've disabled most of them after leaving our service, there might be some information that escaped all these things. Well, we want to bound what that information was. So we are going to have extensive logging within our internal systems to determine when someone downloads a CSV file from our internal systems onto a laptop, so that we can say, even if Bob's laptop is in the wind somewhere and controlled by a hostile entity, I know that the worst that could have happened is there are 637 rows that were generated in February 2025 on that laptop. I know by construction there are no rows generated in 2026, because if those rows existed we would have logs.

This defense in depth is critical to the notion of controls. What is an auditor doing for you, really? One thing they're doing is they are listening to that defense in depth. And they're saying, "OK, all of that sounds very plausible. Now, let's check your work. So you've represented to me that you have total visibility of people who are leaving the firm. How many people left the firm during this engagement window?" OK, 117. "All right, I'm going to select rows 24, 37, 65, and 73 to 78 inclusive from whatever spreadsheet you're looking at. Now tell me: how many of those laptops did you recover? And what was the time between the person leaving the firm and the time that you recovered the laptop, for each of them?"

And we are going to search laptop by laptop in my self-identified sample for what happened. And in fact, at this point in the engagement, there is often a brief silence and then some terrified Slack messaging. And that is one of the value propositions of having independent CPAs evaluate these things. Because we don't simply want someone to check the box on the fact of, "Yes, I get people's laptops back when they've quit my firm, obviously." We want someone to actually check, because it actually matters whether private information is available for sale on eBay or not.

Accountability and liability

So what are some design features of these regimes? One of them is that they are trying to avoid a corporation being used as, to use Dan Davies' term, an accountability sink. [Patrick notes: Dan Davies has previously discussed this on Complex Systems.]

So they will intentionally try to get named humans to sign up for liability for what are, morally speaking, the actions of their employer. So when I say I was a HIPAA compliance officer, that doesn't just mean that part of my job was writing up a risk analysis and implementing firewall rules to get to our servers, although that was part of the job. (I wore a lot of hats at that company. All of them, really.) What it really means is I, Patrick McKenzie, swear to everybody relevant, including the hospitals we are working with, and including, ultimately, the federal government, that yes, the buck stops with me here.

There is, as far as I know, no SOC 2 compliance officer that has to get identified. But one requirement of SOC 2 is that the board has to be involved. And during your board meetings, you are going to have the board adopt certain policies as policies of the company. And that is not simply a tick-box requirement.

[Patrick notes: See CC1.2 of the AICPA’s standards.]

It's designed to say that, if hypothetically there was a cowboy controls environment, and hypothetically, people were saying things in a fraudulent fashion, and saying, "We have this policy," where that policy wasn't in fact actually descriptive of how the company worked, and if everybody was looking the other way, we look at you, board members. And we see you are more durably connected to this company than line-level employees. You are wealthy and sophisticated, and you knew or should have known the truth or falsity of the representations that you made in a board meeting, and you are signing up for liability if any of those representations are obviously false.

The allegations against Delve

So let's turn to Delve specifically. Delve was in Y Combinator in the winter 2024 batch. It was founded by two MIT dropouts. They raised a $3.3 million seed and then a $32 million Series A at a $300 million valuation, led by Insight Partners, in July of 2025. [Patrick notes: Same TechCrunch article cited for the MIT dropout claim.] 

It has been advanced that the founders are Forbes 30 Under 30 alumni, but I did not verify that as I do not consider Forbes 30 Under 30 a positive signal.

Anyway, the pitch is that AI agents are going to compress months of compliance work into two days. And they say in many places in their marketing copy that you'll be able to achieve compliance in only 10 hours of work by the team.

[Patrick notes: As of this writing the Delve home page cites, quote, “days not months.” The following is taken from a Wayback Machine snapshot of June 2025. I believe the above gloss is substantially representative of their marketing.

]

All right, so I want to say something nuanced right now. We're going to discuss fraud allegations in a few moments, but assume the fraud allegations haven't happened. Assume that you are a responsible professional who has been in the room through real security review processes in the past. As you hear the words, "We are going to achieve compliance in 10 hours," you know what you're buying if you buy that pitch. And what you're buying is not a robust process with a lot of professional time involved. However, do I believe that all customers knew what they were buying? No, I don't, and the mechanism is fairly simple.

Everyone has to learn everything under the sun. I happen to have the benefit of being in the tech industry for more than 20 years and have a particular interest in things like compliance controls. And I am not automatically representative of every YC founder who is trying to do their first enterprise deal ever, where that enterprise deal might be $15,000 and they just received an email saying, "Do you have a SOC 2 compliance story? We really need that to move forward here." And that person might not realize that there is any intellectual effort involved at all here. They might think, "Oh, yeah, simple tick-box requirement, right?" And they might be told by someone, "Yeah, simple tick-box requirement. Check here, do 10 hours of work. Most of it's kind of boring. And it'll cost you a bit under ten grand."

They'd say, "OK, that sounds reasonable. There are all sorts of things in life that are just these little annoying tick-box toll gates in doing business. My registered agent, I understand my registered agent does basically nothing for me and charges $150 to $300 a year for that. And it's just this little tick-box requirement I need because I want to stay doing business in California or Illinois or whatever state you happen to be doing business in." That is a correct read on registered agents. It's just this little tick-box requirement, one of these grifts under capitalism. You pay the check and you move on. 

[Patrick notes: A registered agent is a state requirement that you have a named person (or corporate person) at a stable address who can predictably receive official notices and service of process for your business during business hours. One is required to have one to do business in most states. I pay a few hundred dollars a year, in two states, for my LLC’s. Every year they do the same thing for me: confirm that “Yep, nobody sued you this year, so we didn’t speak to any process servers on your behalf. Wonderful, everyone got what they wanted out of this. Here’s the invoice for the coming year, and remember, you legally must buy this service.”]

And so you might think that about compliance services. And they're a bit of a horse of a different color. They don't verify everything under the sun. They are not the one-stop shop for your entire security posture. But they are supposed to document important things that other people will rely on. It's kind of the designed nature of this mechanism.

So, there were recently allegations made about Delve. I'm going to describe those allegations as being an unsigned work of investigative journalism. I say investigative journalism because investigative journalism is an act and not a priesthood. You don't have to be anointed by the guardians of the tradition at the New York Times to do it. You just have to go out into the world and develop representations of facts, and then ideally ask companies for comments.

The independent journalism has been alleged to be from someone who is competitively involved with Delve. Having read the four corners of the document, and some of the social media chatter about it, them being in some way aligned with a competitor would not surprise me at all. And I don't care. The claims are the claims. And the claims are either true or they're false.

And Delve has released a statement about the claims. So what are the claims?

The claims essentially allege that Delve was doing Potemkin compliance. That it sold customers a vision of: "We're going to be your AI-native one-stop shop for compliance under the SOC 2 framework, or whichever framework you care about. And we will ask you all the relevant questions in a way which minimizes the amount of time your team needs to spend on them. And then we will kick this off to a US-based auditor who is going to do the full audit process. And you will get through very quickly with minimal execution risk. And then you will get the document which you need to pass to your counterparties to get those enterprise deals."

And the journalism alleges that Delve pre-populated substantially verbatim-identical reports for various companies, that the auditors were in fact not US-based, that they were Indian certification mills, that the auditors were in fact not doing independent audits, but were doing substantially templated, substantially identical audits, that the Delve software would suggest so-called "remediation" and controls to people, and then after one click of a button, immediately upgrade those controls to the assertion on a Delve-hosted compliance page that this control is actually in place. So, for example, full disk encryption: click "yes" that you're doing full disk encryption, and we say you're doing full disk encryption, even if the full disk encryption is not actually happening.

The auditors were not actually doing any analysis. They were not asking, "OK, tell me how many employees you offboarded during the observation window, and I'm going to select my own rows from that spreadsheet, and you're going to do a deep dive into each of them for me." They were accepting verbatim the pre-proposed audit report from Delve and just adding their signature to it.

The response and evidence

These are some pretty serious claims, and there are many more in the document. Delve issued what those of us in the PR industry might call a non-denial denial, where they say this is scurrilous stuff made up by some anonymous person who might be motivated by competitive interests. 

[Patrick notes: Out of scrupulosity, I will note that Delve characterizes the post as misleading and that it contains claims which are inaccurate. I stand by my characterization of their statement that they do not materially dispute the factual accuracy of the claims. I am aware this characterization is extremely robust.] 

But they don't actually dispute some of the load-bearing claims there. The independent journalism alleges that the reports are substantially identical, including cases where Delve knew or should have known that the facts underlying are not substantially identical. And Delve doesn't say the reports were not, in fact, substantially identical. They say the reports weren't verbatim identical, which, man, that's a low bar for independent auditor reports, right?

[Patrick notes: Out of scrupulosity I will quote Delve’s response verbatim here and let readers draw their own conclusions:

The Substack inaccurately said Delve uses templates across the majority of reports. This is misleading. Most modern compliance platforms allow clients to adopt a fixed control set based on widely accepted standards set forth by the AICPA, ISO, and more. As a result, the overlap in structure and language across reports is expected. Moreover, Delve uses templates provided by auditors, but we do not prescribe final reports.]

They don't dispute that the report was written before any collection of facts. And what they instead say is, "This is routine in compliance. These are templates and they are pre-filled, and nobody writes a bespoke report anymore." So that's true-ish. If you run a security consultancy, you make heavy use of templates. If you run as a certified public accountant, you make heavy use of templates. Boilerplate is boilerplate for a reason. However, what you don't do is write the entire thing before looking at any of the actual facts of the matter. That's an alarming statement.

And if they're innocent here, there should be replete evidence in the world that the claims are false. And identification of that evidence should be relatively easy, such as: "OK, they are misinterpreting the business process here. No one actually sees the draft report until the full evidence-gathering process is completed. Then the evidence alone goes to the auditor, and the auditor does some sort of tick-boxing exercise: 'Yes, this satisfies my concerns with the gaps in X, Y, and Z. And we know these are legitimate because, one, these firms are legitimate, here's a sampling of them. And two, the average timeline for the auditor to go through this involves 37 auditor hours, which we have access to via the magic of SQL.'" Yeah, they didn't say that.

They did not address the location of the auditors at all, which matters. When you say things which are untrue to induce someone to do business with you, that's fraud. And I believe the representation that Delve routinely said that their audit firms were US-based. I do not believe that their audit firms were US-based. I think that it is risible to claim some of the things that they are reported to have claimed and also claim that your firms are US-based. And that's no particular knock on India. I've done a lot of work with firms based in India over the course of my professional career. But you can't lie about things like this.

And so one thing which is a signal flare visible from space for yours truly: Delve, in the context of a commercial negotiation, was reported to have said [Patrick notes: the following is a paraphrase] "OK, our price is some single-digit thousands of dollars. But if you want a discount, we can find you a discount. And if you want a sweetener on top of the existing independent audit and the software services and everything that we're delivering to you, and the handholding from our customer success reps, we will also throw in 200 hours of pen testing," at a single-digit-thousand-dollar price point.

200 hours of pen testing, if you're securing it from a US-based professional at anything like professional rates, is already a mid-five-figure engagement. It can’t be done at single-digit thousands. It certainly cannot be done as a commercial inducement included in a deal at no extra cost, like a box of donuts. [Patrick notes: My knowledge of routine pentesting rate cards across a spectrum of firms comes discussions with Thomas Ptacek over the years, and if it is dated, it is because those rates have risen.]

The journalism alleges that the pen testing was not actually conducted, which sounds quite credible to me. And Delve makes no mention of this in their response. So acceptable responses would have been: one, "As a business process, no, we never do offer people hundreds of hours of pen testing from a US-based professional as an inducement to do a five-figure contract." Two, "We have occasionally offered that as an inducement, but it would only be on our larger enterprise contracts, and it's billed out at reasonable professional rates for the services." Or three, frequently in company PR: "Obviously, that could not happen in a way which was above board. We are undertaking an investigation to see whether any of our sales representatives have said something to that effect, and we'll make appropriate changes in our business practices based on what we learned in that investigation."

What they say is, like, we didn't do any of that. But they don't deny with specificity the various claims that are made. And they essentially make no representation that they will go forward to improve their business in the future, other than continuing to respond to what they describe as guerrilla attacks.

Implausible patterns

So some of the things in that 10,000-word investigation, which you should probably read, that jump out at me. They say that out of 494 examined reports, which were examined because they were leaked last December and ended up in a linked spreadsheet, they're 99.8% identical boilerplate. The auditor conclusions and test procedures are pre-populated even before the firm uploads their network diagrams or descriptions of the company or other information that you would need to inform your audit approach with regards to the things. And so you can't sensibly describe, like, "OK, the company is using cloud services, and AWS is different than Google, which is different than Microsoft Azure, which is different than some of the new cool-kid startups." And so I will have different procedures for AWS versus all these things, or a different set of scripts that I use to profile what a customer's actual usage of AWS is, because it's a ridiculously complicated set of products. You really need to know whether they're using AWS or Azure before you write the procedures which are designed to test those things.

And they said, no, procedures first, and then tell us what you've actually got, which sounds insane. The journalist says, over 259 Type II reports, that the auditors consistently find zero security incidents over every client over the period, and that this is pre-filled. There's never a personnel change, there's never a customer termination, and there's never an insurance claim. And that just strikes the journalist as odd. 

This strikes me as odd, too. Procedurally saying, like, "No one left your service in the course of a three-month observation window," writing that before the three-month observation window actually happens, that's an alarming set of facts, right?

But just on base rates, people leave startups all the time. Even at the best-operated firms in the industry, there's roughly an 8% regretted attrition rate, higher than that in early-stage startups, which you would assume their customer base skews to. And so, if you assume things like, all right, probably 10 to 20 people on the average team, 250-plus companies looked at, and we see no departures over a three-month period. But on base rates, we would have said, "OK, like 5,000 people in the employed set, and then, all right, times 8% would be very generous to them." That already gets you to: we assume that there are 400 departures a year over this universe. And so in a three-month period, we expect to observe plus or minus 100 departures. And you're saying you observed zero? Really? CPAs are swearing to this? That is facially not credible.

[Patrick notes: Regretted attrition is an unfortunate HR term-of-art for “Someone quit, but we’d happily have them in that chair right now.” Non-regretted attrition is the bloodless counterpart to describe firings, managed departures subsequent to the performance improvement plan, buyouts, and the American versions of oidashibeya, where you invite someone to take that next step in their career journey at the earliest convenient moment.]

Another thing the report alleges is that Delve would let people sort of mix and match on which compliance regimes they want, which is a little weird to this former HIPAA compliance officer. They're not the same regime. Some parts of the controls that they have rhyme with each other, but they have very different intellectual underpinnings. SOC comes from a voluntary industry standard. HIPAA, the requirements are literally written into law and then modified by regulatory decisions and similar later. And so even though you think, "OK, HIPAA is self-certification, SOC requires an auditor for it, so maybe HIPAA is like a strict subset of SOC," that's just not true. And so it is very alarming that this business process would get developed. The independent journalist mentioned something to the effect of they don't think this team is actually domain experts with respect to compliance, and I don't know if I'm a domain expert with respect to compliance, but, having been on the hook for it before, yeah, this doesn't look like something that would be developed by someone who is a domain expert on compliance.

Also, fundamentally, for SOC 2 the deliverable is you want a PDF that says this CPA signs off on me being SOC 2 compliant, and that PDF is doing some load-bearing work, looking like the kind of PDF that someone expects to get. But it's supposed to have some actual information in there, with claims about it. For HIPAA compliance, there is no specific form of the PDF. The only document that has a specific specified form is the business associates agreement, the BAA. But the specific controls that you identify in the course of becoming HIPAA-compliant are actually important. The covered institution, hospital or similar, which is giving you information, needs to know: "OK, with regards to diagnosis, where does that go? Where does it live? Who handles it other than you?" And your response to them should responsibly say those things. Simply saying, like, "Yes, HIPAA-compliant: Checkbox Achieved" doesn't give them the information they need to do their jobs and make the appropriate certifications to the federal government, so that they do not get sent to prison for willful noncompliance.

And we'll talk a bit about the observation period. The observation period is: OK, you identify in advance what your risks are, what your controls are, and then for the period of that observation period, we're going to be watching you closely as your trusted, independent, non-biased evaluators. And we're going to see whether these controls actually perform the function. So in Delve's marketing communications, they identified that typically they would want a three-month prospective window for an observation period, which sounds a little bit aggressive, usually in the industry it's six to 12 months, but OK, we'll roll with it, a three-month observation period. The piece seems to allege that they were doing retrospective observation periods where, OK, "You signed on the dotted line with us, and we will forward you to the auditor, and then the auditor will ask questions about what happened in the past three months."

That's a crazy design. This is a trust-but-verify exercise, and the verification is important. The fundamental conceit of an observation period is we know that there are some actors in the world that are capable of saying the right things but either choose not to do the right things, or more likely simply have competence issues. And part of the purpose of the auditor is to catch competence issues and be able to give an opinion very confidently on whether this firm is sufficiently competent to do a very simple thing, get the laptop back every time, or be able to explain why it couldn't happen, and actually check that they got the laptop back. If you simply say, "In the last three months, yeah, I got all the laptops back," then there's no meaningful check there.

The firm that is being audited already knows all of the facts that they'll be audited from. The logs aren't trustworthy because they might be backdated or created with the specific design of passing the audit, et cetera, et cetera. The important part about the examination window is we mutually agree that there is a set of procedures that will be done in the future, and then you are going to be adversarially tested on that. And this, by the way, is one of the many pieces of disconfirmatory evidence that would be abundantly available in the universe where these were real audits. There's a back-and-forth between the auditor and the auditee. Things go wrong in the real world, and there's a paper trail of things going wrong.

The journalism says that Delve made verbal promises that no one ever fails an audit. Oh boy, that is an alarming statement if you've ever been audited for anything. Even if it's true, even if the selection effect here is like, "Only the best of the best, most compliance-oriented people choose to buy their compliance reports for $6,000 from us," you would never say out loud, or you would strongly discourage people from saying out loud, "Nobody ever fails," because the possibility of failure is really important and load-bearing.

And if this business were operating as described, you'd be able to point back to paper trails here where, "No, these are real audits. The average audit generates 32 back-and-forth emails between the auditor and the auditee with regards to laptops that went missing. This person passed away suddenly. We couldn't figure out where their laptop was. But we assess minimal risk here because X and Y and Z compensating controls." And the auditor documents that fact in the write-up. It says, you know, "Of the 117 laptops we expected to be recovered, there were two that were not recovered. We've received information from the firm with respect to those two laptops and do not believe there's a material risk of compromise with respect to them, given the totality of the controls environment." That would be the kind of thing you would expect to see in a real audit report. And you'd have voluminous real audit reports where you could say, "This sort of back-and-forth actually happened."

[Patrick notes: I once accidentally gave a Goldman Sachs alum cause to write, in a deliverable to his client, that the $24.95 misstatement of revenue in my business records was not material (and presumptively not a reason to back away from the commitment to purchase my business). This was on a business sold for the price of a nice car.] 

Heuristics for truth

Why do I find these claims credible? I'll say I haven't independently evaluated them, but the claims are the claims. And there are certain heuristics that one uses as someone who reads a lot on the internet, this is likely to be true versus this is likely to be nonsense. So one heuristic that a lot of people use is: was an AI involved in writing this? And it appears to me that there are several tells within the document that an AI was definitely used in assisting in writing the document. I find AI to be a wonderful tool. If you had simply given an AI the task "write a takedown of one of my competitors," it can't produce all the evidence that is in the document, which is externally coherent. And it can't easily produce, at least not in the current state of the art, the level of coherence between the domain expertise brought by the author and the ways in which something would be noncompliant if it were being run in a ridiculously noncompliant fashion.

They allege the existence of actual reports that are in a spreadsheet, and they allege material facts about those, their level of similarity, that they're completely templated, that there are typos, et cetera, et cetera. If these aren't true, it's relatively easy to say that they are not true. In a court of law you need a positive showing of evidence to establish this. In the court of the Internet, if you allege facts that are either true or they're not true, and the other side resorts to character assassination rather than explicitly saying that the things they are claiming are facts are not actually facts, my heuristic is that those claims are likely to be the honest-to-God truth.

A sufficient number of those claims being honest-to-God truth is dispositive here. And there's always a bit of a "what's in it for you" question: are your financial incentives dictating the truth or falsity of the statements you're making, which is brought up as one method of character assassination. I find it incredibly plausible that a competitor might somehow be involved in either the production of this journalism, or the dissemination of it on social media or on the Hacker News thread or similar. And I don't care. The claims are the claims. 

In the maximally formal New York Times investigative journalism, yeah, sources always have an agenda. That's how they come to be sources. They're leaking against the interest of someone, in the other political party, or in their department, so that they can get a promotion, so that they can clear their conscience, whatever. They've got an agenda, sure. But the journalism is ultimately responsible for making claims about the universe we live in, and those claims are true or they are false.

There are many claims that are advanced in this bit of journalism and they have the ring of truth to me. And a sufficient number of those claims being actually true, I think, is dispositive with regards to questions like: is Delve committing scaled acts of fraud? And on the evidence before us, I actually do believe that Delve is committing scaled acts of fraud, although the responsible professional in me would say, again, everyone is entitled to a trial by their peers, et cetera, et cetera, before being convicted of an offense. Unfortunately, somebody might actually find themselves in front of a jury of their peers.

Naughtiness vs. fraud

I want to talk a little bit about startup culture because it sometimes gets invoked in discussions like this, and often by critics of startup culture who think that we're all "moving fast and breaking things." So there is a real celebration in some circles, YC, through PG's writings, et cetera, of naughtiness. 

And naughtiness is something of a term of art. You know it when you see it. And basically, the notion is that there are, like, two kinds of rules in the world. There's the rule that is an actual moral law that establishes some sort of minimum threshold for behavior, that it is really important that we all follow. And then there's the other kind of rule.

And naughtiness is, with respect to the other kind of rule, you know, do what you can to get around it, maybe like doing the pantomime of complying with it, maybe complying in the least-brain-sweat-required fashion, maybe saying something which is literally true but maybe skirts around the intent of the rule, et cetera, et cetera. And you don't do those for, quote-unquote, the "real rules." You don't defraud your investors. You don't harass your employees. You don't make a product which is unsafe to use in a fashion where the lack of safety would actually be load-bearing.

And so people in startups will say, "When we say 'move fast and break things', which is something that even Facebook doesn't say anymore, we're talking about, like, sharing cat photos rather than airplanes." That kind of work operates under a very different set of expectations, and everyone knows this. And we can be contextually aware of these things, and understand there are gray areas. And I think I'm a person with a reasonable level of contextual awareness, and I understand there are gray areas, and I understand that there are some areas of compliance which are theater. And I think everyone involved in compliance understands that it's theater in some fashion.

And sometimes that theater is designed to require a performance from the people who are engaged in it. So you can think that it is theater. You can whisper that among yourselves. You can joke about it. But you have to perform ingratiating compliance. It's a performance of class, and certainly you need to make the written record align with the ingratiating performance. But you also have to make any observable truth about the unwritten record sufficiently ingratiating. So if you're in the financial industry, and you're in your mandatory compliance thing that everyone gets told they must attend, regardless of whether they're the CFO or a teller, and you read a very scary list of precedents of large fines and criminal sanctions and similar from your compliance officers, and someone cracks a joke, the trainer needs to stop and say something very serious like, "This is not a joke. This firm takes compliance very seriously. We always take compliance very seriously here. And if you treat it as a joke, you could be sanctioned individually. We could be sanctioned institutionally. The most likely outcome is very large fines, but they could do anything to us, up to and including closing this firm." And indeed, that sometimes happens.

And indeed, the performance of ingratiation in compliance does get invoked in actual enforcement actions. For example, take a look at Binance, where Binance's chief compliance officer made any number of very outlandish statements in writing, in what he probably considered water cooler talk. "If you're trying to launder money, come to Binance. We've got cake for you." That isn't an exact quote, but it is an extremely close paraphrase. And yeah, if we've got cake for you, DOJ has an indictment for you later. Although subsequently forgiven, don't ask me why.

[Patrick notes: The exact quote, from DOJ court documents, is: "We need a banner 'is washing drug money too hard these days - come to binance we got cake for you.'" This was written by a compliance employee, not the Chief Compliance Officer as I said in the recording; though the CCO, Samuel Lim, separately paid a $1.5M fine in the same resolution. Samuel Lim’s immortality in future Compliance training sessions was secured by, quote, we are operating as a fking unlicensed securities exchange in the USA bro, endquote.]

So as someone who is a reasonably calibrated professional in tech, yeah, I understand seeing some cowboy behavior in early-stage startups. And your MVP has some rough edges on it. You will not have the same level of internal expertise or gating with respect to security procedures in week six as you will in year six. Sure, understand all of that. But you can't lie to people in ways that induce them to transact with you. That's fraud. That is the definition of fraud. That is the moral center of why fraud is bad. And this particular kind spends people's lives, reputations, and honor without their consent.

We talked a little bit about how the compliance officer is putting themselves intentionally on the hook for decisions which might be diffused within their organization. This is a designed part of these compliance regimes. Like, I want an actual human whose neck to wring if this goes off the rails. The detail about board resolutions and faking board minutes is somewhat alarming to me because the board are offering up their necks to wring as part of these compliance regimes. The buck stops with them. And we as a society have decided it should be the board members here, because, again, the board members are relatively socially established. They know what they're signing up for. They have, hopefully, some level of incentive alignment with the company. If the company does well, they do well. And if the company doesn't do well, they've probably lost money on their investment, or at the very least, lost a kind of cushy gig as a board member where you show up to one meeting every quarter and get paid $250,000 a year.

They're designed to have a level of probity and acumen, and a second set of eyes that is independent from management in some fashion, sometimes observed in the breach. [Patrick notes: It is the routine practice in the tech industry to have many some board members who are either also senior management or known to be so socially close to senior management as to not be materially independent checks on them. This is the sort of thing that gives corporate governance professors conniption fits. Myself, eh, I largely think that if a unicorn is rotten the board will learn about that fact from the indictment, no matter how many of the nation’s leading lights you stack it with. Kissinger didn’t kill Theranos; Carreyrou did.]

So if you're signing your board members up for minutes of meetings that did not happen, you're potentially signing them up for legal liability, among other things, and certainly social liability. Board members are repeated players in the game that is being a board member. Almost nobody is a board member of only one thing for their entire career. Certainly you're impacting their ability to be a board member in the future, because future boards are going to look at them and say, "When this person says that they have done the reading, frequently that they have not actually seen the documents that are signed with their name.” That's disqualifying for this social class for this job.

So it is a bad thing to invoke your board members' attestation in policies which don't actually exist, which do not describe the actual reality of your company. You would be very shaken to learn that that had happened, either as a responsible individual inside a company who wants your board members to sign up for things, or, God forbid, as the board member who learns on Tuesday that you have signed minutes for a meeting that never happened.

A voice in the startup community

So, obviously I'm not speaking for YC here, but I have an immense regard for that organization. I have had a long relationship with them, even though I'm no longer the number two user on Hacker News. I hope to have a long and fulfilling relationship with them in the future. I have an immense regard for YC as a community, as a leading light within the startup ecosystem, and as a community of founders. And most of those founders are very ethical, upright, moral business people. And it is alarming when someone would sort of misspend the treasury of merit of that community by doing fraudulent acts under the cover of their organization.

Both because it tends to infect the other members of the community. Startups sell to startups, film at 11. But also because the… treasury of merit is actually a bit of a religious term, apologies for that one.

[Patrick notes: Treasury of merit is a relatively obscure Catholic doctrine that the ontological and works-based goodness of Christ, the saints, and similar creates an asset on Heaven’s balance sheet which is so enormous as to be capable of redeeming all sins of all past, present, and future humans, even as steeped in iniquity as we are.

This was not precisely what I was aiming for when searching for the concept of a communal reputational resource, but one goes into the recording booth with the brain one has.] 

But the reputation of a community is something like the shared property of all members of it and the shared responsibility of everyone for it. And for people who are elders in a community, and you can identify the elders in many communities of choice, I don't know if I'm a senior decision maker in startups, but I have a voice. Here's my voice saying things.

It's important to take corrective actions on behalf of the community, to say, like, "There are lines here. There are some values at play. When those values are contravened, we identify the contravention of values as being something which is against our interests and against our moral principles, even if it was conducted by people who have broken bread with us, who we liked, et cetera, et cetera. 'No, not like that in the future. Never like that.'"

And so to the limited extent I have a voice on the internet, I am pointing here and saying, "Nope. Never like this."

Advice for the exposed

So, there are some people out here in the wide internet who are reading with a bit of bated breath what has happened here, because they are directly exposed. And so you might reasonably want some advice. I think you should get some advice. And you should get some advice not from someone who works for the internet, but someone who works for you. Call your lawyer. You might have exposure. You want to listen very carefully to what your lawyer says.

But I think I can say something useful.

”I am not a lawyer. I'm not capable of giving professional advice” is often used as a bit of a dodge by people, particularly in public. And so I'll say: I'm not a lawyer, but I have been a HIPAA compliance officer. And I have some amount of expertise as to what one should do in a circumstance which rhymes with this one. And so if, hypothetically, as a compliance officer, as a responsible professional, I had become aware that I had given counterparties a document where that document included fraudulent representations, here are a couple of things I would do.

One, I would stop using that document immediately. Two, I would attempt to assemble a list of all external counterparties who had been given that document and might have made their own choices on the basis of representations that I now believe to be fraudulent. I would kick off an immediate investigation internally of who made the decisions here. What did they know? When did they know it? And at what point did we discover that these were fraudulent? And I would prepare for a difficult but necessary conversation with counterparties. 

I always think that we owe the first obligation to the team, even in preference to owing obligations to customers, to shareholders, and similar. There are some members of the team that might have personal exposure here. I'm thinking, in particular, if you are making representations that the board members had a meeting and voted on something, and the board members did not have that meeting and vote on something, those board members should get a call immediately.

But anybody else who is in the document, who is described as doing something that they didn't actually do, yep, that person needs to be told. You cannot spend the honor of your employees. That is just a fundamental star in the firmament of my moral reality. And so if it is in great ways, if it is in small ways, if you said someone has gone through training and they have not actually gone through training, you need to come clean to them and try to make amends for that.

I think very obviously at some point in the future, you will need a different vendor and/or a different process for getting your act in order with regards to any particular compliance regime that you might find yourself subject to. That timeline might not align with the timeline for disclosure to customers and other affected parties. And I have an immense amount of empathy for that, for people who find themselves in an emergency where they might be themselves innocent of causing the emergency. 

But this is one of the responsibilities of capitalism. We earn the big bucks because we are responsible for things like this. And so you might not have chosen to be in the emergency. But you did make some choices in life, and you owe the world the consequences of those choices. And so you should start performing your responsibilities with dispatch right now. And again, get some professional advice, have the war room running, and treat this as the kind of emergency it is, and not like, "Well, there is a bit of a checkbox issue, but everyone is Potemkin village in this space. We can simply move forward."

And so I think that concludes my thoughts on the Delve controversy. Well, I will note that one of their VCs, not one of their VCs, the lead of their Series A, disappeared the mention of Delve from their website for a while, and then brought it back after TechCrunch took notice of the fact of them disappearing that. That is a bad fact pattern.

I assume we will learn much more about this as time goes on. And so follow around on the Internet and the obvious places for the updates to it. I do not think that I will likely be one of those obvious places because I mostly don't break news. But I felt this was sufficiently central to the beat, and sufficiently central to the moral realities of capitalism, that I thought I should probably opine on it, or not opining on it would itself be a choice.

And with that, thank you very much for listening. I'll see you next week on Complex Systems.