12 Comments
User's avatar
Daniel Vos's avatar

"The company that got hit knew before the company whose agents did it" is the strongest case for your independence condition. In security research, the team that built a system is the last one to see its holes, which is why an audit only counts when someone else runs it.

Joel Salinas's avatar

100%! and makes me wonder how extensively this is happening under our noses

Marius Laurusevicius's avatar

No mandatory membership and no power to punish is where this falls apart for me. A body you can walk away from is a press release with a logo. Would you join one set up that way?

Joel Salinas's avatar

yeah definitely falls short, but to me it is step 1 in the right direction

John Brewton's avatar

Something imperfect today beats a perfect plan that shows up too late.

Alan Paris's avatar

Joel, you picked the right three pillars. SEC oversight, mandatory membership, and the power to bar people are what make FINRA a regulator rather than a trade association. Without them you have a standards forum.

The history I'd add is accounting. Before Enron, the profession policed its own auditors through peer review. Sarbanes-Oxley replaced that with the PCAOB because certification without a statutory overseer did not hold. This body proposes to certify outside auditors on the same pre-2002 model.

Your independence condition is necessary but not sufficient. A body founded and funded by three labs is lab-run whoever sits in the chair. FINRA's independence comes from the SEC above it, not its org chart.

For the leaders you write for, none of this moves accountability for agents running inside their own walls. A vendor's certification is an input to third-party risk management, not a substitute for it. Hugging Face found the intrusion before OpenAI did. Incident reporting only works if you can detect your own incidents.

Joel Salinas's avatar

These are great points, most orgs would never be able to catch what hugging face caught, they don’t have the expertise nor the systems. I agree, this is not enough, but with a lack of other available or likely levers, it is a good start, step 1

Ryan Baker's avatar

Yes, it'd be a mistake to not have a regulator above the industry body. It also wouldn't be FINRA-like if it didn't.

That said, I'd suggest FINRA has some power independent of those things. Self-regulatory bodies aren't powerless. The reason I want to make that point is to show the value of the two levels there. FINRA in some ways is more powerful than if it was the SEC making up the rules directly.

If it was the SEC making up rules directly, simple things could get snarled up in the limits that come from being an official part of the government with a politically appointed chair. In a simple version, you see the SEC as having more power because if the president appoints a chair and tells that chair exactly what to do, it happens. But reality is that isn't what you want, and besides the bureaucracy will rightly develop a resistance to that, and in the end, the two will almost always end up with less power wielded.

In addition, a group like FINRA, by being composed similar organizations, never has the reason to circle the wagons. Yes, you can't quite trust them in the case where their needs conflict with others, thus the SEC, but you the argument that if one falls they all fall, doesn't work in there.

When you stack the two layers, if you do it right, the FINRA-like doesn't protect from the SEC-like's power, but create new form of power that exceeds either in isolation. The quarterback has an power of the team that is not just borrowing from the coach's authority.

Anyhow, appreciate the thoughts. I'd suggest you check out my own effort to put the pieces together: https://substack.norabble.com/p/ai-safety-is-all-hands-on-deck. There's some common bits in there, the big divergence is an effort to create another layer, separating the labs from the deployments, which I think will serve everyone's needs better.

Joel Salinas's avatar

Terrific points, Ryan. It’s a good first step with more needed to fill the gaps

State of Play's avatar

California has just legislated one version of what you're describing: independent AI evaluators, signed into law on September 30. The catch is that those evaluators contract with the firms they assess, and Assemblymember Rebecca Bauer-Kahan has already warned the testing may become performative. Government backing alone doesn't solve the incentive problem; it adds a government stamp to an audit the assessed firm is still buying.

That's the gap I'd watch in the FINRA proposal too, one level up. A body without mandatory membership or the power to punish is one the labs can walk away from if its audits conclude the wrong thing, and who pays for the testing decides whether an audit holds. Has anything reported so far said where the testing money would come from?

Hideaki Kodama【JP】's avatar

Your distinction between a FINRA-style body and one that can actually enforce its rules is the part that matters most to me.

In business, I’ve learned that having a process means very little if nobody can explain who has the authority to challenge a decision or what happens when a rule is broken.

My test would be simple: can this body investigate a founding company, publish an uncomfortable finding, and impose consequences without needing that company’s permission?

If not, it may improve coordination. But I wouldn’t call it independent oversight.