The AI safety conversation got very loud over the past couple of weeks. It’s dominated the news cycle and is only increasing. Why?
A senior researcher at an AI lab resigned publicly over fears about where the models are heading. Lawmakers started floating “kill-switch” requirements for the foundational models. Labs published essays arguing the pace of frontier development needs to be managed. Boards are being told to establish AI governance frameworks immediately.
Unlike the news commentators who don’t even know how to pronounce nVidia, my first reaction wasn’t fear. It was suspicion about the timing. This was my working theory.
The companies calling loudest for frontier AI regulation are the companies that have already won. They have the compute, the capital, the distribution, and the best models they built on ignoring copyright and taking advantage of a “wild west” approach to AI building.
My theory is that regulation with heavy compliance costs doesn’t actually threaten them. It threatens the next company wanting to break into the conversation. I believe the specific thing it threatens most is open-weight models, which are free, downloadable, and structurally different than big AI.
Meanwhile, these same companies are heading toward the largest public offerings in market history. Talking up how powerful and dangerous your technology happens to be is an excellent way to communicate that it’s so valuable and dominant that you can’t not invest in them.
Is the safety conversation sincere, or is it self-serving? I went looking for evidence against my own theory and what I discovered was my skepticism is both well founded… but also wrong.
Where my skepticism holds up
The timing is suspicious. I’m not the only one saying this.
Venture capitalist Bill Gurley wrote that he predicted years ago open models would become the key threat to AI incumbents, and that they would respond with regulatory capture. He has since said that he underestimated how aggressively they’d pursue it. When the White House’s own AI adviser publicly accuses industry figures of running a regulatory capture strategy, you should pay attention.
The sequencing is hard to ignore.
One of the major “let’s pace the frontier” essays came out in the same week a Chinese lab released an open-weight model that observers rated alongside the best available frontier models, at a fraction of the cost. Is that a coincidence? Maybe. If so, that’s quite a coincidence.
The structure of proposed rules favors closed systems.
Requirements around account monitoring, live patching, and the ability to withdraw a model from service are trivial if you serve everything through an API. They are nearly impossible if your model has already been downloaded thousands of times. You don’t have to ban open weights outright. You just have to write rules that assume a closed architecture.
Independent analysts have made this exact argument.
Frontier safety regulation can be sincerely motivated and still create an incumbent-protecting moat keeping out new solutions. The economic effect should be measured separately from the claim about motive.
Where I Got it Wrong
Problem one: They’re also spending enormous money to stop regulation.
If the strategy were simply “demand rules that lock out competitors,” you’d expect these companies to be spending millions on pro-regulation legislation. I was surprised to find that’s not what’s happening. AI-aligned super PACs have raised north of $200 million heading into the midterms, and most of that is aimed at killing state AI laws.
Notice I said state laws. I believe they prefer to have a federal standard instead of 50 divergent state laws and thousands of municipal regulations. A single federal standard you helped write is a better regulation because you set one standard suited best for you.
So, while people will point towards AI companies lobbying for regulation, even this may be interpreted as self-serving.
Problem two: The open-weight threat may be overstated as a revenue threat.
My research on this surprised me. A 2026 study showed open-weight models running roughly 29% of tokens but under 4% of spend, while four leading U.S. labs captured about 95% of gateway spend.
Let me explain this. Open models are being used a lot and paid for very little. If they aren’t yet eating the profit pool, my theory of “they’re doing this to protect revenue” is weak (at least for now). I still believe that the big four see open weight models as a threat to their long-term viability. The capability gap is closing fast as AI spend is growing exponentially.
Problem three: The IPO theory cuts both ways.
My assumption is that danger talk inflates valuation. Create a powerful enough fear and it will translate into a valuation of a trillion dollars.
This has an opposite impact as well. A senior AI engineer resigning to say your product might end humanity is not something that will sell shares. Cyber insurers are already beginning to write exclusions for AI-related incidents. Regulatory pressures should, in normal markets, have a negative impact on pricing, not raise it. I still think the hype helps their bottom line more than the fear hurts, but it’s a dangerous strategy and yet valuations keep climbing.
My Take
1. Sincerity and self-interest are not mutually exclusive.
The most common failure I see in this debate is treating it as a question of motive. Do these AI leaders really believe it? I have no idea and the more I reflect on this, it doesn’t really matter. A person can genuinely fear a risk and also notice that addressing it would benefit them.
2. Judge the effects, not the AI leader’s rhetoric.
Ask the question of what a proposed rule actually does to the market structure. Does compliance scale with company size? Would a well-funded startup be able to clear the bar? Does it assume a closed architecture? Focus on these questions instead of whether the person proposing the rule is trustworthy.
3. The concentration risk is real and not many people are talking about it.
Whatever the motive, any outcome where a handful of companies (or one) control frontier AI is genuinely bad. It’s bad for competition, for pricing, and for anyone whose mission depends on affordable access to these tools. This risk deserves as much conversation and attention as the “catastrophe scenarios.” Unfortunately, human extinction discussion is deflecting the concentration of power conversation.
4. The near-term risks look more mundane than the apocalyptic ones.
The realistic near-term dangers aren’t “sentient AI adversaries." Today’s challenges focus on autonomous agents with too many credentials and not enough circuit breakers. This allows for accelerating attacks that already exist. These are boring, operational, and much more likely to affect your organization than anything in a science fiction plot. Focusing on the sci-fi scenarios is more comfortable than confronting today’s real risks.
An outcome where a handful of companies control frontier AI is genuinely bad.
What does this mean for us?
This isn’t an abstract AI policy debate. It’s a question about the cost and availability of the tools we’ll depend on for the next decade.
If frontier capability consolidates behind a few AI labs, we become price takers for infrastructure we increasingly can’t operate without. They have been selling us AI for a tenth of the actual cost so they can capture market share. Open models are one of the reasons a small organization can run capable AI on its own hardware without a seven-figure annual contract.
When I read the safety debate, I’m not just asking whether the risks are real. I’m asking who ends up holding the keys. Stewardship of what we’ve been entrusted with includes not signing away our independence to whoever writes the rules first.
I don’t think the people raising alarms are lying. I think they’re telling the truth about the risk and, at the same time, proposing solutions that happen to be very good for themselves.
Both things can be true.
When someone warns you about a danger that only they can protect you from, how do you tell the difference between their warning and their sales pitch?