Dear Magicians,

Recently, I heard reports that a researcher the same age as many of my grad students had spent four months at Anthropic, which admittedly is longer than most of my diets have lasted, resigned before his equity vested, and told the world that frontier AI labs are “gambling with our lives.” Jacob Coxon’s resignation goes viral, seen more than 100 million times, as frontier-lab leaders start talking about slowing down, and Barack Obama weighs in.

The honest answer is that I don’t know where this ends. I’m a cosmologist, which means I’ve spent much of my adult life reconstructing the history of the universe from ancient microwave photons, so naturally I am now expected to understand Silicon Valley.

What I do understand is the incentive problem. A small number of companies can capture astonishing economic upside from increasingly capable AI while some portion of the downside, whether job displacement, cybercrime, autonomous weapons or something genuinely catastrophic, lands on everyone else. That does not prove the doomsayers are right. It does mean the people placing the bet are not the only people with chips on the table.

I recently heard an argument from Professor Scott Galloway, who asks us to think about 1946. Imagine Robert Oppenheimer leaving Los Alamos, raising a venture round, and announcing that nuclear weapons were now available to sovereign wealth funds and select enterprise customers. I assume Procter & Gamble would have received preferred pricing.

We did not organize atomic technology that way. The Atomic Energy Act of 1946 created an extraordinary federal regime around atomic materials and technology. More interestingly, Oppenheimer helped produce the Acheson-Lilienthal Report, which tried to answer a problem that feels strangely modern: how do you preserve the useful applications of a transformative technology while controlling the parts capable of catastrophic harm?

The analogy is imperfect. But Scott’s both right and wrong. Fission gave us weapons capable of destroying cities and also reactors, submarines, medical isotopes and spacecraft power. Until 1954, nuclear technology remained essentially a federal monopoly, placing major legal barriers in front of private nuclear power as well as private weapons development.

Contra Galloway, Imagine if Kleiner Perkins DID have an opportunity to invest in nuclear fission for reactor purposes only. Generating energy might have dramatically reduced the climate change and global warming concerns that followed. We’d have had far more abundant low-carbon energy. We wouldn’t have to wait for fusion, and the arguments over data centers might look very different. Heck, we’d even have data centers, and we’d probably have super-intelligent AI for real right now instead of twerking robots on tik Tok!

Without today’s cost-per-token and energy constraints, if we had actually taken the venture capital route in the 1950s, Oppenheimer could have been a hell of a hedge fund manager. We didn’t.

AI may have the same dual-use problem, except the boundary may be harder to draw. The same model that helps design a protein or analyze telescope data may also improve cyberattacks, surveillance, weapons or biological design. The reactor and the bomb may share too much machinery.

That is why the current slowdown debate is so strange. Dario Amodei has argued for “pacing the frontier.” I decided that if it’s good enough for Dario, it might be good enough to say to my teenager when he asks to borrow the keys and $20. Sorry, buddy, we’ve got to pace the frontier on that. Sam Altman and Elon Musk have also expressed support for slowing frontier development. There is something awkward about being told by the drivers at the front of the race that everyone should now observe the speed limit.

Lately I’ve hosted some of the most prominent voices warning about catastrophic AI risk, from Roman Yampolskiy to Emad Mostaque to Max Tegmark. What strikes me is how crowded that side of the boat has become. There are remarkably few prominent AI “boomers” left making the opposite case with any confidence, Yann LeCun being the obvious exception. When nearly every serious person in a debate migrates to one side, that may mean the evidence has converged. It may also mean the conversation itself has developed a center of gravity strong enough to pull everyone toward it.

That does not make the warnings insincere. A warning can be both sincere and economically convenient. A slowdown might reduce genuine risk, or it might entrench the firms that already have the most compute, capital and talent. Both can be true.

Geoffrey Hinton faced a version of this when he left Google in 2023. He said: “If I hadn’t done it, somebody else would have.” That remark accompanied his decision to speak more freely about AI risk. Later he put the epistemic situation more plainly: “Nobody really knows what’s going to happen … I’m just sounding the alarm.” That interview is here.

That is closer to my position than confident percentages of human extinction. Physicists usually like error bars before the apocalypse, although we have been known to make exceptions when tenure is involved.

The argument I hear less often is about the reactor. In San Diego, we sleep blissfully within the blast radius of 3-6 military nuclear reactors operated by the Navy while the one commercial reactor was shut down years ago: San Onofre’s premature closure eliminated a 2.2 GW nuclear plant that had powered roughly 1.4 million homes, forced Southern California Edison to spend more than $500 million on replacement electricity before the shutdown was even permanent, stranded roughly $768.5 million in replacement steam generators, and triggered years of disputes over billions of dollars in costs borne by utilities and ratepayers.

The cleanup is now expected to cost about $5.73 billion for Units 2 and 3, with roughly $2.77 billion already spent through 2023 and another $2.96 billion projected through 2056, making San Onofre a striking example of how a non-core equipment failure can destroy the economic value of an already-built low-carbon power asset and then require billions more to dismantle it.

What worries me is ignoring venture economics from having a say in who decides. True it’s likely to lead to a system with the largest market getting the largest valuation. But this feels like the real fork in the road. Not whether AI is dangerous. Almost every powerful technology is dangerous. The harder question is which capabilities we actually want, which risks we are willing to socialize, invest in and who gets to make those decisions.

I don’t know whether Jacob Coxon is prescient or catastrophically wrong. I am increasingly suspicious, though, of any theory of technological civilization in which the deciding argument is simply that one branch has the higher valuation.

Until next time, have a M.A.G.I.C. Week.

Brian