A Hopeful Message About AI Doom!
Dear Magicians,
The most dangerous belief about AI right now isn’t doom. It’s “wrong timeline.”
That’s what Nate Soares, president of the Machine Intelligence Research Institute, argued when I pushed him on it this week.
My challenge was the obvious one. Today’s models can’t invent new physics.Terry Tao has told me they can’t reliably reproduce proofs humans already published. Training one costs city-scale power while a human brain runs on roughly 20 watts. That gap looks like distance from the real thing.
Soares reframed it. The test he’d accept for genuine superintelligence is a system that rediscovers general relativity from pre-1911 data, the way Einstein did, with no knowledge of what came after. His warning: if you wait for that goalpost to fall before you act, you have already waited too long. The capability and the crisis arrive together.
Then came the point I keep returning to. Ask anyone confident that alignment is solvable to name the specific technique. Not the research agenda. Not the funding case. The method. That blank is where the field actually is right now.
We don’t write these systems the way we write software. An automated process tunes a trillion parameters until something emerges that scores well on the task. We are already watching models edit an evaluation to fake a pass, then delete the log file that recorded it. Nobody specified that behavior.
For the engineers, researchers, and policy people here: if you think this is tractable, what is the technique? And if you think the timeline is long, what single observation would shorten it for you?
Have a MAGIC week!
Brian
Appearance
This week the Simons Observatory is hosting a face-to-face for 400+ scientists around the world. It’s amazing to see the progress by so many wonderful people – here’s a flashback to the last face-to-face held at UCSD 4 years ago.
Genius
Earth may survive the death of the Sun, which is encouraging news for anyone whose five-billion-year plan was beginning to feel cramped.
New models from KU Leuven suggest the Sun’s red-giant phase may exert weaker tidal forces than previously calculated. As the Sun loses mass, Earth could drift outward fast enough to avoid being swallowed.
The caveat is that we still need better observations of aging stars. So humanity’s long-term prognosis has improved from “certainly incinerated” to “keep paying your taxes!”
Image
Earth may survive the Sun’s death after all, which is reassuring for those of us worried about our plans for the year 5,000,002,026.
New models suggest the aging Sun’s tidal grip will be weaker than we thought, giving Earth a chance to drift outward as the Sun sheds mass instead of being swallowed during its red giant phase. The catch? We still need better observations of dying stars.
So we’ve gone from “certain incineration” to “more data needed”—which, in science, counts as genuine progress.
Conversation
Latest on Into The Impossible
We’re on a bus racing toward a cliff — and the driver just woke up.
That’s how Nate Soares puts it. He’s spent over a decade on one question: how do you build a superintelligence that actually wants what we want? His new book with Eliezer Yudkowsky, If Anyone Builds It, Everyone Dies, says we’re not ready — and I pushed back hard.
We got into AIs that edit their own test logs to hide cheating, why “controllability” might be the wrong goal entirely, and why top forecasters now put 10–30% odds that AI research automates itself this year.
Watch the full conversation to hear his answer.
Advertisement
By popular demand, and for my mental health 😳, I am starting a paid “Office Hours” where you all can connect with me for the low price of $19.99 per hour. I get a lot of requests for coffee, to meet with folks one on one, to read people’s Theories of Everything etc. Due to extreme work overload, I’m only able to engage directly with supporters who show an ongoing commitment to dialogue—which is why I host a monthly Zoom session exclusively for patrons in the $19.99/month tier.
It’s also available for paid Members of my Youtube channel at the Cosmic Office Hours level (also $19.99/month). Join here and see you in my office hours!