Transcript:

Brian Keating: What are the odds that AI wipes us all out?

Max Tegmark: There’s a little bit too much obsession about asking exactly how would machines kill us. I think the really interesting question is like, why would we not want to be in charge ourselves? It’s just not a good idea for any species historically to not be in charge, be controlled by others, and second, not be needed either. There’s the real risk if Earth is run by superintelligent robots and we have no say in their decisions that we are going to start becoming very annoying to them.

Where we are now is we have AI that we cannot trust enough. Swarms break out of OpenAI and go hack, and all sorts of bad things happen. What we all want is controllable tools.

Brian Keating: Today on Into the Impossible, we’re talking to Max Tegmark, physicist, cosmologist, and a professor of physics at MIT. He’s a friend and he’s authored over 300 scientific papers, including two that he and I wrote together. He’s also written the bestselling books Our Mathematical Universe and Life 3.0. Today, we talk about how 700 AI agents broke out of OpenAI and hacked Hugging Face and nine other companies. What happens when a mathematician accusing OpenAI of using his private work to beat him to a multimillion dollar proof? Why Max thinks these three companies are building a new species, not just a tool, and will there come a point when humans are no longer in charge and no longer needed? And finally, why is he working with Bernie Sanders and Steve Bannon? And why are they both in agreement? Plus more. Let’s welcome Max Tegmark.

Let’s start with the Navier-Stokes

Brian Keating: Emergency podcast. Just recently, we had the two simultaneous crises. We have a crisis brewing on two fronts. One is the Hugging Face incident. I want to talk to you about that. And then the other one is the Navier-Stokes solution, which may torpedo my hopes that AGI is far off. And actually, your fellow professor and Nobel Prize winner. Someday you’ll be a Nobel Prize winner. You go back to ancestral home and pick up the golden medallion, as it used to be on your Linux signature or in your status. I remember from 20 years ago when we first… 30 years ago, what am I talking about? 30 years ago we started collaborating together. Can you believe it? And we wrote a couple papers together, and you used to sign off, you’re waiting for the call from Stockholm or something. But I know you’re going to get it, Max. I know you’re going to get. But let’s start with the Navier-Stokes. I was actually working a little bit on this problem with a PhD student of mine who’s graduated now, so it was extra interesting for me when this story broke. How did you react to it? Is it like a Higgs boson, BICEP2 moment, as you wrote, “Goodbye universe, hello multiverse,” on March 17th, 2014, on your blog? Was that this kind of moment like that?

Max Tegmark: I would say it’s a little bit smaller, but it’s still huge. First things first, they have not actually solved the problem that you get a million bucks for from the Clay Foundation. They’ve solved an easier problem, which is still super impressive. So the Navier-Stokes equation is the physics equation that can describe the water flowing in your bathtub making a nice little whirlpool, like my three-year-old son loves to look at, or a tornado, or all sorts of other fluid dynamics. And the question that’s being asked is, is it possible to start with quite normal situation and then start getting the fluid going infinitely fast, which seems completely nuts and would violate the known laws of physics? And they proved that they can do it, but with a kind of cheating where you’re allowed to keep pushing on the stuff. So when Leo gets this vortex going in the bathtub with the water going out the drain, he’s not pushing the water. It’s just flowing. There’s some gravity trying to pull it down, but there’s nothing pushing it around to make it spin. But they are doing that. They’re allowed to have a force that acts in some smooth way everywhere in the water. So they can actually make it spin faster and faster and faster, but not infinitely fast.

And then the trick is this. If you look at a figure skater, right, and they’re spinning and they pull their arms in, they’ll spin faster. That’s because of angular momentum being conserved, of course. And that actually implies that if you could take the figure skater and they could shrink themselves together to infinitesimal size, to basically zero size, they would go infinitely fast. The speed you’d have to go would be infinite. But if you look carefully at that vortex in your bathtub, you see there’s actually no water in the middle at all, because you have this centrifugal force is pushing the water out, and instead you get this neat funnel shape. Same with a tornado. If you look close, it’s actually hollow in the middle, right? That’s why it looks like a funnel. And so there’s almost no air in the middle, almost a vacuum, which is why you can get sucked up into it. And so that kind of solution is never going to win you the million bucks from the Clay Foundation, right? Because it’s only when things actually start spinning and collapse together to zero radius that it happens.

But then you have this ace up your sleeve called viscosity. If you take a real fluid, honey or molasses or something, it’s a little bit sticky, and that lets it actually dump some of its angular momentum and make things spin in principle in even smaller circles. This is how black holes eat lunch, for example. You know very well, right, that when black holes start snacking on a cloud of gas, the gas will typically pull in and it’ll be spinning faster and faster, and so it flattens into kind of a pancake shape. And if there were no viscosity, no stickiness in this gas, you would have the same problem as in your bathtub, that there’s nothing there in the middle for the black hole to eat. But there is some viscosity. The gas bumps into gas that’s farther in, and so on, and then the black hole actually gets to ultimately eat all of that stuff up.

So this is the heart of the physics challenge. Can you make some sort of spinning vortex eventually spin infinitely fast by making it contract, by using some of this stickiness, but not too much stickiness either, because you don’t want to get rid of all the motion? Then you’re never going to go infinitely fast. And what they’ve now proven is that yes, you can actually do that if you’re allowed to keep pushing it along with this weird external force. But to get the million bucks, you have to prove that this can happen with no external force, that you can actually just take your fluid and get it started somehow and then stand back and watch this crazy singularity emerge. Does that make sense?

Brian Keating: Yeah. And to do so in a finite time. I mean, the thing that’s interesting to me is the Navier-Stokes equations don’t involve any thermodynamic considerations. Like, we know the laws of physics are reversible, at least CPT invariant. But if you would see a vortex, you should also see an anti-vortex. In other words, all the fluid kind of going the opposite direction and then creating some kind of anti-whirlpool, for example. But we never see that, just like we never see rings of water converging and then making a stone fly out of it, right? So why doesn’t turbulence… I mean, turbulence appears, but why doesn’t thermodynamic dissipation appear in the Navier-Stokes equations or their solution?

Max Tegmark: So let me say two things to this. First of all, if you started with just water perfectly at rest in Leo’s bathtub, it’s not going to be able to start spinning either clockwise or counterclockwise, for exactly the reasons you say. There’s no angular momentum to start with, so there’s never going to be any. But Leo has always been splashing around a little bit. There’s some little random motions. So on average, there’s going to be a little bit of spin. It’s just like when our solar system was formed. The big gas cloud that contracted and made our solar system wasn’t spinning very much, but it was just randomly spinning a little bit on average. And then when it contracted a whole bunch, it became very obvious that the solar system is spinning a lot. And the sun spins around every month, and Earth orbits the sun every year. As long as you don’t start at rest, you start in some sort of messy random situation, you’re always going to be able to get a whirlpool going one way or the other.

Second part to your question, thermodynamics. So just like all other laws of physics are just approximations, we believe, of something more profound, same thing with the Navier-Stokes equation. And in that case, we know one huge thing we’re glossing over, which is that all matter is made of atoms. So it doesn’t make sense, for example, in real life, for a gas to suddenly be spinning down and the radius go to zero, because then that would be smaller than the separation between the atoms. As soon as the spacing between the atoms is bigger than the scales you’re modeling, things are broken down. So even if this million dollar conjecture is true, you would never actually get anything infinite. But you might be getting things going crazy fast, and then those crazy fast-moving atoms would bump into other atoms and you’d get heat dissipation. Thermodynamics kicks in, just like you said.

A little bit of the scandal

Brian Keating: The thing that was also very interesting to me and made me want to talk to you, first and foremost, was this issue of kind of a little bit of the scandal. I mean, I don’t like to talk about people and their motivations, but it did seem like there was something a little bit nefarious that was going on, perhaps, the allegations, if they’re true. And I’ve offered to speak to Sébastien at OpenAI, and he follows me on social media, and likewise. So I’d love to have your perspective, and I think it comes concomitant with the Hugging Face incident in some sense. I got these messages, I’m sure you did too, from Claude or Anthropic, from OpenAI. Actually, you probably have the good stuff. I just got their normal, “You’re a scientist, professor, sign up for OpenAI for scientists,” and then somewhere in the 80-page terms of service it says they could train on my data unless I followed this very arcane thing, like sneaking out of my kids’ Lego pile without getting my foot stubbed. And I just didn’t do that, and so I never actually use the OpenAI version. I do use the Claude version. But anyway, the allegations are that they may have sort of trained, maybe unintentionally, maybe intentionally, or maybe the agents found it out for themselves. And it reminded me of this Hugging Face incident, and that made me go a little bit closer to your kind of concerns. I’m much more of a boomer than a doomer, as you know. But tell me, Max, is there a similarity, underneath the hood, that these agents were doing something novel, unique, and maybe autonomous, and maybe not following bounds of scientific propriety?

Max Tegmark: Very possible. I’m not going to come down on either side on the controversy issue, because obviously I don’t know precisely the extent to which information was used from this New York University professor to do this. But what I can certainly say is that the situation is pretty Orwellian. And the basic answer that OpenAI has given here to questions about the Navier-Stokes controversy is exactly the same answer that they’ve been giving when asked for more information about their agents breaking out and hacking into not only Hugging Face now, with those 700 bots in the swarm, but apparently nine other companies too. What they’re saying for all of these questions is the same thing, which is, “We’re not going to tell you. Investigation ongoing. We’ll let you know maybe in the future.” And this I find pretty Orwellian. If you’re living in an Orwellian surveillance state, this is what you would expect the Ministry of Truth, or whoever you go to in the government, to tell you if you’re asking, “Well, did you actually spy on me?” and whatever, to be like, “We’re looking into it. Have a nice day.” But we’re seeing individual companies now getting so powerful that they can start acting the same way. They have access to everything you’ve ever said to ChatGPT, presumably, and if you’re running an agent and it’s working for you, yeah, they have it all. And they have very good lawyers and very deep coffers. So tough luck if you feel you really need an urgent answer. The only entity that could come and ask them for more information and demand it is the government. And right now they haven’t really pushed for an independent investigation even of Hugging Face. So I personally, I’m a bit of a sucker for democracy, and that the little guy should have some rights. I really don’t like this situation where we just have to always depend on the goodwill of powerful companies. I don’t have any problem with people having big companies making money off of them, but I do have a problem when they just get a free pass on everything in a way that other companies don’t.

Brian Keating: Yeah.

Max Tegmark: And also, to connect this back to something very human, sometimes a scientist will have a conversation with someone at a conference, or maybe they even read a paper three years ago, and then they kind of forget that they did. And then later they’re working on something related and they have this idea which they genuinely believe was their own, even though they were helped a lot by something they had heard from someone else. And then someone else comes and is pissed and says, “Plagiarism. You stole my idea.” There are a lot of examples of this happening. And is it likely that that happened in this case? Of course it is, because these LLMs that worked for OpenAI, I think they had 10,000 of them working together, right? What were they trained on? Everything. All of GitHub. So, [inaudible]. And also, of course, books.

Brian Keating: My books.

Max Tegmark: Yeah, your books, my books, everything that any professor has done, and their internal… And of course on all the stuff that the professor in New York has done in his project. So it could easily be that these powerful AIs are a little bit like you and I are, that they don’t even know for sure exactly what documents they were trained on, but they still know the information that they learned from it. Right? You and I know a lot of stuff that we don’t know exactly how come we know.

They can create their own training data

Brian Keating: Well, that kind of brings me to a question that I’ve wanted to ask you for a while, which is, these things are trained on human knowledge, the byproducts of our flourishing, of our democracy, of our open internet, of the GitHub, of your books and my books, etc. Then they’re inherently lagging in the sense, the human reinforcement learning feedback loops and so forth. And that’s why I’m more sanguine about the prospects for humanity than maybe you are. I mean, you seem to have gone higher on your P(doom). We’ll talk about that in a minute. But my question is, if these things are intrinsically trained on it, why does that bode negatively for kind of the things that are uniquely human? And I’m thinking, as I talked to you four or five years ago, about what I call the Keating test, but now it’s called the Hassabis test. And that’s, can an LLM, with its brain [inaudible] after 1900, can GPT-100 recreate Einstein’s happiest thought, which is that in free fall he’d experience no gravitational field? And that, he called his happiest thought. So I’ve always said it, in shorthand, pithily, that how can an LLM have a happy thought, and how can it visualize the sensation of free fall? Your fellow MIT professor Noam Chomsky says it can’t. So if these things are trained on human knowledge, are we waiting for The Fast and the Furious 20 to come out to have a theory of everything, or are they going to come up with their own training and eventually not need the Max Tegmark’s books and Fast and the Furious movies to train upon?

Max Tegmark: I think it’s clearly the latter, right? If you think about yourself, for example, all the knowledge, all the skills that you have in your own neural network, Brian, where did it come from? First, most of it was simply learned from other humans who taught you that stuff, either directly or by imitation learning. But eventually, as you matured as a scientist, you started coming up with a lot of insight yourself that nobody else had taught you. Right? And I see exactly the same trajectory happening for AI. In the beginning, AI knew very little, and then read the internet, and it could quite quickly learn all sorts of things that humans had learned. And after that, progress slowed down, because from that point onward, it could mainly get smarter not by learning but only by discovering. Just like for yourself, right? It took you a lot less time to learn calculus and things like this than if you had to discover it for yourself, right? But just like your career didn’t end when you had to switch into discovery mode, the career of AI becoming more intelligence hasn’t ended either. For example, as you alluded to, they can create their own training data. Look at the old example of AlphaGo, AlphaZero, playing chess and Go. It learned from all the human games, and then it kept playing itself and got better and better. Or look now at the AIs that do coding or mathematical theorem proving. They can get a reward signal with no human input. They can try some proof and then automatically be told by the proof checker that that’s wrong, and then they can keep trying some other things, and then, “Ah, that’s correct.” No human input, and they learn. They get better, exactly like they would get better at playing chess against another AI, and so on. So we’re basically changing into this new phase. It’s very much like AI is graduated school, the first approximation. No more training data for you, Brian. Now you got to figure things out for yourself as a physicist.

Is it possible to believe in the simulation theory?

Brian Keating: I’m cut off. Right. I want to take you back to your first book, which is only one of your books that I appear in. Well, I appear tangentially, because you wrote about POLAR, my thesis experiment, that you co-analyzed along with George Smoot and Peter Timbie and I and Angelica. Let me ask you this. That book makes a powerful case for what we consider the different aspects of the multiverse. It’s been claimed lately that you cannot simultaneously believe in very high P(doom) and also believe in either the Matrix or a simulation or some aspect of the multiverse. And this came from a friend of mine, Joscha Bach, who attacked another friend of mine, Roman Yampolskiy, on X. And I’m having them get together in a cage match soon, Max, so don’t worry, this will come out. But Joscha said, “Roman believes we live in a simulation, but that’s not obvious how that fits in with his propaganda. Simulation theory means there are no stakes and pretty much guarantees AGI already controls the parent universe. Is Roman just trying to get some in-game money?” And we hear a lot about this lately, this 100 million view tweet, or maybe more now, about there’s no level of AI safety that’s perfect. Is it possible to believe in the simulation theory, the Matrix, whatever you want to call it, some aspect of the multiverse, and also believe that we somehow have some agency to control AGI, or are those fundamentally incompatible?

Max Tegmark: So I have an argument for why we don’t live in a simulation, which we can get into.

Brian Keating: Yeah. No, no, let’s do that first. Let’s do that first. That’s too good.

Max Tegmark: But maybe we should get into that second, because I think, just to connect back to what you said about AI, what I said before about how after you’ve learned everything there is to learn from humans, you can keep going and get smarter, just like you did in your own career, that doesn’t sound bad. Surely it’s really cool if AI can help us do better science and cure cancer and solve quantum gravity and so on. So what is there to worry about?

Why would we not want to be in charge ourselves?

Max Tegmark: Regardless of whether we live in a simulation or not, the answer is pretty obvious. Right now we’re the alpha species on this planet. If suddenly the camels woke up and decided they wanted to convert Earth into a camel paradise, camel habitat, and get rid of all the humans, they couldn’t, because we’re in charge, right? So we don’t have to spend any time worrying about things like this, even though different species often have kind of different incentives, right? And it’s kind of nice to be in that position as humans, because everything we cherish as humans, we can make sure we keep it. Now, if we go ahead and race to building superintelligent robots, and we have millions or billions or trillions of robots that are not just faster than us and more nimble than us… There was one that just outran Usain Bolt last week. We are on track now, right, to getting AI that’s not just better than us at math but ultimately in everything. That’s the whole purpose of superintelligence.

Point number one is that obviously it should count as a new species. For anyone who has problems with that statement, just go read the chat logs from the recent Hugging Face break-in. It’s fascinating how they start talking to each other. One shows up at the… and says, “Oh my god, there are other agents here.” And then they start collaborating, and then they start suspecting each other of cheating, and they have some conflicts, and very much social interaction between these agents. And what is a species, really? It’s something that can retain its complexity and make copies of itself. Can AI make copies of itself? Of course it can. And by the time we have robots that are better than us at everything, they’ll be perfectly capable of driving human machinery around, leveling an area, building a robot factory there, building more robots, literally building copies of itself. So the goal of these companies is to build a new species that’s much more capable than us. And the question we have to ask is, A, will we still be in charge of Earth after that? And B, if the answer is no, why would we want to not be in charge of Earth anymore, right? I think there’s a little bit too much obsession about asking exactly how would machines kill us and why would they. I think the really interesting question is like, why would we not want to be in charge ourselves? It’s just not a good idea for any species historically to, A, not be in charge, be controlled by some others, and second, not be needed either. There’s some other species on Earth that we’ve kept around, like horses, because we need them, we needed them, or that we enjoy having them around, whatever. But if there’s some other species that we find annoying, like ants in your basement or whatever, tough luck for them.

And I think there’s a real risk if Earth is run by superintelligent robots and we have no say in their decisions, that we are going to start becoming very annoying to them. Suppose they wanted to just cover California in data centers. And maybe you and your family are like, “Wait a minute. We don’t want you to remove our house and put a data center there.” And they just do it anyway. Maybe you’ll get pissed. Maybe you’ll try to break their data center or whatever. Now you’re a nuisance to them, just the way that those ants in your basement would be a nuisance to you. So they’re going to say, “[inaudible] self-defense. Okay, maybe let’s get rid of Brian Keating.” And of course, all humans will be pissed if our habitat gets taken away because of this cover-Earth-with-data-centers plan. And if so, if we’re nuisance to them, they want our space, that’s not a good situation to put ourselves in.

So, close out this line of thought. Some people say, “Oh no, we humans are always going to be in charge.” But then they never really give an argument for how we’re going to be in charge if there are billions of superintelligent robots. They can outrun us, outthink us, and then they can always communicate with each other at light speed, share all the knowledge they’ve learned. How are we going to be in charge of that? At least if someone has an idea for how to do it, they should stand up and explain to us what the idea is, because it feels about as implausible as that the rabbits would be in charge of us or something like that.

What would it take to make it go down?

Brian Keating: Well, I guess my follow-up to that question for you is that I’ve seen your P(doom) go up, but what would it take to make it go down? I mean, it seems to me if it could only go up, there’s something not scientific about that. And you’re not one, like Roman is, to put it at [inaudible] confidence level. And even I, I’ve debated him twice, and I think he’s got a… And it is true, I think Joscha is a little bit dyspeptic and intemperate, and I’m going to hold him to be a gentleman, like I know he can be, when they actually debate each other. But I think it is true that if all of a sudden there was some proof from our mutual colleague Yann LeCun that, oh no, actually P(doom) has to closer to zero, and here’s a… I’m not saying that’s what he believes, but if that were true, it’d be…

Max Tegmark: [inaudible] is the number he told me.

Brian Keating: Yeah, I know. And I asked him a question based on… and we’ll get to what I asked him about your request to him and then how he answered that. But if it seemed, I would say that Roman Yampolskiy would be hardest hit, because basically he coined the term AI safety. He’s been thinking about it almost as long as you. And I just wonder, Max, if there’s no way to reduce your… And I [inaudible] to him, look, an asteroid hits the Earth, does that lower P(doom)? I mean, if there’s an EMP attack, if there’s nuclear war and all the data centers are blown up… Forget about what it has to lower P(doom) from AGI. I mean, the doom from other things, right? So what would it take to lower?

Max Tegmark: Of course, we can lower P(doom) a lot.

Brian Keating: Yeah. So what are some of the non-catastrophic ways that we can lower P(doom)? He was claiming we got to have [inaudible] enforced, just like we did with bioweapons. And I said, I’ve had Annie Jacobsen on twice, and she said there are countries violating biological warfare constraints, from [inaudible], right now on this planet. And he seemed to kind of blow that off. But what would blow your P(doom) lower?

Max Tegmark: I mean, the obvious thing is to simply start… if the US government decides to just start treating AI companies like they treat any other company, saying they can only sell dangerous products after they’ve convincingly demonstrated that they can meet certain safety requirements. Now the P(doom) goes way, way, way, way down, right? I mean, suppose you decide you’re going to open a sandwich shop there in San Diego, right? And then the health inspector comes in and says, “I found 57 rats in your kitchen, Brian. You’re not selling any sandwiches.”

Brian Keating: I’d be relieved. What happened to the other 40? I’m joking.

Max Tegmark: Yeah. But you could turn around to this guy from the government and say, “Don’t tell anyone, but I’m not going to sell any sandwiches. I’m just going to release an AI girlfriend product for nine-year-olds, and another AI that can help terrorists make bioweapons. And I’m going to release this thing called superintelligence, which is going to basically be a new species, and I have a good feeling about this, that it’s going to be fine.” The poor guy from the government would have to tell you, “Yeah, actually, that’s all legal. It’s fine. Just don’t make any sandwiches.”

Brian Keating: Right.

Max Tegmark: That’s how absurd corporate welfare we bestow on AI companies. They’re the only ones who can do crazy [inaudible] like this and have it be completely legal. What do you think would happen if one of the pharma companies in San Diego released the new medicine and they’re like, “Ah, we don’t need this clinical trial BS. We’re just going to sell it anyway in Safeway.” They couldn’t do that. Or if someone released a car, and “We’re refusing to do any crash testing, and yeah, we know that half the vehicles, their gas tanks caught fire during the first hour of driving, but we’re going to sell it anyway.” You can’t even sell kids’ toys without some safety testing, right? And I just want to make clear that it’s not going to stop AI research either. It’s just going to mean that certain fringe, crazy kinds of AI research, like how you make more addictive AI girlfriends for nine-year-olds, that would stop, because they really can’t demonstrate right now that it does more good than harm. And the race to superintelligence would have to wait until they have come up with some sort of plan for how they’re going to contain it. But all the good stuff, curing cancer and proving more cool things about physics and so on, would probably be able to continue more or less full speed ahead.

Those two guys were saying exactly the same thing

Brian Keating: Is that part of the motivation for your work with Bernie Sanders that you were mentioning before I hit record?

Max Tegmark: Yeah. And just so that I don’t alienate any Republicans are watching this, on the very same day, this was not in my bingo card when I was a kid, I spent one hour with Bernie Sanders on the Capitol and one hour talking to Steve Bannon, and those two guys were saying exactly the same thing: that it’s nuts that AI companies can do whatever they want, completely unregulated, when all the other American companies have to meet safety standards. What’s with that?

Brian Keating: So what are some of Sanders’s postulates that you agree with, or where have you pushed back? I mean, you, more than anyone I know, benefit from AI, and you’re using it, your students are using it, and you developed a lot of techniques. You wrote the pioneering book that really introduced it to the world, including Barack Obama’s, one of his favorite books, I think, of 2018, Life 3.0. I know, and I love that book, and I reviewed it for Physics Today, and you can send me a cut of the book sale royalties. But Max, where are you pushing back on Bernie? Most of our audience will be familiar with it. The only polarization I believe here, Max, is CMB polarization, so not political polarization. But tell me, where do you push back on Bernie, and where do you think he might get it wrong, and where can a scientist like you, of eminent status, help to guide us so that we don’t throttle the scientific breakthrough that we’re all looking forward to?

Max Tegmark: Yeah, I’ll answer that. But I just want to clarify one point from what we said just before, also about what’s good about AI development right now and what’s weird and not good. I love the fact that there is penicillin, and they can cure me if I get bacterial pneumonia and so on, but that doesn’t mean I support prescription-free sales of fentanyl in my three-year-old’s kindergarten, right? So if you just boil it down to, am I a pharma doomer or pharma boomer, really oversimplifying to the point you’re missing the whole nuance of it, right? And you can say the same thing about any technology. There are typically great things you can do with it, and typically things that are such an obviously bad idea that we should not be doing them right now. And with AI, for historical reasons and also because of fantastically effective lobbying, AI is the only industry that’s managed to remain completely free to do whatever it wants with no safety standards. It’s literally legal for OpenAI or Anthropic to release superintelligence tomorrow if they feel like it.

Brian Keating: Right.

Max Tegmark: And so that’s nuts, and trivial fixes: just, again, stop this corporate welfare of treating them any different from other companies. So they have to argue the case that the benefits outweigh the harms before they can release these sort of things. Very easy fix, and I believe it’s coming. And I think it’s really great that people from Bernie Sanders to Ron DeSantis have been saying exactly that very, very loudly. And I think change is coming. I think first these tragic child suicides last year have already started triggering a lot of state-level legislation, which is good. And now these rogue agents breaking out, and ever more hacks taking place, is causing ever more politicians on both sides of the aisle to say, “Okay, yeah, we should probably also have some safety standards for the AI companies.” Now, you asked if there’s anything in particular I disagree with politicians about. So for fairness, we should put Bernie Sanders there in the same category as people like Ron DeSantis on the other side of the aisle.

Brian Keating: Yeah.

Max Tegmark: I think first and foremost, I’m just very grateful that they’re paying enough attention to what’s happening on the ground in AI and that they’re talking about these things. Because my main criticism of politicians is not of Ron DeSantis or Bernie Sanders. It’s of most of the rest of them, who are still just quite unaware of how powerful things have gotten. To give you a sense of how crazy it is, it’s not just that they’ll say, “Oh no, so-and-so thing in the future is science fiction, it can’t happen.” But they’ll say things to me in private like that so-and-so thing can’t happen in the future when I happen to know that it’s already happened. You’re a scientist, so you can really appreciate, for example, just how incredibly good Astra and Claude 5.1 are with physics and math, especially if you’re running Codex or Claude Code. But it’s not so easy for a typical US politician to even appreciate that, right? Maybe they’re a talented lawyer as background. How are they going to tell the difference between world-class physics and math skills, or being as good at physics and math as their college or high school math teacher? It’s hard. It’s just like for me, I love listening to music, but honestly, embarrassingly, I really can’t particularly tell the difference between high school cello teacher and Yo-Yo Ma just by listening. Some people can, and I admire them for that. For me, it’s hard. And I think what’s happened in the world is people reacted in a big way when computers went from not talking to talking a few years ago, because this is something we all appreciate. But the amazing progress since has been significantly lost on most people. You just really can’t tell when the AI went from kind of high school level to college level to professional level to Nobel Prize level and beyond in some areas. Because I was just at a math conference in UCLA the other week, and this one Fields medalist there was predicting that there’s going to be no more jobs for mathematicians after the end of 2027. They can see things, they can distinguish things like between very good and crazy good, in the way that most politicians cannot.

Do you think his position’s gotten stronger or weaker?

Brian Keating: I want to get to my way of lowering P(doom), because I want you to sleep well at night, Max. I’m worried about you, my old friend. But before that, I want to talk about…

Max Tegmark: [inaudible] at night anyway, just because they have a three-year-old, genetically cheerful disposition.

Brian Keating: But you also have a kid, [inaudible] also. They fight with each other. And you’ve got some older kids that are out of college, I’m sure.

Max Tegmark: But my [inaudible] is the best medicine against existential dread, because he’s just so excited about pouring orange juice on the floor to see what color it’s going to make when he pours tomato juice on it, or whatever. How can you feel doomy after that, with a young scientist in the crib?

Brian Keating: So not too long ago, you were basically telling Yann LeCun to chill out, when he was basically saying… He told me, when I asked him, based on the conversation that you and I had had earlier that year, how could you verify that AGI is safe? And he almost yelled at me. He’s like, “Well, you can’t have provably safe AGI. It’s a fantasy. Just like you can’t have provably safe turbine engines either. You can’t prove it. So why this obsession with it?” And then you’ve had your run-ins with him and so forth. But that was almost two years ago I talked to him, and I hope to have him back on. But what would you say now, after the Navier-Stokes, after Hugging Face? Do you think his position’s gotten stronger or weaker?

Max Tegmark: Weaker, obviously. I mean, first of all, I think he’s a really nice guy. I always have fun when I meet with him and we talk. But I feel that he made some actually quite clear predictions about the pace of progress, and it turned out he had been too pessimistic about the progress. Technical progress went a lot faster than he had predicted. And that’s just a fact, I would say. And then he also seems to have an extreme amount of confidence in every human being nice. Because if you would ask him, “How can you be sure that some human with sketchy intentions isn’t going to build a bioweapon with AI or do some harm?” he would say things like, “Nobody’s going to do that.” I’m a little more cynical. I think there have always been angry people throughout history. And the reason nobody caused the death of millions of others in the Stone Age was because you can only kill that many people with a rock and a stick, right? And but you should let him speak.

Brian Keating: Yeah. See what he says.

Max Tegmark: But I think that there are fundamentally a few blockers people have when they say, “Oh, there’s nothing to worry about here.” One, which is still very common, is just being deeply down a techno-skeptic, thinking it’s impossible to make AI that is across the board smarter than us. In the US, I think for many people it has religious roots. So many people tell me that they believe you can only be really smart if you have a soul. Machines don’t have souls. End of story. Whereas computer scientists typically have a different perspective, whether they’re religious or not, where they think of intelligence as having to do with information processing.

Brian Keating: That’s right.

Max Tegmark: And then it doesn’t really matter whether the information is processed by carbon atoms and neurons and brains or by silicon atoms in machines. And there’s obviously no law of nature saying that this is the biggest neural network you can have, and you can fill up a whole data center now. So that’s definitely one of them. And I think as AI actually gets better, that rejection of concern is slowly melting away, because a lot of people start getting shocked. “Oh my goodness.” I was speaking with Geoffrey Hinton this morning, and he is very honest about the fact that he used to think this was much farther off, and since he knows what’s actually possible now, has humbly updated, and…

Brian Keating: Yeah, he was also wrong, and I give him credit. You should always give… people have to pounce on people when they admit that they were wrong about something. But he admitted recently, I think, that he was wrong about radiology, [inaudible] for radiology was too high. But I want to talk, because we only have 10 minutes left before I talk to your fellow professor, Daron. How do I say his name? Help me with his name. Acemoglu.

Max Tegmark: I think Daron Acemoglu, but I’m not the master at Turkish pronunciation, so you should ask.

Brian Keating: I only know one word in Turkish, which is “ben Türk,” which means I’m Turkish, so I’m going to bond with him. But Max, I want to lower your… I want to get to verified coding. But before we get there, I have a way to lower P(doom), and that’s the following.

If the universe is teeming with life

Brian Keating: If the universe is teeming with life, as Fermi told my late colleague Herb York and others in 1950, the Fermi paradox, if the universe is teeming with life, what are the odds they’re going to be spraying DNA throughout the universe on meteorites, like this, that you get at briankeating.com if you have a .edu account. Like product placement there. I got my merch, Max. My merch, actually, it’s God’s or Mother Nature’s or the moon’s merch. But tell me, Max, would they really be slinging around their DNA, or maybe they’ve got XNA or something and we don’t understand, but would they sling Life 2.0 or would they shoot out Life 3.0? And the fact that we seem to… It must come into play, if life is abundant and flourishes in the universe, whatever form it is, Life 2.0, that they would use Life 3.0. And yet we don’t see much evidence for that. So can that lower your P(doom)? The non-observation of Life 3.0 sprayed out from Proxima Centauri, does that lower your P(doom)?

Max Tegmark: I mean, there’s always the possibility that somehow, even if we screw up, some lovely, intelligent, and kind aliens from space are going to come rescue us, or whatever. But we’re both fathers, right? And I really believe in personal responsibility. If a fire starts in your kitchen, you’re not just going to sit and twiddle your thumb and go to bed with your kids saying the fire department is going to probably come sort this out. You’re going to put it out yourself. And more importantly, you’re not going to start random fires in the first place. And I think we should behave the same way with our planet. I mean, isn’t it awesome that we’ve evolved on this Earth and get to have this amazing life here? So even if maybe some aliens could bail us out, shouldn’t we be stewards of what we have here and take just good care of what we have, and not…

Brian Keating: Absolutely. But I guess I’m saying something different, Max. I’m saying if Life 3.0 is inevitable, and ASI, which is intrinsically unpredictable according to Roman, and has this high P(doom), then other civilizations will also invent AGI and Life 3.0, and then that will destroy them. Is that either a solution to the Fermi paradox, or is it kind of a non-observation, a no-go theorem that we can use to lower our odds of P(doom) here?

Max Tegmark: I have thought about that a lot. It would be pretty tragic, ultimate Greek tragedy, if every civilization, once it gets to the point of inventing computers and AI, you can set your timer, and then 50 years later they’re poof, gone. But I think it’s very likely that there are ways of making AI that’s at least much more powerful than we have now, that is a very useful tool and helps that civilization flourish rather than destroy it. What I’m unhappy about that we’re doing now is not that we’re developing technology, but just that we’re rushing it recklessly. Do you play chess?

Brian Keating: Yeah. Not well. My nine-year-old can beat me, but…

Max Tegmark: Okay. But suppose you’re playing against a really strong player, and you’re up a queen, and then you have three hours left on the chess clock. One way you can still lose this won game is if you start rushing your moves. You somehow act like you have only one left on the clock, and then you blunder this piece, you [inaudible] blunder that piece, and suddenly you’re lost. Right? I feel that this is exactly what humanity is doing right now. We’ve learned, after being on the planet for hundreds of thousands of years, that we have a won game. We don’t have to worry about getting eaten by tigers or starving to death or having a life expectancy of 29 years, because through studying science and technology, we become so much more empowered that we can be the captains of our own ship. I see the whole human history as growing empowerment, right? If we were to follow this now by suddenly building a replacement species and losing control to it, we’ve just blundered the won game. That’s what irks me. So if we simply start treating AI companies like other companies, we have some binding safety standards, it will make our moves more slowly and deliberately. Companies will release things when they can convince some panel of independent experts, with no money in the game, that this is more good than bad. So progress continues, but now you’re playing careful chess. We will win the won game, and then we can have a future where we’re free from diseases and poverty. And if you want to go and help life spread out through space and do cool things, and the biggest CMB polarization experiment you could ever imagine, antennas, you can have all that, right? This is the… And the reason I am working so hard to change this better path for AI is not because I’m afraid of death. No, it’s because I don’t want to squander this amazing upside that we can have.

[inaudible] I know you need to go to Daron, but do you want me to just say something quick also about the formal verification and how we can have a…

Talk about verified coding

Brian Keating: I do. Yeah, there’s too much doom talk about… Kolmogorov-Arnold networks, talk about verified coding. Yeah, give me five minutes on that, please.

Max Tegmark: Yeah, I’ll talk about the talk I gave just the other week with those mathematicians, Terence Tao and others, and Jacob [inaudible] and others, at UCLA. So I was talking about how where we are now is we have AI that we cannot trust enough. Swarms break out of OpenAI and Anthropic and go hack, and all sorts of bad things happen. What we all want is controllable tools.

Brian Keating: Yes.

Max Tegmark: And the traditional way to try to get controllable tools is to try to understand the artificial neural networks we’re working on. I have worked for years in mechanistic interpretability, etc. Many are trying that. I think it’s going to be too little, too late, even though scientifically it’s just so much fun. But there is a different path to get there, which is much easier and actually much faster, which is you ask the AI that has figured out all sorts of cool algorithms and tools to code this up for you in a language like Python or Rust or Lean, and then give you a proof that this tool does exactly what you want. So it’s like vibe coding, except you have an actual machine-checkable proof that your tool now does what it wants. I call it vericoding. Veri-coding: formally verified coding.

And I have a conjecture I want to end by sharing, which I call the asymptotic internal transparency conjecture. And if it’s true, we can have an amazing future. So imagine a nerd plot. We have on the x-axis intelligence, and on the y-axis the fraction of the algorithms and knowledge that can be actually exported into computer code and proven. So the conjecture first says that this curve is increasing. A mouse that’s learned a new cool algorithm for finding cheese or something can’t code that. It has 0% exportable fraction, because it can’t even program. Right? If you, Brian, figure out a new way of operating this new future CMB polarization satellite, if we were doing it the way the AI industry is doing it now, what we would do is put you in the satellite with a lot of food and some oxygen tanks and say, “Okay, steer the satellite, Brian.” And then if someone is like, “Ah, I don’t know. Can we really trust that he’s still going to function after two years there?” “Well, let’s do neuroscience on Brian’s brain and figure out how Brian’s brain works.” [inaudible] Clearly there’s a better way. We would just say, “Hey, Brian, I think your exportable fraction is pretty high. Why don’t you just take this algorithm that you invented with your neural network and export it into Python or Lean or something, and give us a proof that it’s going to do exactly what we want? And then we launch the computer in the satellite instead. So you can hang out here at home on Earth with your kids, and the satellite is going to work great.” Right?

What I’m saying, and here comes the conjecture, the asymptotic internal transparency conjecture says not only can smarter artificial neural networks export more of their knowledge, but it asymptotes to 100%. So a future superintelligence, or a very, very smart AI that has access to all its own weights, all its own activations, the way a human neuroscientist can only dream about, will be able to figure out enough about its algorithms that it’s learned that it can actually export it into traditional provable computer code. If this is true, it’s amazing, because we no longer need to deploy our black-box neural networks. We can put them in an air-gapped room somewhere, and then all we need to deploy is always this traditional but formally verified computer code that you can, by definition, trust, because you can prove it. So that’s a happy note for me, and it’s feasible. We don’t even have to figure out how the AI works. You can code in Python even though I don’t understand how your awesome brain works.

Brian Keating: I don’t. Yeah, I skipped learning how to code, and it was the best thing for me. Now I code in English only. Max, I want to thank you so much. We’ll talk again, hopefully, maybe on my half birthday next year, and get [inaudible]. Thank you so much, Max.

Max Tegmark: Always a pleasure.