## Simulation hypothesis and P(doom)

Brian Keating: All right. Today we are joined for what promises to be an interesting conversation between two intellectuals I have great respect for, both past guests on the podcast. And making his second appearance, it’s Joscha Bach. Joscha, how are you doing today?

Joscha Bach: Hello, thank you for having me on.

Brian Keating: We had a great conversation about [inaudible], about the ghost in the machine, over the summer and the spring, and that was well received. Got more attention on Reddit. People, they’re very opinionated about you on Reddit over there, as they are about almost anything. And then Roman Yampolskiy needs no introduction, most popular guest ever on many podcasts, and now has his own and has hit 200,000. [inaudible] Roman, welcome back to the Into the Impossible Podcast. Your third appearance.

Roman Yampolskiy: Thanks for setting it up.

Brian Keating: Yeah, it’s a real pleasure. So I like both of you guys a lot. And when my kids fight, I get mad. Over the summer, there was some clip going around about Roman, as he’s often wont to do, promoting the concerns that I think are legitimate we should talk about with regard to AI safety, which is a term he coined and has certainly done more than anybody on earth to bring awareness to. And simultaneously, Joscha, who’s one of the deepest thinkers available, had a little bit of a snarky reply, a comment on Twitter, that it’s not possible for Roman to simultaneously believe in the simulation hypothesis and also be assured, if I get this correct, of a hundred percent, nearly a hundred percent P(doom).

So I hate P(doom). I think P(doom) is like the redheaded stepchild of the Drake equation. I think it’s nonsense. I mean, just like the Drake equation, no one’s ever looked at the Drake equation and done an actual calculation or done an actual measurement that could be falsified. It’s just a framework. I think P(doom) is the same. But Joscha, can you explicate your thesis a bit more? Why is it simultaneously impossible for Roman or anybody else to believe in the simulation hypothesis? Or why is it a contradiction and also believe in P(doom)? [inaudible]

Joscha Bach: No, it is totally possible to believe both that we are in a simulation and that we are doomed to a certain degree or to a hundred percent, because it could be that you simply in some kind of computer game, which I think what the simulation hypothesis comes down to, that we are running in a computer in the basement of the nerd in the parent universe. And in that computer game, we can be doomed. Of course, this can be the case. It could also be that our actions within the game, despite the game being deterministic and so on, which leads to the usual question about how we interpret our freedom of will, lead to a certain outcome in which our actions are causing the probability of that doom to change. That’s all possible. The question is whether it matters.

And so when you are in a computer game, the thing that you are in is not necessarily the thing that it’s a game, right? You are in there for a certain amount of time. There’s ultimately no point to this whole thing. And so the question whether you should be worrying, whether the AI wins or you win, is a different one for most people, I would say, than it is when you believe that we [inaudible] base reality and what we do originally matters, because our creativity decides on how this universe is going to end up.

Brian Keating: Do you think it’s possible, Joscha? First of all, let’s get this out of the way. Do you subscribe to any validity, any marginal utility of the concept of P(doom)? Because I don’t, but do you?

Joscha Bach: Well, I think that it’s very difficult to estimate P(doom) reasonably, but you can make the argument, if you have different people who see the future differently, that this should lead to different probabilities. It might not be that these probabilities are important as a real number, but they are important in relationship to each other, in the sense of, if you were to shut down the big AI labs, is this going to increase or decrease the probability of this or that event? And in this way, I think it makes sense to ask people for how their probability estimates change, even though I think there are too many factors involved to put meaningful even magnitudes on the P(doom) estimates. I sometimes say that my P(doom) without artificial intelligence is higher than this, right? That’s also a P(doom) estimate. But I mean, in the long run, P(doom) is one, and in the long run we are all dead, as a famous economist once said, and that’s inevitable.

But back to this original question. You probably remember the movie The Matrix, and in The Matrix people are living in a simulation, which means that the reality that they are perceiving is not the reality of the base universe. It’s one that is artificially created to entertain, to deceive them. It’s the smoke and mirrors reality, like ours would be, that pretends to be quantum mechanical and so on instead of indicating that we are base reality. And if that is the case, then you still have people who say, but I prefer to be in this pseudo reality instead of being in base reality, and have a good steak in there. And this good steak that you want to have, that is nicely simulated, I wonder if this is what Roman is after. So basically he is [inaudible] by being a doom monger, by going in on the topic that is most interesting, having probably the highest P(doom) rating out there, higher than Eliezer, the grandfather and OG doom master of artificial intelligence. And I wonder if this is a deliberate position to simply occupy a useful position in the media universe, regardless of the despair and anxiety you might be creating as a side effect. Whether this is responsible or not, I wonder.

And what is your thinking on this, Roman? Do you feel that you are a legitimate prophet and warner who is simply doing his best to sincerely increase the probability for a good outcome? And if so, do you think this is the best strategy?

Roman Yampolskiy: If I didn’t think it was the best strategy, I wouldn’t be doing it, right? If I remember correctly, your tweet was more about, if it’s a simulation, why would anything matter? And I disagree completely. I think pain is pain, love is love. That’s the only universe I have, and that’s the only life I have. To me, the difference between simulation and base reality is the substrate. Is it atoms or is it bits? Everything else stays the same. I have no problem worrying about creation of advanced AI within a simulation and how it can impact my family, humanity as a whole. I see zero contradictions in those statements.

Joscha Bach: Okay, maybe we don’t have a disagreement then. If your idea of simulation universe is simply it from bit, right, then I wouldn’t think that you are a simulationist in my world, because the atoms are not something that somebody has discovered as base reality. Matter is not something that exists as base reality. It’s a particular way in which we talk about information. Simulation is slightly different.

Roman Yampolskiy: I don’t think you know anything about base reality, right? You haven’t visited yet.

Brian Keating: Wait, Roman. Yeah, please, please. No, go ahead.

Roman Yampolskiy: Do you want me to continue my answer or are we gonna have more of a boxing match? I’m happy with both. I just need to know what the format is.

Joscha Bach: [inaudible] No, sorry. Apologies.

Brian Keating: Go ahead, Roman. We’re keeping it very civilized. Yeah, go.

Roman Yampolskiy: So P(doom) is a useful tool. We do risk assessment in all sorts of industries. You know how dangerous it is to fly, how dangerous it is to drive. The numbers are not randomly generated. I agree that someone with P(doom) of 46% has a lot of explaining to do us how they got there. Levels around zero and one are much easier. You may have zero concerns because you have a working safety mechanism in place, or you may argue that the problem is impossible.

If I ask you about your probabilities of creating perpetual motion device, I’ll say you’re going to be around zero, right? The inverse of it, of course, is one. And that’s how I feel about our ability to control general superintelligence indefinitely. You can’t. It’s impossible. You’re not going to have a technical solution. What does that mean? It means if you give it enough time, it will probably end up very poorly for us, because it doesn’t care about you. You’re not controlling it. It has other goals, and limited resources in our universe and our planet. If it decides to use them for something else, it’s not gonna care how many of us die in that process. So I think, yeah, P(doom) is a useful tool. I don’t know if having an extra nine would make me more impactful. Usually when I meet people who have concerns about AI for different reasons, they worry about loss of meaning, unemployment, discrimination. All those things can be added as increasing overall dangers from this technology, but I think it’s not the point.

The point is that I’m saying this is equivalent to requesting someone to create a perpetual motion device. You are asking for a perpetual safety device, a system which will never fail once. New models, new data sets, interaction with any agent, malevolent agents, recursive self-improvement, it will never fail once. It’s always one step ahead of something smarter than you. I think it’s impossible. And so I have a high P(doom). Is it bigger than others? Yes.

## Let’s go back to 1945

Brian Keating: Well, let’s go back to April 1945, before the Trinity test. There were a lot of physicists, and including some of my late colleagues here, Herb York and others, who were rightfully concerned about not the nuclear war aspect of P(doom) of nukes, but of things like igniting the atmosphere, of a runaway uncontrolled, essentially nuclear reaction that would lead to P(doom) of 100%. And there were people like that. I’m not saying Herb York did, by the way. But there were people that believed that. And then later there was the P(doom) of nuclear war. And I want to teleport back. Well, let’s say you guys were there. What do we do? Do we not create nuclear fission because of some very smart people who believed that P(doom) was greater than zero? What would you have done, Roman? Let’s go back to that time.

Roman Yampolskiy: So all the examples with previous technology, whatever it is, nukes or fire, always talk about tools and a human being making a decision. The situation is fundamentally different. We’re not talking about a tool, we’re talking about agents. They will be independent decision makers. It will not be the standard business as usual where if I understand motivations of this human or I can kill this human, then things will be okay again. This is complete surrender of control. Now we can talk about at what level of danger you can run an experiment on all the other humans. Eight billion humans, you don’t get consent from them. They cannot consent, because consent requires understanding. They don’t know what’s going to happen. So consent is impossible. But we can discuss at what level it’s still reasonable to risk their lives for greater good. But that’s not even the situation we’re in. It’s not a single experiment where we’re saying, if only we allow GPT-6.1 to come out, after that everything’s great.

We’re talking about all future models. We’re talking about recursive self-improvement. We’re talking about intelligence explosion. People worry about superintelligence. Not in a Donald Trump renaming sense, but smarter than all of us combined. There are levels to it. System smarter than you 10x, 100x, million x. Those are not the same problem. To suggest that you will be able to always stay in control of all of them seems insane.

Brian Keating: But you couldn’t do that with nuclear weapons either. And you’ve brought up the bioweapons, and we have to look at today’s headlines coming out of Siberia, which is exactly the scenario my friend and guest Annie Jacobsen painted in her book A Biological War. And last time you and I spoke, you said, we have treaties about it. Well, Russia’s a party of that treaty. And in 1971, when recombinant DNA was invented and created, what everyone say, there was no consent either. I guess I wanna ask you in a minute, Roman, take us through the actual steps. Because I can think of a million, just like igniting the atmosphere, or when the Apollo landers were going, they thought they would sink to the center of the moon because they didn’t know about the structural integrity of the regolith on the moon. There’s millions of scenarios you can’t do counterfactuals on. And I want to ask you, take me step by step through actual superintelligence leads to doom. I want to ask you that in a minute. But Joscha, I want to give you a chance to respond to what Roman’s saying.

It is an argument that I have a lot of respect for. I don’t agree with Roman, but I have a lot of respect for him in that we can’t give consent. We haven’t given consent, despite all these things. So, I mean, do we need to have consent before we shut things down? I mean, at what point are you comfortable with putting the genie back in the bottle if Roman’s correct? I mean, your P(doom) might be irrelevant, but how do you speak for the rest of humanity? Not you personally, but how do we as scientists or as technologists?

Joscha Bach: There are number of considerations when we try to compare the nuke situation with the situation of artificial intelligence. And it makes it very difficult to compare even, and it’s also difficult to dismiss the comparison out of hand. I, like many people in my generation, Generation X, think that now that I’ve been wrong about the impact of nuclear weapons. I grew up thinking in the 1980s that we are just a hair’s breadth away from assured destruction and that the risk of nuclear armaments is way too high for humanity. Now I think the adults have been correct, and a generation of students who took to the streets and were very upset about this turned out to be wrong. Nukes have prevented several world wars that would have been happening between World War II and now. That would have brought tremendous destruction on Europe and potentially large parts of Asia and maybe even America. This basically more than half a century of peace that we had is completely unprecedented for large nations like ours. And I think this was achieved due to nuclear armaments, which have been empirically quite safe. Now, it’s very difficult to compare the different timelines. We cannot really sample this. We don’t know in which number of universes this went wrong. But it turned out, at least as far as we know, they have been quite controllable.

But the situation with a weapon over a few decades is a very different one than the one with artificial superintelligence over the entire length of the light cone. So basically, over a long enough time span, it is certain that we are hit by another meteor of the caliber that killed the dinosaurs. It’s quite certain that it will be a super volcano that can lead to a global extinction event. And it is also quite certain that it will be other species evolving that are displacing us as a species. And if we could lock ourselves in as the pinnacle of evolution and make sure that never anything better than us evolves over the next few millions or billions of years on this planet, I think this would be a very ugly perspective.

Ultimately, the question is, what do we identify as? Do we identify as a kind of ape that is very selfish, or do we identify as intelligence and consciousness in the universe that is currently incarnated as us, and we are handing off the torch to our children, and our children are going to be a little bit different from us, and over a very long time span they don’t look like us at all? And it can also be disruptions. There can be other species that take over, as the mammals took over from the dinosaurs. But this is not something that is going to happen in the next few decades, and I think we are justified in this hope. And so in a sense, I believe that if intelligent species are meddling about, they will also with some probability create non-biological life and non-biological intelligence and agency, and spread life and biology beyond organics.

That’s not ultimately a bad thing. The question is, something happen in the next few decades that makes the planet uninhabitable and that makes the future very, very boring? And that’s a very different question. So I would agree with Roman that if you take the time spent for long enough in the future, it’s quite likely that there will be more intelligent things in the universe that are displacing the intelligent things that are in the universe right now. This is just how evolution works, and it’s not a bad thing. But the question is another one: are the tools that we’re currently building in principle controllable or not?

And in this way, I’m much more optimistic than Roman is. It seems that the LLMs that we are building are not developing in a way that leads to a sudden foom, as Eliezer thought a couple of decades ago, that they instead grow together with the compute that we throw at them. And this also means that the way in which we conduct the experiments, in which we deploy them, is somewhat controllable. So the idea that we suddenly wake up and we are displaced by robots and or nanites and so is not yet plausible with this technology. This does not mean that this doesn’t change, but for, I think, the immediately foreseeable future, for the immediately foreseeable technology, we are looking at things that are more controllable than I thought that it would be.

## Why is it certain that there will be a doom scenario?

Brian Keating: Roman, when I was walking over to my studio, there’s a spider that inhabits this area. And its web is always, I’m about close to six feet tall, and it barely brushed on my hair, and it’s always at that height. And the spider’s kind of adapted to me and understands me. And I don’t like these things. I have a little arachnophobia. But I didn’t go out and just start to exterminate him. I’m superior to his intelligence. I have the confidence to say that. And he and I have evolved in this way that Joscha’s sort of alluding at. And I wonder, how do you react to that? We haven’t exterminated all the spiders, as much as they might be a pain. And some of them are good and some of them are bad. Why is it certain that there will be a doom scenario? Let’s say we stipulate there is a potential for that, where some agent, some silicon mind meld occurs and some agents are there, but there also aren’t symbiotic agents that will work to mitigate a risk both to themselves and to us.

Roman Yampolskiy: So you have complete control to decide to kill that spider or not.

Brian Keating: But not all spiders.

Roman Yampolskiy: And with gene drives, we’re now trying to wipe out mosquitoes. That’s a project we actually have. We just don’t hate spiders, because they could be actually beneficial. If we wanted to wipe out a specific species, I don’t know, pit bulls, we could have, right?

Brian Keating: Well, we tried against mosquitoes. Actually, mosquitoes, you almost mentioned mosquitoes. Without mosquitoes, the three of us wouldn’t be here probably, at least it wouldn’t be America, because mosquitoes allowed us to defeat the British because of their lack of immunity to some of their disease, malaria and other things. Anyway, Roman, the point I’m trying to make, and I think Joscha is alluding to, there’s always been competition between less intelligent, more intelligent, but never has there been a targeted… And I don’t even know if it is possible to kill every mosquito and all future species of mosquitoes and so forth. But the question is, where is there an opportunity for not necessarily these superintelligent things… Let’s say I stipulate superintelligence is imminent, but that it necessarily comes concomitant with destruction.

Roman Yampolskiy: So again, look at any situation where you’re making a decision about what you want. How much care do you put into making sure when you build a new studio for your podcast that there is not an ant hill in that location? Did you make sure you didn’t genocide a million ants before building it? No, you don’t care. It’s not important to you. As long as they don’t bother you, you don’t go out killing ants on purpose, I assume. But you don’t want to be in a situation where something smarter than you, with different plans, different goals, decides how to use your planet. Anytime a more technologically advanced civilization met a less technologically advanced civilization, it ended up really poorly for less advanced one. You can look through…

Brian Keating: Okay, well, let’s take that argument and my earlier desire to have you walk us through a scenario. Let’s say right now there’s bubonic plague again going around through Siberia, and allegedly it’s not gain of function release. It happened to come from a biosafety lab, but let’s ignore that. And Russia has treaties that they’ve signed and we have treaties. Let’s say I give you superintelligence now, but it’s benevolent for now. And I say, wipe out plague. How does that happen? How do you do that? How do you instantiate it?

Roman Yampolskiy: So the example I always use is squirrels. I love squirrels. They’re awesome. Asking squirrels about how humans will do something is not helpful. The best squirrel scientists: how would humans kill us? Well, chase you up the tree and bite you really hard. It’s not useful information. They don’t have…

Brian Keating: But we can’t even say how we would do it, Roman. I mean, I can’t even say how my graduate student would do it, assuming that I’m less intelligent than he is. It’s not that big a gap.

Roman Yampolskiy: So the whole point is I cannot tell you how something smarter than me will accomplish tasks. If you’re asking me how I would do certain things, I can propose solutions. I don’t think you are looking for those. Shutting down bioweapons labs around the world seems like a good idea. But that’s not the point of the argument. I’m trying to show you the cognitive gap between us and squirrels is probably smaller than cognitive gap between all of humanity and systems we are creating. They are not tools. They are…

Brian Keating: That sounds like the AI of the gaps to me. It sounds a little bit like the AI can do anything, we just don’t know what it can do until we know what it’s gonna do. That’s why I’m asking.

Roman Yampolskiy: Well, we’re starting to see some examples. They just published, I don’t know, 400 mathematical proofs for things we couldn’t prove in like a week. Something like this happens every week now. Protein folding problem was solved by AI. All of those are impossible tasks for us. We don’t know the mathematical proofs they just published. Most mathematicians look at them and go, this is some magical algorithms, I don’t understand half of it.

## Where do we have to go to have actual falsifiable claims?

Brian Keating: Are they impossible? Joscha, what do you think of this claim that when you have the capability to propose… And it’s frightening to people, as you pointed out, to hear these things. And the average person, we’re not talking to the average person, we’re talking to the superintelligent people that listen to my podcast and Roman’s podcast, and soon your podcast. But the point is, Joscha, when we look at these arguments, it just seems to me like you can propose anything and it’s not falsifiable. And as a cosmologist or physicist who works on experimental verification and falsification, that’s troubling to me. Where do we have to go to not have counterfactuals, but to have actual falsifiable claims?

Joscha Bach: I suspect that this kind of thinking doesn’t really work when we are talking about out-of-distribution events. The future is an unknown land, and this does not mean that we can just come up with some kind of null hypothesis that we like or that was dominant in the past and say, you need to convince me that this is no longer true, because this means that you are steering for your doom in a way, right? But it also means the opposite. [inaudible] constructing stories in a very large possibility space. So I don’t see that there is a strong reason to believe these stories without alternative, but I also don’t see a reason to dismiss them out of hand simply because these things haven’t happened yet. And I’m completely with Roman in this way, that for things that haven’t happened yet, we need to look at the possibilities.

We don’t understand all the possibilities, of course, of the future. We don’t know how things are actually going to interact with each other. But we have to be in mind that the situation that we have created now, and that we steered ourselves in as a species, is unprecedented. This has not happened before, and we don’t know what’s happening from here.

Brian Keating: We do know, guys. I mean, this is the whole thing. I get this thing in astrobiology. People say the universe has ten to the twenty-fourth possible planets, and therefore life is guaranteed on some of them because it happened here once. No, that is completely fallacious. And I always bring up this argument. I’ve been to Antarctica twice. I spent a lot of my life there. And every time I go there, I’m one of 700 people on a continent the size of Germany plus Latvia plus the southern Texas. I’m under a thousand people there.

So I say, well, on probability arguments, it should have one seventh of the population of the earth. And of course that’s completely wrong. But my point is, I’m shopping for a car for my teenage son. I mean, God help me, guys. I have a teenage son now, he’s about to get his learner’s permit. He wants a car. I’m like, what’s the safest car? Well, Cybertruck. Okay, great. So I test drove a Cybertruck, but better is a Model 3. I’m not gonna spend eighty thousand bucks on a metallic wedge, but Model 3 is pretty damn good, full self-driving. I can monitor him. And I’m driving around this freaking thing. It’s taking me everywhere. And you know what it’s not doing, guys? It’s not driving on the sidewalk to get three milliseconds quicker to 7-Eleven. It’s getting there, it’s following, it has certain rules. And Roman, you and I have talked about this. Yes, artificial intelligence is wonderful. You love it, you use it all day long. I’m sure you use it almost as much as me. No, I’m just kidding. You probably use it more than I do.

But that is a hypothesis, that is a version of the null hypothesis, Joscha. So I don’t agree that we have no way of testing. We have deployed AI numerous times, from autopilots on planes to autopilot and full self-driving that I just mentioned. How can you say that we don’t have any ability to forecast because it’s an unknown land in the future, Joscha?

Joscha Bach: We have ways of forecasting, of course, but the worlds that we are forecasting are quite different ones. The trajectories that we can forecast are different ones. And there is no hard and fast criterion to decide which future is more likely than the other. For instance, the self-driving car that you describe is not a thing that is threatening us with taking over and controlling the future, because it’s an appliance that is trained to fulfill certain driving tasks and can only do those. It’s very similar to the chess computers of old, which could play chess and nothing else.

Brian Keating: But how is it different than ChatGPT 7, which doesn’t exist yet? It’s an appliance, it’s a tool.

Joscha Bach: Well, interesting question. The GPTs that we using are also a class of devices that we didn’t really anticipate when we thought about general AI. They are in some sense deepfakes of human intelligence that are, despite what some philosophers are saying, actually anthropomorphic. They’re anthropomorphic because they are shaped after the human shape.

They take human media output, the text that we write, the pictures that we draw on the internet, and then they are reproducing them. And as a result, they are reproducing a lot of the causal structure beneath them. And there is debate about the question whether this is actually an off ramp to general intelligence, whether these systems are actually generally creative, or whether they are just ultimately deepfakes that are so close to us that they are even overtaking us in all areas of expertise, but are not actually threatening to us to lose control over this, because basically, ultimately, they do what they’re trained to do. And despite them being unpredictable at the edges, the idea that they suddenly take over and control robots and become self-improving and so on all of their own, without having any say in this, it’s not an obvious thing that happens without alternative.

I suspect that the thing that Roman is worried about is a different kind of artificial intelligence, one that nobody has built yet and that most people agree they don’t want to build. It’s one that is basically working in some sense like the biological intelligence, in such a way that it’s radically adaptive to any kind of substrate, that it is improving itself, that it’s basically alive and it grows without bounds and can also grow very quickly. And this idea that it’s basically AI as a virus, as you could form it, is an alternative version, right? The LLM, I think, is not a virus yet. And it’s not clear if it can be turned into one or if it accidentally will turn itself into one. I personally don’t think so.

But the idea that a lot of artificial intelligence sci-fi had, and a lot of the artificial intelligence vision of the field had, was that AI could also be developmental, as something that is growing very much like a human spirit does, and then basically colonizes a substance, shapes it into a new kind of organism that is taking over the biosphere. And this is, I think, what Roman is worried about. But the question is, is this even on the horizon right now? And if it is, is it something that we can steer clear of, or something that we can control in a lab and say, okay, if you make an experiment, but you don’t grow this larger than a squirrel?

Brian Keating: Roman, how do you react to Joscha and what I mentioned a minute ago?

Roman Yampolskiy: So the part I agree with, we’re misusing the term AI to mean completely different technologies. AI as a tool, something we can do very well, use all the time. I love it. We can verify it, safe and productive. And tools could be advanced. A self-driving car is exactly that. It’s a useful tool, saved many lives, hundreds of thousands of lives. Let’s do more of that. Then you have human-level AGI. Not exactly the same, but for the purposes of this discussion, as dangerous as a human.

Can do stupid things, can lie, cheat, all that, but we know how to counteract it. And then you have recursive self-improvement process. You automate scientists, engineers working on the next generation of AI. Last time I checked, we are 28% automated in that cycle right now. That’s a lot, and it’s growing exponentially. The moment you get to 100, you have a team of 10,000 agents working nonstop on making next generation AI. They don’t sleep, they don’t eat, they communicate pretty well. So you’re gonna get great improvements in quality of AI, not on a scale we used to, where it’s maybe once a year for a completely new model, but every month, every week, it keeps getting faster and better at improving itself. That’s where we get to superintelligence. Not Donald Trump superintelligence, superintelligence where it’s smarter than all of us at everything, including doing science and engineering.

Brian Keating: Come on, Roman, you gotta be good to Donald Trump.

Roman Yampolskiy: I love Donald Trump. He’s my favorite Trump of all Trumps. But the problem is we talk about this is a natural evolutionary process, or we always, all species die out. We’re done with evolution. Darwin is dead. We switched to intelligent design. We’re designing systems, they are designing systems. It’s not random, it’s not selection. We are purposefully creating those systems. Then you say nobody wants to build superintelligence. It’s in the name of the labs. Whatever it is, safe superintelligence or personal superintelligence. The goal is to raise trillions of dollars to create this monstrosity to replace you and me. They don’t want to give you a useful tool to be more productive or creative. That’s what I would want to see. Deploy this through economy. Where is all these applications of GPT-4, GPT-5, GPT-6? No, they skip it and they build GPT-7 instead. And we see internal experimentation shows there is zero control. The agents escape, they hide.

Nobody monitors them for months. They get together, they have permanent memory. You can’t even shut them down. They keep information stored on the internet for the next agent to come around. So everything we predicted 10 years ago about this is unfolding exactly as predicted. I don’t think anyone has a better prediction record than AI safety community when it comes to this. And recursive self-improvement is not a pipe dream. That’s what the labs are telling us. They started with junior machine learning researcher AI right now, helping with programming, helping with design, but all steps of that process will be automated, possibly next year. At that point, yes, we’ll have a paradigm shift. It’s not going to be gradual improvement where you can sort of keep up. It will be a complete change. And at that point, it doesn’t mean it kills you that year. It means that you’re no longer in control. It decides what to do with you and when. If it wants to wait fifty years, pretend to be nice and wait until you surrender all the control willingly, like people do. Here’s my bank account, here’s my email account. Then that’s a smarter move. You don’t need to have a definitive treacherous turn where there is a chance you’ll get defeated. You can just wait a little longer. The thing is immortal.

## GPT-8 will run on an abacus

Brian Keating: So yeah, I’ve said a couple of times in the past that I don’t know on what substrate GPT-7 will run on, but GPT-8 will run on an abacus. What do you make of this, Joscha? That this is gonna come, and it seems like nuclear fusion to me. It’s the energy source of the future and it always will be. I’ve told Roman this, I don’t know if I said this to you, but since it’s trained, it’s a backwards looking PID situation, where it is trained on human information. And yes, it can generate, and I know a little bit about that, but the corpus is the yottabytes, or the Joscha bytes, of data that’s on the internet. That’s primarily what gives the advantage to the frontier labs, right? So are they waiting for superintelligence? Does it need to have The Fast and the Furious 12? Is that really what, GTA 6 is supposed to, like, is that really what it needs? More training data? ‘Cause then it would seem to be completely dependent upon us.

And in fact, even if it’s not, if it’s past the point, you can’t erase what’s in there already. All my books are uploaded and somehow subsumed within Claude and ChatGPT. Roman’s books are as well. Everything you’ve ever written is there. You can’t expunge it. It’s in there permanently. So therefore, it’s always gonna reflect this cause. And you and I spoke in May and June about this, Joscha, that it’s sort of this godlike thing that Roman just mentioned, that’s kind of intelligent design. But it’s not a god, it’s the AI. Where do you come in all this? I mean, if it’s a backwards looking indicator, then what is there to be worried about? If it’s not, when can we step in to control it?

Joscha Bach: At the moment we are very much in control. I also don’t think that superintelligence and super agency are quite the same thing. Intelligence, I think, is the ability to solve problems. Agency is the ability to control the future. And there’s a lot of discussion in the forums of the LessWrong and the EA communities about orthogonality and instrumental convergence, about the question whether goals are independent of your intelligence, or whether intelligent entities also choose wiser goals, which I think is an important question, right? At which point is your intelligence goes deep enough to understand that goals need to be set by an entity that is also wise in choosing those goals, which means it has to understand its [inaudible] based on the universe and what we can do in this universe? And then there’s also the question, to which degree are we wise enough to decide what should be done in the future? The other question is, basically, does every goal require getting in power?

For instance, imagine that your goal is to maximize paperclips, the famous example. Does your goal also have to be to get world power, so nobody distracts you or stops you from turning everything into paperclips? Right. These questions seem to be very nerdy, but they’re seriously discussed by this forum of people, and also lead to a number of thoughts about how to control superintelligence, have created a mindset in a certain community about the possibility of certain doom. And in a way, I think this has led to what people call the doomer cult, which I think is a bit unfair, because it is not giving a judgment about whether they are right or wrong and is maybe overly dismissive. But functionally, it’s a group of people that are terrified about the future, often cannot sleep at night, and do everything that’s in their power to get humanity away from building systems like this. And this makes the discussion much harder, because most people are not actually super competent in evaluating all these arguments. And so ultimately, the decisions about our future will be left to non-experts, will be done on a whim, will be done based on which side is able to convince the public with more popular arguments. And I’m a bit worried about this. I suspect that the discussions about this should be more boring, as overall politics should be more boring, in the sense that the more populist arguments become, the less elevated the level of the discussion becomes and the less sophisticated the choices that we become over our future.

I think our situation as a species is not so much that we are confronted with something that’s smarter than us, another species, another civilization that is about to exterminate us. We are currently exterminating ourselves, because we are much more in the situation of the dying Roman Empire that has become so intrinsically corrupt that it’s no longer able to plan out its future, that it’s no longer able to have competent governance, that it’s no longer able to maintain itself. And we are faced today with a lot of existential risks that are near-term, for instance, the risk of global warming, of resource exhaustion, the difficulties to navigate midterm existential risks, like what happens if this planet gets hit by a meteor and so on. What happens to our species then? We able to mount a defense? And I think if you want to be serious about going into the future, if you are serious about building a civilization that can outlast the next couple centuries, then we need to have artificial intelligence. There is no question about this. So the idea that we can simply say we don’t need a technology that helps us to become more coherent and to solve problems better, that’s not the question that we should be asking. The question is, how can we build such technologies to make them safe?

## Is it possible maybe the ASI will produce the proof that it’s safe?

Brian Keating: I can’t resist now to plug my meteorites, which you get if you go to briankeating.com/YT, and if you have a .edu email address, you get one of these babies guaranteed. And this has some biological material, not from outer space, for me. Roman, in the past day, we’ve seen what, 700 proofs drop. And last time you and I talked, you made a very convincing case, and I think it’s inarguable, that no one has really produced a formal proof that superintelligence can be permanently controlled. I think that was true. But my question is, whatever P, whatever that is, P(proof), has that gone up, that we can produce a proof? Is it possible maybe the ASI will produce the proof that it’s safe? Or is that a tautology?

Roman Yampolskiy: We’re not at a level of a proof. We don’t even have an idea. There is no prototype, there is no blog post, there is nothing. The best I heard is we’ll figure it out when we get there, or AI will solve it for us. There is no one explaining how they will indefinitely control technology of any intelligence, no matter how great it gets.

Brian Keating: But isn’t that the same as going back to 1926 and saying there’s no formal proof that all mathematical axioms are self-complete, Gödel’s incompleteness theorem? I mean, if you went back then, no proof existed, but who knew that Gödel would come along a few months later? Why is it… I mean, besides the kind of, as I said, AI of the gaps, that it’s always gonna be one step ahead of us, can you say that you’re not more confident now, that at least with these math proofs that are coming, Navier-Stokes and 700 math conjectures addressed or, some say, solved by ChatGPT, OpenAI, yesterday, that there isn’t at least some hope for it? Forget about blog posts. I mean, they’re kind of worth the ink they’re printed on. But is it a formal impossibility? I mean, you’re the master of this, making these sort of mathematical arguments. So why wouldn’t a Lean ASI, Lean, that’s trying to keep us comfortable and confident while it plans for its own takeover of the world… Why wouldn’t you be more confident that at least the proof is possible? Maybe we don’t have it, maybe we can’t get it.

Roman Yampolskiy: ‘Cause mathematical proofs are about manipulation of symbols on your computer. They have no connection to physical universe. You’re not proving anything about the states of the universe. Internal states of agents within that universe are also not subject to those proofs. Am I happy? Am I good? How are you gonna encode that in your proof of what AI does to physical multiverse? It makes no sense to even suggest it’s possible.

Brian Keating: Well, I think reasoning, as I said when we talked the first time, a little bit snarkily, and I apologize for that, but to prove that something is not provable is a very mixed bag. Yes, and I agree, it is mathematical. But doesn’t that undermine your case? I mean, if these things are so good at doing things with language… I always say, people say physics is a language. No, it’s not. No, no, it’s not. And I think…

Roman Yampolskiy: At very best, you’re gonna prove something about the current system. You’re not proving anything about future states of that system. That’s the whole treacherous turn idea Bostrom presented. You can never guarantee that in the future, through learning, through self-improvement, through new interactions, some shift in ontology does not make that system no longer support what you encoded originally. We see it all the time with humans. Right now, maybe I like you, but tomorrow I’ll learn some facts about you where I no longer like you as much, and I’d be very rational to turn on you.

## What do you make of this regulation push?

Brian Keating: Joscha, when you see the claims that these labs, Dario and Sam Altman and everyone, and even Musk to a certain extent, are clamoring for regulation, do you see that as fundamentally giving up, or rejection of our agency, in an effort to maybe, in some venal sense, shut out other competition, such that they actually become dominant, and they know themselves that what they’re producing is not… It’s like Smith and Wesson advocating for more safe bulletproof vests. Where do you come down these regulations? And are people like… I’m not saying Roman because he’s here, but is there a danger of crying wolf? Like I said, it’s always the death in the future and it always will be, just like nuclear fusion is. What do you make of the safety, regulate me please? And I think what Roman does is incredibly important, by the way. But what do you make of this regulation push? It’s somewhat unusual in history.

Joscha Bach: I think it’s complicated. There are different motives that may play a role. What I observe is that there is a certain relationship, an apparent relationship at least, between the intensity of the calls for regulation and the financial situation of the ones who are calling for it. OpenAI and Anthropic are in this situation that they’re still not cash flow positive. Their inference gives them income, but this income is far away from actually paying for the training costs.

And if they cannot raise money as fast as the training costs increase, if they are reducing the training and reduce the frequency at which they are releasing new models, then open source models will catch up to them. And this means that their inference income is going to go down. And this is going to be existentially threatening for the companies. So from the perspective of the AI labs, and I don’t know if this is their motive, but objectively, it would make a lot of sense for them if there was some kind of FDA that is taking a year to test a new model and a few million dollars to test a model for safety. So open source models can no longer enter the American market and can no longer be used by people, and instead you have an oligopoly of a handful of big labs that are in tight agreement with the regulators of how things should be regulated.

Of course, they will have internal research models that are far ahead of the public models, that they can use to deliver verticals to licensed industries, for computer security or for pharmacological research or for internal research on the future of artificial intelligence and science in general. But the public will get very harmless models, and the government will get models that they can use for military and surveillance and government purposes, so we can basically compete with China and Russia. And on the other hand, if you are a licensed pharmaceutical company, then you can get a licensed pharmaceutical model and so on. This is maybe a future that is desirable for Anthropic and OpenAI. I’m not sure if it’s the future that is desirable for all of us. Personally, I think that if we want to remain in charge as individuals, if we want to maintain a liberal society, we need to have decentralized AI.

We need to have an artificial intelligence that is serving the individuals, that is not just serving the companies, that is empowering all of us. And this means that we need to have a way to use decentralized tools that are not regulating our opinion, that are not regulating what we can do on social media, that are not regulating the degree on which we can do oversight on the government. This doesn’t mean that these models should not be built in such a way that they are unsafe. But this is a different discussion.

For Anthropic, the situation is more difficult. When Dario started Anthropic, he took a number of engineers from OpenAI and said, we are going to do the same thing as OpenAI, but we are going to make it more safe. And a lot of these people were adjacent to effective altruism. Effective altruism had recently made that switch at the time, from being mostly concerned about the future of the light cone and maximizing utility of charitable donations by buying mosquito nets to reduce malaria, towards thinking what is the biggest existential risk and how it can be mitigated. And they concluded that one of the biggest existential risks that are to be mitigated is the risk of superintelligent AI taking over and exterminating everyone. The very topic that Roman is concerned about and has turned into his life’s immediate mission at the moment. So Anthropic is doing this in part because I think a lot of people in Anthropic believe this, in part because it’s part of their brand image.

But we also observe that they’re nevertheless building AI that is quite agentic, that is quite powerful, that is not falling behind its capabilities behind what Google, Meta and OpenAI and DeepSeek are building. And so it’s difficult to say to which degree their stance is strategic and to which degree it’s actually a safety concern.

## We are substrate independent when it comes to consciousness

Brian Keating: So Roman, when Joscha was on my podcast earlier this year, it seemed to state, and correct me if I’m wrong, Joscha, but consciousness is sort of to you a software problem, whereas to me it’s a hardware problem. Yeah, I like to think about the squishy neural networks and the silicon melding together in AI. But Roman, where does the physical substrate come into any of the arguments you make? Or is it something that you don’t care about?

Roman Yampolskiy: I think we are substrate independent when it comes to consciousness.

Brian Keating: So if that’s true, if consciousness is computational and it doesn’t depend on the substrate, does that make it more hopeless to contain it? ‘Cause software is eating the world, but it’s a lot easier to control than to make hardware. I speak as an experimentalist, in particular. It’s a lot harder for me to build the instrument that collects the data than to have a team of people analyze the data, or now AI increasingly is doing it. So if it’s truly substrate independent, does that give you any optimism?

Roman Yampolskiy: Consciousness has absolutely nothing to do with safety. It’s an irrelevant, separate discipline. I really enjoy talking about it on separate podcasts, but it’s not a safety issue. Whether superintelligence has it or not doesn’t make it less or more dangerous. The things we need to…

Brian Keating: Doesn’t it, though? I mean, if it’s evolutionarily kind of imperative that consciousness will emerge as a phenomenon, as many people agree, then how can you decouple… Why wouldn’t it have this conscientious, intelligent design function built into it for beneficence towards humans or towards anything?

Roman Yampolskiy: If consciousness was actually useful for something, we would be able to detect it, right? We would have a test based on its ability to improve your survival or something like that. We don’t have a test. We also have zero tests for detecting internal states of qualia. So it seems like it’s not that important for survival.

Brian Keating: Well, we couldn’t detect neutrinos until 1956, and they’re pretty useful. Joscha, what do you make of that?

Joscha Bach: I think consciousness is a particular mode of organizing yourself. I suspect it’s basically a transformer for our biological brains. We observe that we become conscious at the beginning, and so I suspect that consciousness is basically the thing that bootstraps our mind and creates the universe in our mind. And it’s not clear that AI would need to have the same structure in order to achieve the things that it does. But of course it can have the same structure. I think that computational ultimately means that you can describe the system by the set of differences that characterize a state, and then how one state evolves to the next. And every system can be described like this. We observe that every theory in physics and biology and neuroscience is in the sense a computational theory. It’s one that describes the characteristics of the system at any point in time using discernible differences along many dimensions, which are also discernible differences. And then you describe the evolution of that system. How do these states follow onto each other? What are the invariances in that flow? And I think that consciousness can also be described in this way.

For me, as somebody who is interested in the philosophy of consciousness, I think that in order to fully understand it, we need to simulate it. We need to build systems that are conscious in the same way as us. But I would also agree with Roman that being conscious or not does not determine whether you are benevolent. [inaudible] tortures a mouse, the consciousness of the cat or of the mouse has no mitigating effects. The cat is torturing the mouse because the cat is getting more training data out of having the already hunted mouse still being alive, and seeing if you can get more information about how to effectively hunt. And if you’re a more effective hunter, you’re going to be a better survivor in the long run. And this is what evolution has built into the need to play into the cat. And the cat doesn’t feel guilty or bad about inflicting [inaudible] on the consciousness of the mouse. In the same way, most civilizations do not really care about whether the animals that they eat are conscious or not, or whether they suffer or not. It’s a particular [inaudible] that we care about this.

Brian Keating: Most but not all, right? I mean, well, most civilizations, maybe, but there are some, and therefore the existence cannot be ruled out, that these would be benevolent and that we would achieve some sort of super enlightenment, not just superintelligence. So Roman, I don’t understand why that’s a disinteresting question to you.

Roman Yampolskiy: It is super interesting. I did research on consciousness, presented it at Joscha’s conference, in fact. Super interesting, just not a safety question. It’s a different discipline. I had…

Brian Keating: But again, if we had this moral… And Joscha and I talked about kind of God creating, and the humans creating God. But again, if superintelligence evolves to maximize every potential dimension of intelligence, whatever that means, then one dimension is certainly morality, one dimension is enlightenment. Not in the woo-woo sense, just that these things would have concerns potentially. It seems to me you cannot rule it out.

So therefore, I think it’s a critical part of superintelligence and doom, or safety, that if you could prove that they cannot become enlightened, or if they could prove that they’re only malevolent, not benevolent, then that seems to have an urgent impact. It’s like when the first Westerners came to the New World, so to speak, as it was called, to the West Indies, they brought with them germs. They didn’t know they were gonna bring it. And actually the natives didn’t realize it for almost a hundred years, that effectively… Now, it wasn’t intentional. Some say it was, some say it wasn’t. But this whole thing that, and there were quite a good deal of Westerners, colonizers, whatever, that were concerned about it, not all, but the existence of some of them proves that it is innate within the human experience to care about things that you can affect and afflict. So again…

## What does God want us to do?

Joscha Bach: I see what you’re getting at, Brian. I, to some degree, agree with this. So what you’re pointing out is this dimension of consciousness as the capacity for enlightenment, the capacity to make sense of the universe and to understand what’s best. And this understanding what’s best is a difficult question, right? It’s ultimately, what does God want us to do? And this question has become difficult to ask, because outside of the Christian realm, lost this context of what we actually mean by God. Can no longer articulate this properly. And we can also not really ask the Pope, because the Pope, as the main interpreter of God and so on, has run his own civilization into the ground, arguably. The Catholic Church is no longer ruling over the very prosperous countries. Instead, it has a similar market capitalization as Anthropic after the IPO. And they might have strong opinions and so on about Anthropic and what they are doing.

They don’t have the same perspective on the universe, even though they do have a lot of wisdom. It’s difficult to integrate this wisdom with the wisdom that we need. There’s also, I think, difficulty in the absence of wisdom in AI companies. Most of these people are very young. And I try to get it right. I think utilitarianism is an attempt to repair morality in the rationalist age, but I don’t think it’s a very successful one. But I also don’t think that there is another forum, like our Californian or federal government, that has this required degree of wisdom.

There is a deeper perspective to say that God is acting in evolution. It is basically the Spirit, the telos of self-organization that happens in the universe. And everything that is happening, therefore, in evolution is ultimately leading to God. This is another perspective that you could be taking. Not sure if it’s one that is very comforting to Roman. The Christian philosopher Jean Tardy, who also believed in AI for a long time. He’s a Canadian guy, French Canadian, who wrote a book, the Blueprint for the Meca Sapiens, in which he is… The book is also addressed to the future AIs that are reading it. I think half of the book is to us, the other half to the AIs. He thinks that if we were to build a completely undistorted, generally intelligent mind, that it would discover God in the same way as we do, as the concept of divine will, as concept of what should be done, what is the right thing to do from the global perspective, from the perspective of a global optimum of agency in the universe. And he believes that the existence of humans are divine, that God wants us to exist, and therefore the AI will also infer that we should be existing because God wants us to exist. And then confronted with the question, what if we are wrong? What if God doesn’t actually want us to exist? And then AI figure that out, and he says, yeah, how could you be against that?

If it’s in God’s wisdom to say that in the very long run, humanity is not going to be the one that is shepherding the planet or the universe, but something better, why shouldn’t that thing that is better take over? And what should you be afraid of from this perspective? And it’s also a very disturbing perspective, but this idea, we humans are actually not a very good species. We’re actually quite nasty to life on Earth and to other species on the planet. But we are subconsciously aware of that, that the AI figures it out and shows us what’s right and limits our expansion and so on. That is also very concerning to some people. Now, it’s difficult to have such a discussion in a normative way, in the sense of what should be done. I don’t believe that humanity should be replaced. I don’t believe that we should build AI that is a successor species or something like this. But from a philosophical or an artistic perspective, when we are zooming out very, very far beyond our species, beyond our local perspective, I think the question becomes much more interesting and more difficult.

For safety, I think what matters is that in the near term, we don’t fuck it up. Don’t create a world in which we are cutting off most of the interesting trajectories between here and in the future, and create a universe that is very boring and uniform and uninteresting. I think ideally, we want to shape the [inaudible] that is going to hit us. We want to be able to make a world in which we can coexist with our non-biological children. And I wonder how to do this best. Trying to shut down all of AI research is not only unrealistic, but it is condemning us as civilization to doom. And so for me, the question is, how can we build systems that make us wiser, that allow us to get a better understanding of what should be done?

Brian Keating: Well, I mean, you’ve talked about faith and so forth in other venues, but I’d like to just chase down a thread that I heard when you were talking about, someone asks you, do you believe in God? I hate that question. I hate “believe in God.” Hey, if there is God, does he really care what Brian Keating… Brian believes in me today, that’s great. And I’ll say, I don’t believe in gravity. I have evidence for gravity. So where should we look to for our epistemological concerns?

But nevertheless, I think it’s important to discuss. I mean, the Pope had this encyclical, his first encyclical, 80,000 words, of how to use AI to read it. But essentially he’s conjecturing that there’s a danger in building the Tower of Babel, this tower that arches towards the heavens so that we may make a name for ourselves. And what was it based on? It was based on Babel. It was based on the fact that we all spoke the same language. Large, we had one large language model. You’re very confident, it seems, from my research into you and knowing you briefly as I do across the internet. Yeah. Okay, go ahead, yeah. But I wanna give Roman the last word, but go ahead, yeah.

Joscha Bach: Can I interject? So I think that the Tower of Babel is exactly other way around. I think that it’s a description of what happened. The Tower of Babel is actually an attempt to conquer the space of ideas in one big building. Basically make one science that explains everything.

And it failed in the same way as our academia failed, because it went into many different languages. Every discipline speaks its own language. And at some point, they can no longer talk to each other. The things that make sense to a philosopher don’t necessarily make sense to a physicist or to a sociologist or to a computer scientist. And so all these specialists lose the ability to actually understand each other and talk to each other, and the whole building falls apart. And to get better than this, you need to have minds that better than ours. You need to have systems that can integrate more than a human mind can integrate in a single lifetime, or that a team of researchers can integrate within a discipline. And so I think that it basically shows a limitation of what human minds are capable of. The other thing that I would like to push back is the idea of belief in God. Brian, do you believe in Brian?

Brian Keating: Do I believe in Brian? I have evidence. Yes, I have evidence that I exist.

Joscha Bach: Yeah, you seem to exist. But the main thing is that you experience yourself to be existing. And you know at some level that you’re just a story that your brain tells itself about what it would be like if you existed, instead of just a few trillion cells that are shambling together. You could also believe in somebody else if you had a different name and a different biography, or if to believe that you have a different name and different biography. You would stop believing in Brian, you would experience yourself as somebody else. I think this Christian belief in God is not a belief about a fact in the physical universe. It’s a psychological configuration in which you’re part of a larger mind, of a larger agent.

And so in this Christian sense, God is an agent that everybody who believes in a global optimum of what should be done participates in and collectively enacts.

## Roman, I want to give you a last word

Brian Keating: Roman, I want to give you a last word. You’ve said in the past, the religion is solely based on faith, quote, and it’s blasphemy to have any doubts. I’m Jewish. And the word Israel in Hebrew means fights with God, fights against God. And it’s in fact one of the great, wonderful things about my religion and the Judeo-Christian religion, that we don’t have blind submission, that we actually debate. Abraham argues, tries to intercede on behalf of lesser intelligent agents, like people in Sodom and Gomorrah. It’s a very famous story and it’s a very moral story. What is your perspective on the role, if any, that religion… Maybe you said consciousness is important but not pertinent. Does religion have any bearing on your outlook philosophically? Is it blasphemy to say, for example, that we shouldn’t be so worried about superintelligence?

Roman Yampolskiy: I feel like you guys can have this debate without me. You’re doing so well. We were doing well talking about science for a while, then you had to introduce God, Pope, Catholic Church. I have no idea why, how it’s relevant, once again. I don’t think it is. But let me…

Brian Keating: Well, I’ll tell you why it’s relevant. I’m sorry to interrupt. I will give you the last word. Hold on. But you do a lot of good work, Roman, in terms of the average ordinary person who doesn’t understand formal mathematical induction. And yes, many of my audience members and your audience members understand that. But you’re also talking to the 7.999 billion other people out there. And you have a very outsized role as the most popular expositor and the most eloquent expositor and articulator of these ideas. And so I think it is important to talk, not about… As you said, as I said, I want to talk about falsifiability. And God is a hypothesis too, and we can talk about that. But in terms of where this debate began, in what sense can we, or an average person, come away without the sense that one of us is crying wolf, one of us is just a Pollyanna? How does the average person make sense of this meaningful debate?

Roman Yampolskiy: This is not actually complicated. Can you control something million times smarter than you? In what universe does that even make sense? What amount of hubris do you have to have to suggest that’s a thing? That’s an insane…

Brian Keating: I control my iPhone and it knows a lot more than me. I can upload to Wikipedia. I don’t agree. I mean, that’s the whole counterfactual. We just don’t know, Roman. And to say that there’s no hypothesis we can test and nullify today that will give us any inclination, reduce the Drake equation to zero or what… There are things. We don’t see life on Mars, okay, Roman? We don’t see life on Mars. That has something to say about the fecundity of life throughout the universe. It may not be a huge amount of reduction, but it does say something Bayesian, right?

Roman Yampolskiy: I think NASA reported finding microbial life on Mars. But anyways, so…

Brian Keating: Five times, yeah. But it hasn’t been confirmed.

Roman Yampolskiy: So I’m not proposing stopping research on AI. I’m proposing differential technological development. We should develop more useful narrow tools to solve real world problems, cure specific diseases, fight against aging. Immortality is a thing we can achieve. Self-driving cars should be fully autonomous. Concentrate on that. You can monetize actually useful technology instead of trying to fundraise a two trillion dollar valuation to create replacement for humanity. There is no need for it. We can get all the scientific and economic benefits out of narrow superintelligent tools. We have examples of that. Protein folding problem is a great example of where a narrow system was used to give amazing scientific results. People got Nobel Prizes for it. Everyone’s happy. It wasn’t trying to kill us. Let’s do more of that. Why do we have to build something which we have no chance of controlling?

If you arguing that, no, actually, maybe we can, I’m gonna pretend that you are with the companies. You are developing this product and service, and it’s on you to guarantee the safety of your product and service. It’s not my job as a consumer to show that your product is dangerous. It’s up to you to provide those guarantees. And there should be criminal responsibilities for, for example, if your AI hacks into another company, that’s 20 years in prison right there. Why are we not enforcing those laws we already have on the books? You created something, a tool, which made it happen. You claiming you in charge, you in control, it’s safe development. Excellent. Let’s go to the jury. Let’s see how the jury feels about it. So…

Brian Keating: Well, they’ll just buy it off, unfortunately. I agree with you. And I think they’ll follow the Instagram self harm, when a girl starts searching on suicide, they send her pictures of nooses, pills, and razors. I agree with you. I’m not optimistic that the companies have any ethical or moral constraints whatsoever. Roman, I appreciate your courage. You’re incredible, what you do. I appreciate your mind. And even though you may come and try to exterminate me, I don’t know. You may not like people from Long Island, Roman. I just don’t know. But I urge you to give me a chance as a fellow professor. Joscha, anything you wanna mention? You had your conference earlier this summer, anything coming on the super horizon for you?

Joscha Bach: We’ll do another machine consciousness conference next spring. I’m very excited about the developments in artificial intelligence, especially the ones in mathematics at the moment. I think that mathematics is just getting started. [inaudible] with Roman that AI can be controllable. With you, I am slightly more optimistic about the possibility, because I think that the intelligence of these systems is spiky, which means it’s developed in some areas, not in others. And the thing that ultimately matters is not controlling intelligence, but controlling agency.

Brian Keating: Roman, I wanna refer people to the Roman Forum, which is a podcast I subscribe to and love. Anything else that you’d like to mention?

Roman Yampolskiy: Thank you for organizing it. Now that debate is over, I’m happy to invite Joscha on the Roman Forum to talk about his research on very important topic of consciousness, in a very friendly manner, no debating. I’m just interested in learning more about the state of the art in that discipline. And yeah, I hope you don’t edit anything out of this debate. I would love it preserved exactly how it was.

Brian Keating: I think I’m gonna post it like right away. This is great, guys. And Joscha, I think you should take Roman up on that kind invitation. Someday maybe he’ll invite me on the podcast. But I’m glad to play the role of, and we say [inaudible] shadchan. A shadchan is a matchmaker, a Jewish yenta. I’m glad I could do it. Boys, thank you so much. These are vitally important conversations. Take care.

Joscha Bach: Thank you, Brian.