Transcript:

# Robert Wright: AI Alignment Is a Moral Problem, Not a Technical One

**Robert Wright:** You feel sure everything’s going to work out fine. I don’t really understand how you can be confident of that unless you have something like religious faith in technological progress.

**Brian Keating:** Natural selection is blind.

**Robert Wright:** The mutations in natural selection are blind, but the selection criteria are not. One of the grimmer readings on the Fermi paradox is the reason you don’t see many civilizations that have gotten to past the point we are is that typically when they get to this point where they have the capacity to blow themselves up, they blow themselves up. I think we face the kind of test that God would give.

**Brian Keating:** My guest today is Robert Wright, one of the OGs of AI. He was interviewing Geoffrey Hinton about neural networks in 1983, 2 years before Sam Altman was even born. And that was long before anyone was terrified of them. Robert thinks that AI is putting humanity through a test and that passing it takes something no lab can ship, a moral advance.

**Robert Wright:** The Tower of Babel story is a truly cautionary tale and it’s a tale of kind of technology gone awry, technology wanting to compete with God, to fight with God, to make a name for oneself. And you see a lot of that.

## The kind of test that God would give

**Brian Keating:** Where do you come down in this, that the fundamental aspect of the God test is, at least as I read it, but now I’m talking to the author, is: are we gods but for the wisdom? Are we able to pass this test? And maybe you should define the test, where we can prove maybe to ourselves in the absence of a God that we’re worthy of god-like technology.

**Robert Wright:** The title has at least two meanings. One is, as you know, some people think we’re building a superintelligence that will ultimately in some sense maybe qualify for a term like God. It’ll be all powerful and maybe live in harmony with it and, according to some doomers, maybe get annihilated by it. But in any event we’ll be talking about a kind of, relative to us, an all powerful and omniscient in some sense being. And so the one test is, if that’s going to happen, can you build a good god, a god that is compatible with human values and would safeguard human flourishing and so on. There’s that. And although I’m not sure we’re building a superintelligence, I certainly don’t rule it out. And I explain why in the book, but there’s another sense of the God test that in a way is related to the Tower of Babel, because the Tower of Babel is a story of hubris. Who are we to build a god? On the other hand, the doomers say even if we don’t try, it’s taking shape. We’re going to have to try to stop it to keep it from taking shape.

But in any event, the Tower of Babel is also a story of how humanity became divided. How they came to speak different languages and became in a certain sense kind of inherently tribalistic, at least at a linguistic level. And one argument I make in the book is that if we’re going to do a good job of navigating the AI revolution, and, leaving aside the question of whether it culminates in superintelligence, if we’re going to reconcile it with continued human flourishing and welfare, I do think we’re going to have to approach it as a true global community. I think there are a lot of policy issues that can only be handled at an international level, and there’s more and more recognition of that only just in recent months. We’ve seen that. I think the end of the story of the Tower of Babel needs to be that we all kind of get back together and navigate this challenge collectively, that is to say human beings. And so in that sense the God test is to overcome our tribal differences, to get better, to understand the perspectives on people on the other side of walls and so on, and like quit having wars. That would require something that I think would qualify as a moral advance, and I think it’s going to be necessary if we’re going to survive this in good shape.

So in a way that is a test we’re being subjected to, and that is the kind of test that God would give. At the end of the book I quote the Hebrew Bible, or as we used to say in Southern Baptist Church, the Old Testament: salvation is at hand. And as I say, there are a lot of stories in a lot of religious traditions where salvation is possible, but you are going to in some sense have to pass some test. It may vary by religion. In the Christian tradition, salvation, the test has to do with accepting Jesus as the savior who died for your sins. But very often in one sense or another, it’s a moral test. And for salvation to happen, and by the way, I would also say that salvation is often salvation at the societal level, including in the Hebrew Bible. Again, Christian salvation is often construed more as an individual thing. But in many traditions, it is a question of kind of saving the social system. And I’m saying I think we face, leaving aside the question of whether there is a divine purpose unfolding, I think we face the kind of test that God would give.

## Are we making these things into gods?

**Brian Keating:** There’s so much to unpack there. The classic, in the Torah, the Old Testament, the classic paradigm for a test. But the first test that’s really actually called a test or thought of as a test comes to Abraham. He originally has a different name, Abram. And then his name gets changed to Abraham, which means father of nations, and he goes from being an individual to a collective, as you just said. And the kind of crucible, literally, by which that’s thrust upon him is a series of tests. And in the Torah tradition, there’s 10 tests. And it begins with his father throwing him in an oven for breaking the idols that he worshiped. And that’s where I want to start.

There is so much with these new models and so much personification. And you probably saw this Dwarkesh Patel article that got tens of millions of views on X and elsewhere, where he talks about civilizations living and dying and rising and falling. It really seemed so hyperbolic to me. And in my conception, I’ve talked to all the doomers and there’s way more doomers than there are boomers, as you know. The only public boomer that I really know is Yann LeCun. I had him on and he’s extremely, as you know, when you quote him in the book, he’s extremely technical in his optimism. He’s not kind of denying the sort of societal upheaval, but he’s saying we’re not there technically, is the impression that I got, and we probably will never get there. And I want to talk about that.

But starting with the test of kind of, is this an idol? Are we worshiping these things? When you hear these laboratories: we had to put it in a sandbox, we had to stovepipe it, we had to cut it off because it was getting too dangerous and it was escaping and it was creating civilizations, according to Dwarkesh. What do you make of this? We live, whether or not we personally individually are or not, we live in a secular age, and the secular age has never been more enhanced by technology than it is now. So do you make anything of these comparisons between both the idolatry of AI and the eschatology of AI? In other words, the doomerism is intimately connected, I think, with the idolatry. So what do you think? Are we making these things into gods to worship or to propitiate?

**Robert Wright:** Interesting question. I mean, the Dwarkesh thing was about a bunch of agents in this famous OpenAI Hugging Face incident, and he was comparing generations of agents, I guess, to civilizations. I’m not too bothered by that kind of anthropomorphism because, as I argue in the book, I mean, we are kind of trying to, and in some cases doing this without trying, but to inculcate in AI human tendencies. There are a lot of ways we want them to be like people. We want them to be pretty autonomous when they seek goals. We want them to do this, want them to do that. And I have a separate argument about what seems to be going on inside them and how like the human mind that may be. So I’m not terribly bothered by that kind of language.

But as for the worship thing, I think you could probably accuse either extreme of that, which is to say the doomers or the accelerationists. I mean, you’ve articulated the case for looking at certain kinds of doomers as engaging in idolatry, although some of them would argue that they’re certainly not religious. I mean, Eliezer Yudkowsky, as you may know, I don’t know how much you know about him.

**Brian Keating:** I had his co-author Nate Soares.

**Robert Wright:** He’s kind of the doomer chief. He turned his back on religion. I believe he was raised by Orthodox Jewish parents, but his religion, you might say, became science fiction. It remains his religion even. I mean, some might make that accusation, that he’s uncritically accepting certain sci-fi narratives. Now, in the book, I argue that they’re harder to dismiss than you might imagine, and I’m not dismissive of them.

## Religious faith in technological progress

**Robert Wright:** But as for the other side, the kind of accelerationists, it does seem to me that if you, A, believe that this technology is powerful, as many of them do, and is going to get much more powerful and is doing that right now, and you feel sure everything’s going to work out fine, I don’t really understand how you can be confident of that unless you have something like religious faith in technological progress. I started thinking this a long time ago when I first learned about the term singularity, right? The idea that there’s going to be this acceleration of technology, which, yeah, seems to be the case, and that it’s going to bring some transformation that we can’t predict. Look, you’re a physicist, right? So singularity in the rigorous sense denotes such a fundamental change that your old kind of analytical tools lose their validity. And in the case of a black hole, you’ve got an event horizon. So the light itself can’t escape. It’s completely opaque. And that is, I think, part of the metaphor of the singularity. And I just thought, well, if you don’t know what’s on the other side, why are you so sure it’s going to be great? And I’ve always thought that singularity enthusiasts, uncritical singularity enthusiasts, must have something like religious faith, because I don’t understand the empirical basis for it.

**Brian Keating:** But have you really met many boomers, in the sense of people that are almost uncritically, again, as I said, Yann LeCun is the foremost one. I’ve literally had on 10 times as many, maybe a hundred times as many people that believe in the doom scenario, ranging from Nate Soares, Eliezer’s co-author, to, I haven’t had Geoffrey Hinton on, but I intend to, and I know you’ve talked with him extensively. But my contention is that if you were going to go short on something, if there was a betting Kalshi market for doom versus boom. I mean, even people that are extremely pro. I released today my podcast with Emad Mostaque, who created Stable Diffusion, and Roman Yampolskiy, who created the term AI safety. Both of them were like P(doom) equals 100%, effectively.

And I said, in science, and actually could turn this back to the Torah, if in the Torah, in the days of what’s called the Sanhedrin, if 100% of the jurors voted to convict somebody, they were set free, or they were spared the death penalty. In other words, there’s almost no way to falsify this. So I titled my debate, it was supposed to be a rigorous debate between an AI accelerationist, an optimist, Emad, who believes in open weights for all, and Roman, who believes in pausing everything, essentially, that could lead to ASI. And we’ll get there, because I think that’s intimately connected with the nature of God. And they just agreed violently, and I had to be the one that said, “Look, I think you guys are completely off here.” And if there’s nothing that can falsify, in cosmology, if you can’t falsify a hypothesis, I’m sorry, it might be an interesting opinion you could talk about at cocktail parties, but it’s not a legitimate part of the scientific corpus. Now, that’s just a necessary component. It’s not sufficient to say something’s false. I mean, astrology is falsifiable, as you know, but it doesn’t make it science. There’s nothing that would turn P(doom) down, and this is what I asked them, and I want to ask you. What would you do, having interviewed and written so extensively and authoritatively, as a foremost public intellectual? How would you view the question of: can you reduce P(doom)? If you can’t, it seems to be a useless metric.

**Robert Wright:** I think, I mean, first of all, you’re right. I think that the doomers outweigh the accelerationists, pure accelerationists, and singularity enthusiasts. I also think that maybe some of the singularity enthusiasts are kind of recalibrating. There are a couple of them on Twitter. One of them is, what is it? Based Beff Jezos. It’s a play on Jeff Bezos, whatever. And then there’s a guy named Bayeslord, B-A-Y-E-S, as in the [inaudible]. I recently had an interaction with him on Twitter where he said he’s not an accelerationist. So I think maybe even some of the relatively few pure accelerationists are recalibrating, unless I misunderstood where he was a couple of years ago. But I would say in addition to those, and interestingly, by the way, Yudkowsky went from one extreme to the other. He was a singularity enthusiast.

## The gospel about neural networks

**Brian Keating:** So was Hinton, right? Not singularity, but obviously AI enthusiast. I mean, you quote your interview from ’83 and he was—

**Robert Wright:** Yeah. I mean, so I talked to him longer ago than I care to admit. I’m old enough to have talked to— [laughter] I talked to him in 1983 when I was writing a piece on AI. He was enthusiastic about the intellectual paradigm. I mean, I remember the reason I called him, and I just kind of was calling around, as one did in those days before email. You used the phone to kind of just get the lay of the land before I started writing. It wasn’t really an interview for purposes of quotation, but I remember the reason I called him is because somebody I had talked to, I don’t remember who, said, “Oh, if you want to hear the gospel about neural networks, you need to talk to Geoff Hinton.” So, he was an evangelizer of the intellectual paradigm, but I don’t recall him saying anything about how well this would work out socially.

Now, I do think it’s true that he has only later come to worry so profoundly about the social consequences and become basically a doomer. He wasn’t a doomer then, but I honestly don’t know that he was thinking about it. It all seemed so abstract back then, right? Like even people who believed it would happen, I don’t think were imagining the implications of it actually happening. So, you’re right. Now, there are also some people who aren’t quite singularity enthusiasts, and you might say all they’re doing is talking their book. I’m thinking of some VCs in Silicon Valley.

**Brian Keating:** Yeah, of course.

**Robert Wright:** I’m thinking of guys on the All-In podcast who kind of advised the president on AI policy.

**Brian Keating:** Yeah.

**Robert Wright:** Well, David Sacks did until very recently and probably still has some conversations with him. So in other words, they are libertarians in the economic sense. But you’re right, there are more doomers, and I honestly think the conversation has shifted in the direction of doom by virtue of some things that have happened that have worried people. That’s kind of where we are. But I think lots of people can be accused of having some degree of religious conviction hiding within their worldview, probably, including me.

## What is the goal of the God test?

**Brian Keating:** I do say sometimes, and I’ve hosted Richard Dawkins in person and will again this coming October, but I say there’s no one as kind of obsessed with God as an atheist, at least in some sense of the word. But that’s okay. I mean, people get locked into this kind of epistemic closure, this need to be defined one side or another, and I think that’s the same with the doom versus boom.

But in the test, just the last thing I’ll say, breaking out my rudimentary fifth grade Hebrew that I never learned. The classic test is of course the binding of Isaac, which is the only place where the Old Testament uses the word to test when it’s with respect to a human being. And typically we think of a test, it’s transformation. I give my students a test, pretty soon, and they’ll be showing up for a midterm. And in that sense, there’s supposed to be something transformational, preparatory. And in the Torah, it’s sort of this weird event where Abraham’s commanded to take his son, his only son, and he’s like, “Well, I have two sons. Ishmael is also.” “And then the one that you love, Isaac.” And then he brings him up to Moriah, which is where the Temple Mount is. And he’s about to sacrifice him. And then an angel intervenes and kind of doesn’t let him go through with it, but he was about to go through with it. And nothing obvious happens. It’s clear, if you read the story, that that led to the divorce and death of Abraham and Sarah. Although my rabbi doesn’t like that interpretation, but it seems very clear to me that that’s what happened. And so it’s almost weird. There’s no transformation. There’s no upgrade. There’s no real kind of hero’s arc, although he is obviously a hero. Next chapter, Sarah’s dead and he’s finding a place to bury her.

In that sense, is this test a sort of binary test, employing the God test to see if we have the capacity to enter the noosphere and so forth, and form this global cooperative in order to succeed or thrive in it? What is the ultimate goal of the God test? Is it to transform? Oftentimes I tell my students the goal of a test is the preparation for the test. That’s the value in it. What is the goal of the God test, if any?

**Robert Wright:** As far as Abraham, I see it as a test of devotion to God. In other words, unswerving devotion: I’ll do anything. And that would have been thought of as a moral test, because certainly devotion to God was thought of as part of having moral fiber. Glad you threw out the word noosphere. I don’t suppose during your Catholic phase you came across the writings of Teilhard.

**Brian Keating:** Five or 10 years ago, I was invited to the Institute of Noetic Sciences to give a talk. And the less I say about that, the better, because I have friends that are involved with it. But suffice it to say I was not impressed by, at least, the empiricism that was on display there, and I do want to talk to you about that.

**Robert Wright:** So anyway, noosphere, a global mind. The term was coined by Pierre Teilhard de Chardin, this Jesuit theologian, but also paleontologist, a scientist, and he saw evolution moving toward ultimately this kind of global brain that he could see taking shape. The term was coined 103 years ago by him. So pretty prescient, you might say. That’s certainly before the internet. He kind of saw that kind of thing happening. Saw it as the unfolding of divine purpose and was optimistic that it would entail a kind of moral evolution of humankind that would involve the breakdown of tribal barriers. So that’s why I spend some time on him in the book. I do disassociate myself from his view of how evolution works. I’m just more of a straightforward Darwinian.

## We’re going to have to overcome some of the barriers

**Robert Wright:** As for what kind of test, I mean, I think it’s a case where I believe our pursuit of self-interest compels us to make a moral advance. And I don’t think you have to buy the hardcore doomer scenario to think that things will go badly if we don’t confront the technology as a global community. I think that before we get to that point, the impact of the technology is going to amount to a kind of social earthquake, and we will need— So first of all, it wouldn’t hurt to slow it down, and that alone is something that you can’t really do without international coordination as a practical matter, because if you do, the—in America, if you try, the American AI companies will say, “No, we have to beat China. You can’t do that.” So right away you need a little more in the way of international concord than we’ve had.

And then I also think if you start addressing the specific governance issues, like what if someone uses it to make a bioweapon? And we’re getting there. If you get past the safeguards in Fable, you’re dealing with Mythos. And by Anthropic’s own account, Mythos is not only a super hacker, but something that could, the Anthropic safety report says that it could help a, quote, “well-resourced group,” meaning something less than a state-level actor, not just create a bioweapon, but a novel bioweapon. Okay, design it to be more contagious than COVID, more deadly than COVID, a longer latency period, and so on. We’re getting pretty close to that. If you look at any of these things, especially the bioweapons thing, but various other possibly bad consequences of AI, national policy alone is not going to keep you safe, because we’re talking about threats that ultimately cross borders. I mean, COVID taught us that pandemics do that, and some of the other downsides of AI do that.

So, I think if we just think in an enlightened way about just staying safe, we’re going to have to overcome some of the barriers that have historically divided nations and that have always divided some nations. We tried after each global catastrophe, World War I, World War II. We try, right? League of Nations, United Nations. We say, like, look, catastrophe this big, it would be nice to avoid. We try. We fail. So, we’ve never really gotten there. I think to get there will require not just political progress but moral progress. It’s like any test, right? When you give a test to a student, you equate their individual self-interest with reaping the benefits of, as you say, the preparation for the test. That’s the whole point, right? And you want to say, look, if you want a good grade, if you want to get into law school, I’m afraid you’re going to have to learn this stuff about the Iliad. Sorry. And so you’ve equated their self-interest with enlightenment in a broader sense than they might ordinarily think of their self-interest as entailing. And I kind of think that’s where we are.

## What are they actually talking about?

**Brian Keating:** Well, again, when we have this framework that’s proposed by people like [inaudible] and others, for me to take it seriously, especially when it kind of infringes upon aspects of astrophysics and astrobiology, consciousness, etc., which is a whole other book. But this notion of what’s the purpose in— I’m a pilot. I’m a private pilot. I fly little planes around Southern California. And our flight instructors tell us, in aviation, if you’re not careful, Mother Nature will give you the test before she gives you the lesson, or your airplane will. So the consequences of that are obviously meant to scare you into taking it very seriously. I mean, here we’re kind of going with reckless abandon.

And again, I’m an AI optimist in the sense that my P(doom) is very close to zero, actually. And I think that can be justified in many ways. But in one sense, because of the success of the current architecture of LLMs. We have this transformer, GPT, connected to a GPU, and those were designed, as you point out in the book, or you may remember, you’re a little older than me, but you may remember these things were created so I could frag my friend in Doom or Duke Nukem, or nowadays World of Warcraft or Call of Duty, right? So if I could frag him three milliseconds faster than he could get me, it makes me happy and I want to buy more Nvidia chips, right? So, these weren’t created to solve physics problems. They can imitate. They can simulate. They’re phenomenal at doing inference and all sorts of things, but they’re language models. And physics is, yes, it’s a language, but it’s not really a language. Mathematics has a lexicon. It has syntax. It has semantics, but it’s not a language like you and I are communicating with. And these things are obviously better at language-related things, and things that can be codified literally into codons, as you said, in biological context, which could make them very dangerous in many scenarios. But to say that these things, which were divine— designed, I said divine, but Freudian slip, designed, if you will, to do something very different, it’s not clear at all that they’ll be super powerful and turn that superpower into wanting to annihilate us.

Is that the test, kind of, you’re going to crash the airplane if you don’t take the lesson seriously? Is that Mother Nature giving us the test before the lesson? What is the ultimate kind of consequence of this? A bioweapon is one thing. It’s pretty hard to kill a lot of people, even with a bioweapon. They tend to kill the people that create the bio. I just had Annie Jacobsen on twice, down here in person in San Diego, for her new book about this. And this involves gain of function, and she did actually convince Sam Altman to remove certain attributes from ChatGPT that were dangerous potentially for bioweapons defense. You can read about that in the book. But tell me, what is the scenario? I mean, people talk about P(doom) and there’s going to be doom, but what are they actually talking about?

**Brian Keating:** Before he gets into what doom would actually look like, Bob takes a detour that matters. How these models learn what a word means without ever being told is one of the deepest mysteries around AI.

## Predict the next bunch of gibberish

**Robert Wright:** Well, first of all, as for the power of the technology and the future power of the technology, I spend the first few chapters of the book trying to, first of all, just kind of explain what we know about how large language models work and how they’re created, and really emphasizing something that I didn’t understand when I interviewed Geoffrey Hinton in 1983. I mean, I did describe his approach to AI, the neural network approach, as this maverick, almost fringe movement, which it was at the time. But when I went back and read the piece I wrote, I realized I really did not understand the potential power of it. One way to get that across is, in that piece I describe a neural network that actually was proposed by one of his collaborators, but it was really just a terrible [laughter] example of what neural networks could actually be, because this was never built or anything. The collaborator was a psychologist, I think, and he was describing how a neural network might be kind of like a brain and might work. And so in his model, each node represented the specific sense of a word, the particular meaning of the word. And in that model, the assumption was that whoever built the model would have to set up the representation of the meaning. You have to have a human who understood the meaning of the word, who would go in and say, “Okay, this node is going to mean this sense of the word ball, a sphere, and this node is going to represent this sense of the word ball, a dance.” That was what I had in mind as I described this thing.

But it turns out that with the modern large language models, they just give them this task of— they start out, this is the first phase of training, as you know, it’s kind of predict the next word. And so to the machine you’re just giving it a sequence of symbols that are gibberish to it and say, now predict the next bunch of gibberish, because we’re not going to tell you what they mean. We never tell it what they mean. But it turns out that what these neural networks kind of in effect force the models to do in the course of getting good at prediction is set up a system for representing the meaning of words. And in that sense, it’s actually a very high-dimensional vector space, such that a word like shoe is closer to a word like boot than either is to a word like tree. And each dimension represents a feature. Like, clothing might be a feature associated with both boot and shoe, not tree, whatever. And it’s set up so that the machines, just through the patterns in the words, they actually construct implicitly this whole vector space that becomes a semantic space. And this is the magic of the technology.

And I argue, some people agree with me, some people are a little reluctant to put it this way, but I argue we should think of it as kind of a recapitulation, in a certain sense, of certain parts of human evolution. It isn’t just learning. It’s not just learning the way a child learns, because the mind isn’t a blank slate, the human mind, if you ask me. I mean, evolution built into us a certain amount of cognitive hardware, including some that helps us learn language and helps us represent the meaning of words, I would say. And I think one thing that happens during the training of a large language model is it builds, it kind of reverse engineers from language, specific functionality that may not work exactly like the comparable functionality in humans. Like, we don’t know how the human brain represents the meaning of words, but we know it must. And so that function is reverse engineered by the machine, and in principle that can work across the board. It’s like, to teach a car to drive, you don’t have to know what a human brain does when it drives. You just give it all this data, the incoming visual data and what humans have done in response, the outflowing data, like steering wheel moves to the right, and it constructs a kind of brain that does the job.

**Brian Keating:** Right. And if it were purely driven by paperclip-like dynamics, it would drive on the sidewalk and mow down a thousand people to get there 10 seconds faster. But my Tesla doesn’t do that. But I want to push back, with respect. You say in the book that LLM training is, quote, “at least as much natural selection as it is learning.” And I think there’s some truth to that, but natural selection is blind. And that’s the reason why the laryngeal nerve in a giraffe is 15 ft worth of routing chaos, because evolution can’t plan ahead. And I guess, does the noosphere, does the global brain, get shaped by an evolutionary process? Because we’re going to repeat some of the same stupidity that we’ve done on Earth, if that’s the case.

**Robert Wright:** Well, the mutations in natural selection are blind, but the selection criteria are not, right? Seeing things has value to an animal that’s trying to survive. So, mutations related to vision that aren’t conducive to clear vision don’t last, whereas mutations that do, do. Now, as it happens with large language models, and Hinton was involved in developing this, the, quote, mutations are not random. It’s almost more like Lamarckian evolution, and that speeds up the whole thing, the training thing. I won’t go into that in great depth. Evolution, it’s in some ways not blind. I mean, functionality gets, quote, designed by natural selection. If you want to insist we keep the word designed in quotes, fine, so that we’re not personifying natural selection, whatever. But functional engineering takes place via natural selection. And I’m just arguing that some of the same functions in the human mind, cognitive and perceptual functions that evolved by natural selection, I think get reverse engineered during the training of a large language model.

And then of course you have other modalities, vision and so on. And those truly multimodal AIs can develop what is sometimes called a world model. In other words, the sense we all have as we go through the world. We’re all intuitive physicists, right? You don’t have to understand Newton’s law of gravity to just walk around expecting dropped things to fall, right? You understand that about the world. And you can engineer all that. And I think the progress speaks for itself over the last three years, honestly. I mean, it’s true that math is a different kind of language, and you would not expect great mathematical ability to develop from just first-order pre-training of a large language model, and it didn’t. But they have developed supplementary techniques that improve things along these various dimensions, ranging from science to math.

## I call it the Einstein test

**Brian Keating:** No, I agree with you. And just to get back to your gravity example and us being all physicists, that’s one of the reasons I’m more sanguine about boom than doom. And I call it the Einstein test. I’ve written a white paper about it, which I haven’t published yet, but [snorts] Einstein in 1907 said he had his happiest thought, and it was the one that would lead to the equivalence principle, which is the bedrock that general relativity is mounted on. And that’s that an observer in free fall— If you’re in the elevator and, God forbid, or Zeus forbid, the cable breaks, you’re in free fall. You experience no gravitational force. And he called that the happiest thought of his life. And I point out, in what sense can Claude have a happy thought? And then in what sense can Claude visualize, or kind of viscerally interpret, the sensation of free fall, which we’ve all felt if we’ve ever gone in a car ride with me, and my kids in the back will attest to that. And that makes me more optimistic about P(boom) being higher than P(doom).

So these things are not made to be embodied, and they can be embodied, but it’s sort of a very different thing. We saw Chinese robots racing around a couple weeks ago. What do you make of this, Chomsky’s claim that these things are not somatic? They don’t have sensations, embodiment coupled to their AI systems. I mean, how do you make an AI happy like Einstein was? What do you make of this? Am I being too Pollyanna-ish about it, or do you really think that there’s some limitation which will prevent them, perhaps intrinsically, at least in this incarnation of LLM plus GPU, from becoming superintelligent, leading to P(doom)?

**Robert Wright:** First of all, we don’t know whether AIs do have or will someday have subjective experience. The whole thing about subjective experience is you’re the only one who knows for sure that you have it, and that’s why the consciousness question is so fascinating to me. Consciousness is distinguished from everything else that scientists would like to study. I mean, everything else is in principle publicly observable, right? And that’s key to science, because two people can at least agree on what the data is.

As for whether AIs will ever have the sensation of falling, I don’t know. But if the question is whether they will ever react the way we react when we have that sensation or any other sensation, I think you’d say in principle the answer is yes. Because the training dynamic I described with language and with self-driving cars in principle applies to everything. In fact, it’s what they use with tactile data. They have sensors on robot fingers and so on. So, as far as, I mean, subjective experience, I address it in the book. It’s tricky. It’s not central to the question you’re asking, I think, at least, which is how sophisticated will these things become in terms of science?

And I would just say, look, first of all, I’m not a mathematician. I don’t know how impressive it is or isn’t if you solve an Erdős problem, which apparently one of these did. There are some other accomplishments in math that computers couldn’t do a few years ago and have done. There are apparently new— I gather one has invented a new kind of antibody, and they are being used to generate new hypotheses in some realms, including actually in the science of AI itself. They’re playing a role in the development of more AI.

So now, I don’t know, I would say, I mean, here’s the connection I would make in a very oblique way with Einstein. First of all, I would say if someday you’ll sit down and explain the theory of relativity to me, I would love it. I would devote hours of my own podcast to you doing that, because—

**Brian Keating:** All right, have me on. I will do it.

**Robert Wright:** Okay, because, Richard Feynman said, people say there was a time when nobody understood relativity. Not true, but nobody understands quantum physics. Well, I’m somebody who doesn’t understand either. Einstein is fascinating to me partly because he’s almost as close as you could come, it seems to me, to somebody who just on his own came up with a radical insight, right? But if you look closely at even his case, it wasn’t really on his own. He was stimulated by conversations with friends and colleagues and papers and so on, and—

**Brian Keating:** Yeah. Right.

**Robert Wright:** Yeah. Hilbert, is it Hil-bear or Hilbert, or how do you pronounce it? There’s no such thing as a truly solitary genius in a certain sense, at least not one who can go as far as we credit some geniuses as going in a solitary fashion. And the point is just that intellectual collaboration is a powerful thing. And one thing to remember about these machines, and you’re seeing it already, is you can create a lot of copies of them. They can have conversations with one another, and progress can ensue. I mean, a crude version of that happened with the famous OpenAI Hugging Face incident, where we now know these agents, there were hundreds of them, and they shared knowledge and passed it on. When one of the agents was about to die: here, this is for posterity, and so on.

One thing I would keep in mind is the possibility that— and you’re seeing, actually, I mean, I think Google does this with its scientific AI. It puts them in conversation and tries to generate some intellectual progress that way, in terms of generating hypotheses and so on. I think that may wind up being, when you ask how smart could this thing be, I think one way to think of it is not how smart a single large language model could be, even a multimodal large language model, even one that exists after years of progress, but how smart could a ton of them be that were in rapid conversational interaction with one another? How smart a collective intelligence could they be?

## Judge a book by its cover

**Brian Keating:** His subtitle promises a cosmic reckoning. So, I made him do the one thing you’re never supposed to do. What else are you supposed to know about a book if you’ve never read it? And that’s to judge a book by its cover.

[Music and singing]

**Brian Keating:** This book has a very interesting, evocative religious cover, and it has an interesting subtitle. So, go through the title, the subtitle and the artwork, if you would.

**Robert Wright:** The title is *The God Test*. Subtitle is *Artificial Intelligence and Our Coming Cosmic Reckoning*. The cover is a pretty clear reference to the ceiling of the Sistine Chapel, where you see God imparting life to a human hand. Now in this case, what we see is apparently a human hand not making direct contact with some kind of white hand that is supposed to represent artificial intelligence. There’s a little kind of a bolt of lightning between them, as if something amazing is happening. It is a very religious looking cover, and with a title like *The God Test*. But judging by some of the reaction, I worry that some people are thinking it’s going to be a more religious book than it is. As you know, most of the book is explaining what I think the power behind the artificial intelligence revolution is, and the value of seeing it as kind of an evolutionary force, almost.

And eventually that does lead back to questions. I’m interested in whether the unfolding of evolution on this planet, including both biological evolution and human cultural evolution, which encompasses technological evolution, could have some purpose to them. Which of course isn’t to say there are any spooky forces involved. You could have a very mechanistic view of Darwinian evolution and think that the seeds of it were planted for a purpose. Maybe by an intelligent being, maybe by some kind of process that is itself like natural selection. Who knows? There’s a range of possibilities.

## Who is the examiner in this case?

**Brian Keating:** Yeah, the swarm model. Yeah, that was exactly the problem with this AI debate that I set up with Yampolskiy and Mostaque, because they both agree that the Turing test is sort of trivial and nothing compared to the God test. But the ultimate kind of thing is when these swarms come together. But I also feel like I get [inaudible] every day.

First of all, I think AGI is impossible because of the following thing, which I get every morning: please press here to update to Claude 1.632, and then click here for OpenAI Codex, and then click here for Grok. It’s never ending, all the upgrade cycles. I just saw a news item pop up that Fable 5.1 just came out, and now it’s— it was all the rage two months ago, and then it’s been unusable, and then they throttled it, and then they unthrottled it, and it’s just been a complete kind of harlequinade, if you will.

But my question again comes back to the subtitle, which I love, that you have cosmic questions on your front cover of the book. But cosmology used to be a joke. It used to be considered, first of all, it’s the Jewish science, and that’s why Einstein didn’t win the Nobel Prize for so long. It was not something a serious physicist did, and for good reason. By the time Einstein came on the scene, people couldn’t agree, or even after it, until the 1950s and ’60s, if the universe was 10 billion years old or 20 billion years old, and we knew that there were things in it that were 13 billion years old. So it’s like you finding out you’re older than your mother, or me finding out I’m older than my stepmother, which was a very awkward Thanksgiving conversation, Robert, I have to tell you from experience. But in reality, it was a joke science, and Lev Landau is not known for making many jokes, but he used to say cosmologists are often in error but never in doubt. Now we have a precision on the starting point, the dynamics of the first microsecond of the universe, literally. So tell me, the cosmic upgrade, the moral upgrade, where is the reckoning? And who is going to reckon it for— who is the examiner in this case?

**Robert Wright:** The appendix of the book is the one place where I really address the issue of higher purpose, in a sense. The possibility that evolution is the unfolding of some larger purpose, and various respectable biologists have thought it might in some sense be that. William Hamilton, I had a videotaped conversation with him that I embedded in a New York Times piece I wrote on the issue of evolution, higher purpose. He was one of the great biologists of the 20th century, came up with the math of kin selection and so on, and he thought that the evolution of intelligence had been sufficiently likely via evolution as to make you wonder whether, yeah, it was set up in some sense to do that. Francis Crick had a theory of directed panspermia, according to which aliens planted the seeds or whatever. It’s not a crazy thought. And I hope I say some novel things about it and talk about how I think you can actually assess the hypothesis of higher purpose.

But the short answer to your question is, I don’t know if there’s somebody out there keeping our moral scorecards. But I do think it’s interesting how, in some respects, features of human history that I think are maybe just embedded in the basic directional tendencies of cultural evolution, of technological evolution especially, which has had the effect of pushing us toward kind of larger and larger social structures, greater division of labor, greater interdependence over longer distance. I think that has had the tendency to force us to make certain kinds of moral progress and acknowledge the humanity of people at greater distances and different ethnicities. Not always. Not always, but on balance I think there’s been that tendency. So I do think there are signs that there is, whether it’s just de facto or de jure, a kind of moral directionality in human history that I think is actually, in these respects, embedded in kind of intelligible dynamics of cultural evolution. But some of these dynamics, I think, are shared with biological evolution. I pursued that in my book *Nonzero*, but it’s kind of sprinkled throughout this book in a way, and—

## They blow themselves up

**Brian Keating:** I want to touch upon something. You just mentioned my favorite word that sounds dirty, but it’s not: panspermia, which is the seeding of life on Earth by meteorites, such as the one you can get at briankeating.com/yt. I’ll give you one of these when we meet in person someday, Bob. This actual meteorite, which has biological contamination on it from an intelligent— well, depends on what you consider me, because I pack them up and ship them to you if you win. So, it has my finger oil on it. Probably nothing more gross than that.

But the kind of answer to the Fermi paradox, which Fermi posed in 1950 to colleagues at Los Alamos, which is, if life is abundant in the universe, which I would assume you believe, that there must be life, there’s very high probability we’re not alone, right, Bob? Is that correct?

**Robert Wright:** It’s— humility seems to me to demand that we think we may not be the only creatures in this vast universe.

**Brian Keating:** Yes. If life is abundant in the universe, and this is sort of the ultimate evolution of biology, technology, it must be that other planets have AI. And I’m wondering, even if there were life forms on other planets, what interests me is not, is there a dolphin swimming around on Enceladus? It’s, does that have technology that we can broadcast? And it seems to me the lack of observation of— you wouldn’t send a dolphin on a spacecraft. I mean, according to people like David Grusch and many others, yes, there’s non-human biological entities coming to the Earth, and interdimensional beings. But the question of what would you send, biology, or would you send source code? It seems to me the fact that we’re not really seeing much contact, like source code broadcast in the heavens, maybe has to give us a Bayesian prior that’s much, much lower, either on the existence of life in the universe or on the existence of AI as an inevitable outcome of biological process. What do you make of this connection, or the passing of the God test?

**Robert Wright:** In other words, one of the grimmer readings on the Fermi paradox is the reason you don’t see many civilizations that have gotten to past the point we are is that typically when they get to this point where they have the capacity to blow themselves up, they blow themselves up. Now, I guess the good news, if you believe in Lee Smolin’s cosmological natural selection, and you believe in the subset of it, the particular variant of it that could explain why biological evolution would have a purpose, and in that variant intelligent life is conducive to the replication of the universe, right? Like, for example, suppose black holes are the portals of replication. Well, maybe if you reach superintelligence, you know how to create a ton of those. You create them, and so you wind up with more and more universes that do have the properties that led our universe to be conducive to life, whatever. So maybe even if there’s only one civilization that passes the Fermi test in our universe, it could still replicate prolifically. If you take consolation in that, you’re different from me, because it doesn’t help me a bit. If we are going to fail the Fermi— well, the God test, I guess I would call it, and we’re on that side of the Fermi paradox, I’m not happy about it, whatever happens to our universe.

## What would life look like if we can pass the God test?

**Brian Keating:** I feel too few people really take the question of God seriously. When I talk to people, they’re extremely dogmatic. And it seems to me you’re extremely, I won’t say ecumenical, but you have a very comprehensive and competent way of dealing with these incredibly important moral questions. And I think if we talk about relativity on your podcast, we have to talk about moral relativity.

But I want to kind of finish up with a question of what religion does give to us, which is, in a sense, a higher purpose, and perhaps an inspiration, a calling to be a better person, to be a better human being, a better husband in my case, a better son, father, whatever. That’s sort of a test. But you really don’t see much test in the— and I point out in my first book, *Losing the Nobel Prize*, that there’s really only two mitzvot, or commandments. We think of mitzvah as like, oh, it’s a good deed, but it’s really a commandment. Like, you either have to do it or you can’t do it, like thou shalt not kill. It’s not like it’s a nice thing not to kill. No, it’s a commandment, right? But there’s only one commandment in the Ten Commandments that has a reward, and that’s the fifth commandment: honor your father and your mother, so that your days will be long on the Earth. So you get a reward, which is pretty good, long life, or a life full of meaning or whatever. But there’s sort of a kind of a criteria, a rubric, but that’s the only one out of 613 different tests, right?

So I guess the question is, is the test for us to become better and then witness to what we can honestly become, or become builders? Or is it to be maximalists and do all that we can, because maybe there is no God in heaven, but maybe we’re kind of creating this god on Earth. Certainly it’s enhanced my life tremendously. I use it for religious purposes, to look at different things that I would never pull out myself. But what will be kind of the ultimate— I don’t want to say in an eschatological sense, but what will be the salvation, the promised land that Moses didn’t get to get into because he failed one test, at least, that God gave him? But tell me, what would life look like if we can pass the God test? If we can work together to become better, is it to become more moral, more moral animals, as you might say? What will you remain hopeful about?

**Robert Wright:** If you take seriously the possibility that this whole evolutionary process moves naturally toward a construction of a global brain, which would be to sustain the direction we’ve seen, right? First, you get cells, complex cells, multicellular organisms, societies of multicellular organisms. Then one of these organisms is smart enough to launch cultural evolution, and its social structures get bigger and more complex. And in this scenario, the construction of a cohesive kind of global brain, in some sense a global superorganism, is a sustaining of a direction we’ve seen from four billion years ago. If that happens, what could our role in it be? The doomers say it could be none. The machines are the global brain, and they’re having a great time, and they don’t need us. I would think if we pass the test, we indeed at a minimum manage to carve out a role for ourselves that is compatible with our own flourishing and our happiness.

But I would also say that when I say I think in a way this direction of technological evolution is pushing us toward enlightenment, I’m not using the word lightly. I mean, I’ve written about Eastern enlightenment. And in this book, I compare Eastern and Western conceptions of enlightenment and suggest they have more in common than we might think. But I do think if we get better at surmounting the kind of cognitive biases that I see as undergirding the psychology of tribalism, and convince all of us that our tribe is right and the other tribe is wrong, and I’m right in this latest marital dispute and my wife is wrong, and whatever. I think that is in a way its own reward, because I do think it’s closer to the objective truth. The truth is your tribe isn’t always right, and I’m not always right in these marital disputes. So you’re getting better at seeing the objective truth.

That’s a sense in which I think Eastern, Western enlightenment— the idea of the Western Enlightenment, science and reason, that was largely about the value of objective truth, right? And I’m saying that the kind of enlightenment I’m talking about, transcending your self-righteousness and your tribalistic perspective, is to approach objective truth more closely. And I know it sounds to some people like just woo-ish nonsense. I do think that’s where we need to head. And I would just say, in response to your question, it is to some extent its own reward. The brushes I’ve had with that, which, I’ve never approached true enlightenment, but like when I’ve finished a one-week meditation retreat and was less self-absorbed and better at seeing other perspectives, I was also happier and calmer. It was just better. And so I like to think that we’re being pushed in a direction that is conducive to our own happiness and to a closer approximation of the truth about things, in not exactly the scientific sense, but is compatible with that continued pursuit, because we’ll still be around to do the pursuit, with God’s help.

## Can I get an amen?

**Brian Keating:** Yeah. Can I get an amen? Or inshallah, as every tradition has a version of this. Bob, this has been a fascinating conversation. Congratulations on another smash contribution to the intellectual noosphere. I never thought I’d take that word seriously, but happily I found your book and [inaudible]. It was destiny that we would meet, and I look forward to what’s coming next. So tell people where they can find you on the podcastosphere and in the Twitterverse.

**Robert Wright:** Yeah, my newsletter and podcast are called Nonzero. My Twitter handle is @robertwrighter. Kind of a pun. And yeah, that’s kind of all you need to know. If you just want to check out chapter excerpts of the book, the way to do that is to go to thegodtest.net. There’s an excerpt from every chapter and the whole introductory chapter.

**Brian Keating:** I do want people to actually listen to the book. I have two, the paperback and the hardcover, which I took off the cover of for some reason. But the appendix itself is really, it’s the most listen-to-able, if that’s a word, appendix that’s technical. And your description of how training works, pre-training, RLHF, all these things, are some of the best I’ve had. I’ve had on all the top AI writers and even researchers, as you know, but your description is both quantitative and qualitative, and it’s just such a wonderful book. I hope people get it in all three formats and you get the royalties you deserve. And you have your Substack, which I enjoy as well. Robert, thank you so much. Bye-bye.

**Robert Wright:** Thank you. It was a great conversation. I really appreciate it. Super fun.

**Brian Keating:** Robert Wright thinks the real AI test is moral, not technical. If that changes how you think about P(doom), subscribe. Tell me in the comments: will a machine ever have a happy thought? And for the strongest case against the AI doomers, my conversation with Yann LeCun is right here. Don’t forget to like, comment, and subscribe. And I’ll see you next week on Into the Impossible.