Today, always speaking with Stuart Russell. He is a professor of computer science and engineering, and you see Berkeley he's also AY adjunct, professor of neurological surgery, and you see San Francisco. He is the author of the most widely read textbook on the subject. Today, I artificial elegance, a modern approach and over the course of these ninety minutes, or so we explore the topics that you may have heard me raise in my TED talk anyway. Stuart is an expert in this field and a wealth of information, and I hope you find this conversation as useful. As I did. I increasingly think that this is a a topic that will become more and more pressing every day and if it doesn't for some reason, it will only be because

scarier, things, have distracted us from it. So things are going well, yes we worry more and more about the consequences of ay or so it seems to me, and now I give you store Russell. I'm here was Stuart Russell store thanks for coming in the park ass, you welcome, Our lesser should know you ve been up nearly all night working on a paper relevant to our topic at hand. So double. Thank you for doing this, no problem. I hope it will be coherent. While you you, ve, got. Now nearly infinite latitude not to be so perhaps you can tell us a little bit about what you do. Have you describe Europe job at this point, so I met Professor Berkeley, a computer, scientists, and I worked in the area of artificial intelligence.

For about thirty five years now, starting with my phd it's for most of the time I ve been. What you might call a mainstream. A researcher work on Machina paying and Power ballistic reasoning planning game playing all the things that that day, I people work on and then the last few years, although this has been something is that's concern me for for a long time, I wrote a textbook in ninety ninety four, where I had a section of a chapter talking about what happens if we Accede in I mean what happens. If we actually build machines at a more intelligent than us. What does that mean so that was forever in into intellectual question in its become. A little bit more urgent in the last few years as progress is accelerating and

The resources going into a I have grown enormously, so I'm really asking people to take the question series What happens if we succeed? As you know, I've join the chorus of people who really the last two years, Have begun worrying out loud about the consequences of a I or the consequences of all of us, not building it with more or less perfect conformity to our interests and one of the things about this choruses that is mostly made up of non computer scientists and therefore people like myself for Elon, musk or even physicists like MAX tag. Marken Stephen hawking are seemingly dismissed with alacrity by computer,

scientists who are deeply sceptical of these worried noises, we're making end. No, you are not so easily dismissed because you are, you have the the really the perfect Bonatti Data of a computer scientist, so I want to get us into this territory and are actually want in. I don't actually know that you are quite as worried is, as I have sounded publicly serve as any difference between your taken, mine. That would be interesting to explore, but I also want us to to at some point I'd like you to express the the sound is basis for this kind of skepticism, that you know that we are crying Wolf in a way that is unwarranted, but before we get there right at one, ask you a few questions to to get our bearings. The main purpose here is also just to educate our listeners about me to what artificial intelligence is and what its implications are, whether if everything goes well or everything, goes less than well. So very a disarmingly, simple question here at first: what is a computer? Well,

so pre much everyone. These days has a computer, but that doesn't necessarily understand what it is. The way is presented to the public, whether it's your your smartphone or your laptop, is something that runs a bunch of applications and the applications. Do things like edit word documents of a face to face video Jack, and things like that, and what people may not understand is is that a computer is is a universal machine that any process that can be described precisely can be carried out by a computer, and an every computer can stimulate every other computer, and this. This property of universality means there at tat. Intelligence itself is something that

a computer can in principle, emulate- and this was realized. Among other people, by eight lovelace in the eighteen fiftys when she was working with Charles Babbage they had this idea that the machine they were designing might be a universal machine, whether they couldn't defined that very precisely, and so the immediate thought is well, if its universal than it can it can carry out the processes of intelligence, well as ordinary mechanical calculations. So a computer computer is really end thing anything you want that you could describe precisely enough to turn into a programme to relate the concept of information to that they sound like very simple questions, but these are very good at this Certainly deep questions. I'm aware, I think everyone understands that out there is, is a world the real world,

and we all know everything about the real well, so it could be one way or could be another. I think it could be theirs gazillion different ways. The world could be in all all the cars are out, there part could be parked in different places, and I wouldn't even know it So there are many many ways that we could be an information is just something that tells you a little bit more about what you know what the world is wit, which way is that is the real world out of all the possibilities that it could be And as you get more more information about the world through typically, we get it through are: eyes and he is an increasingly we're getting it through the internet, then that that information helps to helps a narrow down the ways that the real world could be,

and Shannon, who is a logical engineer at MIT, ah figured out a way to actually quantify the amount of information. So if you think about a coin flip, if I can tell you which way that coin is going to come out, heads or tails than that. That's one bit of information and of that lets you give you gives you the answer for a binary choice between two things: And so from info in theory we have. We have wireless communication, we have the internet, we have all all the things that allow computers to talk to each other through physical medium so information theory, has been in some sense. The complementary or the hand maiden of of computation- and

Allowing allowing the whole infiltration revolution to happen is an important difference between what you just described, computers and in the information they process and mines fell asleep justness aside for that for the moment? But if I asked you what is a mind, would you have answered that question differently? So I I think I would because the mind the word mine carries with it this this motion, as you, as you say, of consciousness, it's not I the way you can't really put aside the notion of consciousness, except if you're taught me talking about, are the unconscious mind, you know it likely all the unconscious cognitive processing. We do his mind a misnomer there without consciousness it might yeah unconscious.

Kind of like saying, artificial grass. It isn't grass, but he kind of like grass, so just to give you a quote John Hoagland has written a lot about he's a philosopher and he describes the notion of strong, I as it as it used to be called. As building machines with mines in the fallen literal sense so So the word mind there is carrying the idea that there is true, conscious awareness, true semantic understanding and perception. No perception is sexual experience and I think this is an incredibly important. Thing, because without that nothing has moral value. There are lots of complicated physical processes in the universe, you know stars exploding in rivers. In you, know

issues, melting and all kinds of things I've got, but none of that has any moral value associated with the things it generate moral value? Are things that have conscious experience? So it's that's a very it's a really important topic, but I have nothing to say about it. What However, will not yet, I guess says it: we're we're gonna get there in terms of if consciousness is at some level, just an emerging property of information processing. If, in fact, that is the punch line at the back of the book of nature will then we need to think about the implications of building conscious machines, not just the intelligent machines, but you introduced ay, ay, term here which we should define. You talked about strong verses, weak ay. I guess the the more modern terms are narrow versus general artificial intelligence. Can you define those horse right so that the wood

the words strong and weak about change their meaning over time so strongly. I was, I believe, a phrase introduced by John, so in his chinese paper romanian slightly than that, but what he meant was the version they. I that says that if I build something weird human level intelligence, then, in all probability, it's gonna be a conscious device that that the the functional properties of intelligence and consciousness are inseparable. And so strongly eyes the sort of the super ambitious form with a sigh and we K. I was about building I systems that have capabilities that you one that you want them to have, but they don't necessarily have the the consciousness or indeed the first person experience.

So that and then, and then I think, there's been a number of people. Both inside and outside it feels it of using strong, and we are in various different ways and now largely you. See stronger. I and some time generally I or artificial general. Talents to mean building. I systems that have the capabilities. Comparable to or greater than those of humans without any opinion being given on whether this consciousness or not, and then narrowly, I meaning they. I systems, I don't have the general multi, they might be very capable, like alpha, goes very quickly we'll go player, but it's too narrow and sensitive cant do anything else. So we don't think of those general purpose in telling its role and given that consciousness is something that we just don't have

a philosophical handle, let alone a scientific handle on. I think, for the time being, we'll just have to put it to one side, and the discussion is gonna have to focus on capabilities on on the functional properties of intelligent systems. For this, as other term, one here is in this area, which strikes me as a an actual Roy, a term that their names almost nothing possible. But it's. Human level ally, and that is how it often put forward, as currently they nearer landmark too soon. Or intelligent I or something it's beyond human, but Seems to me that even our narrow at this point, you know the the calculator in your phone or anything else that gets good enough for us to dignify it with the name intelligence very quickly,

comes superhuman even in his narrowness, so is the phone is a better calculate than than I am or will ever be, and if you imagine building system that is a true general intelligence, its learning, not confined to one domain as opposed to another, but much more like a human being and that it can learn across a wide range of domains without having learning and one domain degrades learning. Other you very quickly, if not immediately will be talking about superhuman ere. I, because presumably this system will it's not going to be worth calculator than my phone right? It's not going to be a worse chess player than deep blue. It's like I did at a certain point. It's going to very quickly be better than humans at everything it can do so it is, is human level. I am mirages it is some service of awaited to think about that counts So. I think human level I is, is just

notional goal, and I I basically agree with you that if, if we can achieve the generality, of human intelligence, then we will probably ex seed on many dimensions the actual capabilities of humans. So there are, there are things that humans do do we really have no idea how to do yet, for example, what what humans have done collectively in terms of creating science we don't know how to get machines to do something that we can. We can imagine that, theoretically possible. We know it's some somewhere in the space of programmes. There exists a program that that could be high quality scientists, but we don't. We don't know how to make anything like that. So it is possible that we could have.

Human level, capabilities on sort of money on all the mundane intellectual tasks that don't require these really creative re formulations of a whole conceptual structure that happen from time to time in science and in this is sort of what it's happening already right, I in in all in, as you say, in areas where computers become confidently, they quickly become super competent, and so we could have super competence across all the mundane areas, like the ability to to read an answer suited the kinds of questions that You know an undergraduate could answer by reading it. Reading a book we might see those kinds of capabilities, but it might be then quite a bit more work which which may we

learn how to do to get it to come up with the kinds of answers that the truly creative and deep thinking human could do from from looking at the same material But this is. This is something that at the moment is very speculative mean what we. What we do see as the beginning of generality, so you often see people in the media claiming Orwellian oak computers can only do what they programme to do there only good at narrow tasks. But when you look at, for example, Deak, U N, which was Google deep mines first system that they demonstrated to this learn to play video games and learn com. Lee from Scratch, so it was like a new born baby opening its eyes for the first time it no idea what kind of a world ASEAN. It doesn't know that there are objects or that move or there such a thing as time or good guy

is in bad guys or cause, or roads or bullets or spaceships, or anything. Just like a new born baby. And then, within a few hours of messing around with with a video game, essentially throw cameras reefs looking at the screen, it doesnt have direct access to the internal structures. The game at all is looking at the screen very much the way Human beings interface with the game. Yeah exactly are, the only thing it knows is. Is that once it more points and so within a few hours, its able to learn a wide range so so most of the games at Atari produced. It reaches a super human level of performance in a few hours entirely, starting from nothing. That is important to say It's the same algorithm playing all the games, not it's not like deep blue, that is the best chess player, but he can't play TIC tac toe and will never play TIC. Tac toe is so completely different. Correct approach,

this is this is one are really. We know it could be a driving game, it could be stays invaders, it could be packed and ban it. Could the undersea yo see quest with submarines? So in that sense, and when you look at that, you know if your baby did, that woke up, you know the first day in the hospital and by the end of the day was, was beating everyone beating, while the doktor. That Atari Video games, you be pretty terrified, so you and it is demonstrating generality up to a point. I there there is certain characteristics of video games that don't hold for the real world in general main one of the main things seeing that in a video game, the idea Is it you can see everything on the screen in the course of the real world in any given point this tonnes in the real world that you can't see, but it all it still matters, and then

So with video games they they tend to have very short horizons because you're supposed to play them in the pub when you're, drunk or whatever, so they typically on. Unlike chess, they don't require deep thought about the consequent long term consequences of your choices so but Ill Adam LOS do things which is certainly important They like Dick, you and and various other reinforcing learning systems are beginning to show generality, and we seeing with the with the working computer vision that the same basic technology, these the convolutions deep, networks and with their and their recurrent cousins, that the technologies with fairly small modifications not really can Joe changes just sort of minor changes in the details of the architecture can learn a wide range of tasks to do an extremely high level,

kooding recognising thousands of different categories of objects in photographs doing speech recognition? learning to even right captions for photographs, learning to predict, what's gonna but next video and so on. So for so so I think we're. Arguably you, if there is to be an explosion of capabilities that feeds on itself. I think we may be in the beginning of it now. What are the implications, with respect to how people are designing these systems. I e, if I'm not mistaken me, If not all of these deep learning approaches generally machine learning approaches are essentially black boxes, in which you, you can't really inspect how the the algorithm is accomplishing, what it is,

accomplishing is that the case, and if so in or wherever it is the case, are their implications there. That we need to be worried about her. Is that just a novel way of doing business, which doesn't raise any special concern What I think it raises two kinds of concerns, one we three, so one is a very practical problem there Why? It's not working you re dont know why it's not working and there is a certain amount of blundering read about in the dark. Some people call us graduate student descent. Which is that's a very nerdy joke, so great gradient descended, you know walking down down the hill. Is is a way to find the lowest point so graduate students send meaning that you're you're trying out

from system designs and in the process you using up graduate students at a rapid rate, an end that is it that's clearly, a drawback, We know- and I in my research I generally favoured techniques where the design of a system is derived from the characteristics of the problem that you're trying to solve and so the the function of each of the components is clearly understood, and you can. You can show that the system is going to do what is supposed to do for the right reasons and the Black box opera. There are people who just seem to have great intuition about how to design the architecture of these deep learning network so that they know they produce good performance? I think they're also practical questions from the list the point of view that there are a lot of areas, for example medical diagnosis or

treatment, recommendations, recommend being for or against parole for a prisoner, is approving credit or cleaning credit applications where you really want a explanation of why the record nation is being made without that people simply won't accept that the system is used. Some one at one of the reasons for that is that a black box could be making decisions that are biased, racially buys, for example, and without the ability to explain itself, then you, you can't trust that the system is unbiased and then there's a third reasons which I think is what, behind your question about why we might be concerned with will systems that are entirely black box that

we said we cannot understand how the system has been is reaching its decisions or or what it's doing that gives as much less control. So as we move towards more more capable and perhaps general intelligent systems. The fact that we really might have no idea how they working or what they thinking about so to speak. That would give you some concern, because, then one one of the reasons that the ay I community often gives for. Why, then did not worried rice are the people who are sceptical about there being risk is that while we design these systems, you know, obviously we We were designed them, so they did what we want, but if they are completely open

black boxes that you don't know what they're doing them, that that sense of control and safety disappears. Let's talk about that issue of what passed from corporate control problem. I guess we called the safety problem as well, and this is many people were soon will have watch my TED talk where I spent fourteen minutes Warren. About this, but just perhaps you can briefly sketch they concern here. What is what is the concern about Generally I getting away from us. How do you articulate that? So you mentioned earlier that this tick can be to scientists and busters book. Super intelligence was certainly instrumental in was instrumental in bringing it to the attention of all of a wider audience and young people like Bill gates on Elon, musk and so on. But the fact is that these concerns have been articulated by the central figures in computer science and ay. I so much

going back to age, a good environment well I'm in during himself right so people a lot of people may not know about this. I'm just canoe read a little quote two hours During gave a talk on BBC radio Radio, three in nineteen fifty one, so he said if a machine can think might think more intelligently than we do and then wish we be, even if we could keep the machines in a subservient position, for instance, by turning off the power at strategic moments, we should, as a species feel, greatly humbled? This new danger is certainly something which can give us a anxiety. So that's a pretty clear period. Eve super intelligent ay, I we could have a serious problem. Another person who talked about this issue was no, but we

and about window was day. One of the leading applied mathematicians of the twentieth century use the phone. Under of of a good deal of modern control theory. And automation site is often called the father cybernetic. So he was he was concerned. Is he saw Arthur Samuel, complain programme, ah in nineteen, fifty nine learning to play checkers by itself a little bit like the dick. You end that I described learning to play video games, but this is nineteen. Fifty nine and more than fifty years ago, learning to play check is better than its creator. And he saw clearly in this the seeds of the possibility of systems that good our distant. Human beings in general, so

He was more specific about what the problem is. That of cheerings warning is, in some sense the same concern that guerrillas might have had about humans if they had would you know she's a few million years ago, when the human species branched off from from the evolutionary line of the guerrillas at the guerrillas, had said to themselves. You should we create these? when being's right, gonna be much smarter than us yeah. I kind of makes me worried right and in the EU they would have been right to worry because as a species there they sort of completely lost control over their own future n and humans control, everything that that they care about. So so ensuring is really talking about This general sense of unease about making something smarter than you is not a good idea, and what we said was was this: if we used to achieve our purposes a mechanical agency with whose operation we cannot interfere effectively,

We had better be quite sure that the Pope was put into the machine is the purpose which we really desire so this taking sixty nowadays, we call this the value alignment problem. How do we make sure that sir, The values that the machine is trying to optimize are, in fact the values of the human who is, trying to get the machine to do something for the values of the human race and in general, and so now we know actually points to the sorcerer's apprentice story as a typical example of wind. You give a got off to a machine, in this case fetch water, if you dont, specify correctly, if you don't cross, every tee and dot every eye and make sure you ve, covered everything, then genes being optimizing. They will find ways to do things that you don't expect

Those ways may make you very unhappy, and the story goes back to the king MIDAS, five hundred in whatever B c, where he got exactly what he said, which is the thing turns to gold, It is definitely not what he wanted. He didn't want his food and water. Turn the goal of his relatives to turn the gold, but he got what he said he wanted the stories with Jeannie's the same thing right, you you gave her wish to a genie. The genie carries out wish very literally in the near the third way. She's always can you under the first two does I got them wrong and the problem with Cooper, intelligent, I is that you might not be able to have that third wish, or even a even a second, where yeah, so she so if you get it wrong, you and you might wish for something very banal fine sounding like you know, could you cure cancer? But if, if you haven't told the machine that you want cancer cured, but you also want human beings to be Eliza was simple way to keep

cancer in humans is not to have any humans, a quick way to come up with a cure for cancer, is to use the entire human race as guinea pigs or for millions of different national drugs. That might you a cancer, So that's all kinds of ways. Things can go wrong and you know what we have governments. All over the world. Try to write tax laws that don't have these kinds of loopholes they fail over and over and over again, and there are only competing against ordinary humans in tax lawyers and rich people, and yet they still fail despite there being billions of dollars at stake, so track record of being able to specify objectives and constraints completely. We are sure to be happy with the results are track. Record is, is abysmal and

Fortunately, we don't really have a scientific discipline for how to do this so generally, we have only scientific disciplines, a I control theory, economics operations, research that are about how do you optimize an object but none of them are about. Well. What should the directive be so that we are happy with the results? So that's really. I think. The modern understanding, as described in Boston, was broken and other papers of why a super intelligent machine could be problematic is because, if we give it an objective, which is differ from what we really want, then we will. We basically looked like creating a chess match,

with a machine right now. There's us with our objective in it with the directive we gave it which different from what we really want, so it's kind of like having a chess match for the whole world. Ah, were not too good a beating machines, a chess, so it s a great image, a chess match for the whole world. I, when I drove out on a couple of things you just said there, because I'm hearing that these skeptical voice, even in my own head, even though I think I have smothered it over the last year of focusing on this, but in amazingly easy, even for some. Unlike me, this was really committed at the framing of my TED talk, where it is it's just. I was talking about these concern

and there any value alignment problems actually. But the real message at my talk was that it's a very hard to take this seriously emotionally, even when you are taking it seriously, intellectually, there's something so diaphanous about these concerns and they seem so far fetched, even though you can't give an account where I have certainly haven't heard. Anyone give an account of why, in fact they are far fetched when you look closely at them. Feel like you did the idea that you could build a machine that is super intelligent and give it the to cure cancer. Fetch water and not have anticipated that one possible solution to that problem was to kill all of humanity or to fetch the water from your own body, and it just seems it. We have an assumption that things. Inconceivably go wrong in that way, and I think the most compel

inversion of of push back on. That front has come to me from people like David Deutsche, who, you probably know, he's one of them, the Father of of quantum computing or worthy the concept there. A physicist at Oxford has been on the pike asked he argues, and this is something that I don't find compelling, but I just want to put it forward and I am told him as much. He argues that Super intelligence entails an ethics. If we built a super intelligence system, we will have given it our ethics inside to some approximation, but it will have a better ethics than ourselves, almost by definition and to worry about the values of any intelligent systems. We build

is analogous to worry about the values of our descendants or a future teenagers where they might have different values, but they are an extension of ourselves and now we're talking about eight, an extension of ourselves that is more intelligent than we are across the board, and that could be slightly misrepresenting him here. But this close to what he advocates that there is there is, something about that. They should give us comfort it almost in principle that there's just no. Obviously we could stupidly build a system that is gonna play chess for, though the whole world against us, that is malicious, but wouldn't do that and what we will build is, by definition, going to be a more intelligent extension of the best of our ethics. I mean that that's a nice dream, but as far as I can see its is nothing more than that

There is no reason why the capability to make decisions successfully is associated with any particular goals, your family of gold structures. So let me give you a much simpler example. Take a chessboard, you can look at a chess board and you could say well the notion of czech major just cause and of inherent in the chessboard in any intelligent system would cease from chess board and and and the shapes of the pieces that that check made his is the inherent objective of this game, and when would therefore learn to play chess in and be an enjoyable chest partner for us, but this is completely false because you can play reg which ass or you can play. Suicide chess is neither to completely in some sense opposite objectives, the opposite, definitions of what winning means

in the game, and neither of them is inherent to the chess board and I can build a programme to do either of these two things. So there's no there's no sense in which objectives that would tend to produce behaviour them that we are happy with. Ah I, just intrinsic to the world We know that their activity is certainly don't. Don't think that viruses, and so on that day, just do whatever they do to achieve whatever they want, which is you know which evolution has has created them to do so adjust. It just seems like wishful thinking that has absolutely no causal basis. Other than that, you wanted to be true. The fact that Humans have been talking about analogy to our own descendants. So humans, We evolved over well,

globally hundreds of millions of years to function in a certain way and one of the things that we have is we have built in biological wiring. That gives information about what is desirable, we don't like hunger. We don't like pain. We are drawn to pleasure, and so on, in the desire to reproduce these things are built in biologically their approximations too. What we might think of as an evolutionary fitness function, the and sometimes are not very good approximations. For example, many of them can be satisfied by by ingesting certain chemicals and that usually does not lead to evolutionary fitness even

It saddens fight all seems to satisfy all the boy. Biological bearing so in some sense we can. We can short circuit and fool the drives that biology build into us in a way that would be evolution narrowly unfit we also have, I think, a way of learning about. Are you know what our objectives are and should be free observing the behaviour of other people around us, and this is a key idea that we ve been using in our work on value alignment, which is the idea that, when any intelligent end, DE behaves in the world is revealing information about its internal preference structure, about what it once you know, and as this is a very common sense idea. If you see me struggling

out of bed, bleary I'd going down to the kitchen and doing stuff with a machine that makes grinding and steamy noises and so on. Eventually, I get myself a cup of coffee and a drink it you can turn off. While this behaviour that that that's that's what I'm trying do I, they get. I wanna have a cup of coffee in the morning it is perfectly normal. As humans perceive each other, we are constantly trying to explain. I mean this is sort of how how the human mind works. We were always trying to explain we perceive and when we perceiving television behaviour by others we're internally explaining it in terms of objectives, and I think is likely that the human species actually has a built in. Process not just for explaining the behaviour of others in terms of objectives, but actually then assuming those objectives for ourselves as as we grow up,

So we internalize and may become our own objectives because they are explaining behaviour of appears and parents, and so on. So I don't mean to say that this is the end of the story, but these two things the the biological wiring and this more social process which which people sometimes called. Cultural transmission of value system, but is not necessarily cultural transmission by some weeks aiming to you. Ok, this. These are the values of our culture, rule, one rule two or three there's much more pervasive process of of. I'll use absorbing values from from the of

if you're of everybody else in your society, so I think these these processes are very complicated and they still don't work that well I mean, from time to time, groups of humans sort of go off the deepened and end up it values systems are highly inimical to the continued existence of the groups. Are they gonna? They can come cultures that are very vital. Onto a nihilistic, and that doesnt end up well, nothing things! Don't go well for groups that develop those kinds about these systems, but what about them? later claim. Where that we will we'll be safe, because we will together these super intelligent machines, quite literally to our brains. We will essentially become the Limburg system of these new machines

and therefore, by definition, their goals, both long term Instrumental will be anchored to our own values system. You see That is a durable basis for hope. Not really, I do think, that's a possible direction that that we, we may see electronic enhancement of human cognition and people hold that out, as I hope in the sense that will grow our own intelligence by by electronic prostheses. That will enable us to compete in this chest match that people think of is inevitable and unfairness. I should say I think, that's David Deutsche his position as well they'll be extensions of ourselves in in every conceivable hence, but we don't use machines. Currently, that way mean there there are lots of- I mean there- are just lots and lots of machines here there's

machines that kind of are extensions of ourselves. You maybe it maybe my laptop is he's an extension. It serves as an external memory and and remote communication device and so on, but when I about Hydro Electric Power Station, the gets it's not an extension and myself it's it's a whole other thing in a bicycle, ass, sort of an extension myself, but is a is a jedi craft and just an extension to me. You know I mean it's. It's is quite happy to fly around without me or I could be a passenger in it and I think we will. We will have. Intelligent systems that a separate from our brains physically separate. We know we want that. We want robots that can where we can say. Ok, you know go into

is coal mine and rescue the minors who are buried in some Rockville You know, there's all kinds of reasons why why you want autonomy and not have only devices that are physically connect? to our own brains. What I want to do what she is away from his idea that that we forecasting were either forecasting do more with forecasting you'd appear, it seems that there are developed and possibly could follow, that that don't seem to have very good long term prospects for human rights, and there are other parts that do, and I think what we have to do is find the path that do have good long term prospects. For this when race, and that means understanding how design a I systems so that they are highly capable that they do that. Find things that we want them to be able to do, but at the same time that day

We remain fully under control and so visitors the girls I have I have a new centre was funded by the open philanthropy project and so that the goal of the centre is it. We calling it proved to be beneficial hey. I systems we would like it What sort of pious hopes that, if we design thing this design things this way, it should be good we're. She won't mathematical, theorem, saying that certain kinds of designs for I systems are intrinsically safe and, oddly enough is quite easy to explain in a sense one key idea. So if you go back to what we know said right, we better be quick. Sure that the purpose put into the machine is the purpose which we really desire so that that's operating on this function that the only way to design and intelligent machine

that we formulate the objective, and we put it into the machine- am that's precisely the source of the problem right. What have we done put exactly the right objective and we will don't even know how to write down properly we're gonna have all these problems book and you can you have useful machines where you don't put in a precise objective and the answer seems to be yes, we want the machine in particular because we don't even know ourselves really. What would our true objectives are? What will truly make us happy or not? Happy things? We don't know it yet then We want the machine to be explicitly unburden, about the objective that the human has The machines goal is still to try to optimize it to help the humanist. Whatever it? Is? They really want

the machine needs to be clear that he doesn't know what that is released in the night. Precisely and oddly enough, this is this sounds like a very simple idea, but it hasn't really been explored in the field, because, as I as I mentioned earlier, you know all these fields ay. I statistics operations, research control, theory. They all assume that the objective is just something this exhort specified Minnesota by definition, correct And is put into the machine, and why what I would argue is the best that assumptions false and if we have machines that are uncertain about Jackie, then they will behave in very different ways. For example, they will, though, ask questions like what would it be a good idea? If I did this or did you do with? Did you really mean that you everything everything to turn to gold or ok, how bout doesn't sound rights, not consistent with other things? You said in the past.

Last year. Maybe what you meant was just the things I point two and say: Abracadabra will then I'll make those things turn into gold, but you. Don't want your food and drink to him to go because then you'd be dead, and I know I'm pretty sure you don't want to be dead right. So that's the kind of conversation you want to have with AI system and It can only happen with an eye citizen. That is explicitly the uncertain about your objective and you can tell us All kinds of things about what you want- and this is just its evident about what you really want, but it should never be taken as literal truth about what you want And by definition it would remain perpetually open to your saying: that's not what I wanted or that's not what I want and that's not what I meant so that the EU that you can get it would be a break to pull. It is somewhat counter intuitive. If you imagine, building a truly super intelligent

general eyes so that its this is a mine that exceeds us in every capacity that we have, and the only difference is that we built it to serve us in some sense again was leave consciousness out of this injustice. That you can get human level intelligence and beyond, without consciousness necessarily coming along for the ride. As you said at the outset, the existence of consciousness create another problem which we might want to talk about, but now we have an ethical obligation towards this super intelligence system that can be made happy or suffer or be deprived of happiness in in ways that perhaps we can't even imagine- and I would argue that if we build a superhuman, I that was superhumanly conscious. Well, we ve just built something that is actually ethically of greater concern than than we are just as where do of greater concern within squirrels and crickets up the Leslie. That aside, we have a super intelligent artifact that we have no reason to think is conscious, but

if it is in fact, super intelligent across the board and the what would that dialogue be like with our own goal? our own preferences, because at some level this thing- one at one of the reasons why we would want this thing is to tell us what we should want and how to deepen our wants and goals so as to maximize human flourishing to a degree that may be currently unimaginable in how do you keep a a slave that is in every respect? more competent to judge the wisdom of what you're doing than you are. I mean the diva great question. I mean that the The notion of human flourishing is itself problematic. You know who used to define what constitutes human flourishing. Some people might have visions of

you know, conquering the universe and others might have vision of you know a stable, peaceful, dutiful agrarian existence and who knows- and I think one point to Make- is that It would be nice to have a choice as opposed to being given by conflict by limited resources by the sort of inexplicable tides of history that that sometimes just leaders to utter catastrophe you'd be nice to have a choice. How we make that choice this day? I can't tell you how we make that choice, but one thing is clear to me at least: is that one of the things humans, value is autonomy and a future where every decision is made by a machine everything difficult decision is made by machine and and are you

a material needs are met. This doesn't feel to me like a desirable future and it it might be the case that, but what a icy systems do for us in the long run he's give us this freedom of choice to have to have some ability to shape our future and of these catastrophic problems that we frequently get ourselves into, but to some extent, to stay largely in the background so too, to give us freedom and help us understand. What we do really want, but not to be constantly apparent for us, and So this is a story by E M Forster called the machine stops, which I I recommend to all your listeners and

written in nineteen o nine, I think nine. I it's an amazing story because it has the internet. It has basically ipads. Video chat has moves massively online open courses, in fact, is what most people do is is give milk lakes, fears and listen to move lectures and and the machine is its golden? The book, the machines that looks off to everyone material needs to the point where humans become in full, people day they don't go, outside anymore. They don't even bother visiting each other because they can do video chat. They become very, very a feat, and in fact it even the sort of couch potatoes obesity is described in this nineteen. Oh no, story which we think I was a very modern thing had heard I've never read it. I dont associate Forrester with and hd well

the ability to ya. You truly amazing. In fact I would say it's much more pressing than most of what HD wells route. So what happens to human society is. Is that as a society, he becomes gradually more more dependent. On the machine until it cannot function independently at all, and gradually loses its like the disk process that we talk about now with with pilots, for example, I regret the human body gradually loses its understanding of even how the machine works, and so when it when it stops working he came in races in real trouble, so we we clearly would want to avoid that kind of future and and if you, If you ask me what what is this scenario is a bust him has is paper clips, an area where we accidently tell us, you can us machine to make paperclip certain ended, we know turns the whole planet into paperclip.

All the Kansas scenario that I mentioned. So though, it a very simple just thought, experiments just to illustrate the point that seemingly innocuous or even honourable goals by themselves can be arbitrarily bad, but he you, if you ask me what is a likely. Scenario that we may follow. That will lead us you know into a dead end. It seems to me that this in Feeblemind, Andy, Skilling and gradual loss, Autonomy is is a much more likely scenario do occur Eve, and we have to be vigilant against this Have you ever seen, Wally, which is a very first it movie. That's precisely the the scenario that its describing for the future of the human race that we become in feed, obese and completely dependent on machines? serve all of our material needs.

And we lose autonomy. Almost completely yellows are actually had another podcast gas David crack. Our who runs the Santa Fe Institute has its concept of two different types of cognitive artifacts cooperative ones and competitive one. The cooperative ones actually make us better, even in their absence his example is it with. You, learn to use an abacus You actually become better at calculating, even without the abacus you can. You can do that you can. You can internalize the advocates and you can do calculations that wouldn't otherwise be able to do and language is a cooperative artifact and at things things at the concept of numbers for things that we have invented that we we can internalize whether or not they there actually devices in the world, and yet there are other things that are cognitive, competitive artifacts that we actually we get worse because we have delegated all of this operation.

To a machine or to some other artifact which, when it breaks down, we no longer know how to do the thing we used to know how to do perfectly well, weathers long, division or or anything else flying plans. Yap driving cars is in the very near term, something we are going to, probably to the to the benefit of tens of thousands of lives every year. We will blissfully forget how to drive cars at some point, because it will be irresponsible to drive your own car once robot cars are truly safe you can argue that even without a I, the internet is already almost there in terms of if this machine stops it's hard to picture what would happen to a society like the United States with it with an internet that lasted six months. Many think we are already fatally dependent on our technique. Long before we ve birth, superintends and I did about face.

We recently very, very dependent. I mean our economy would would almost collapse, I think we would muddle through their way squarely in some sort of disappear for a good long. While, yes, I think we would, we would see possibly starvation in cities. You know the complete termination of a large fraction of industrial production and air transport? asian systems and so on Do you see this need only when an airlines, Online reservation system goes down on. You know they end up having to cancel tens of thousands of flight. So if it goes down for several days so yeah, I think this. This is just a small example.

Of something that we need to learn. How do we avoid this? Because, in other is always easier to take the institute's convenient without thinking about the loss of autonomy that it entails and before you lose out before you suffer any consequences of lost autonomy, you would have the immense power of having successfully delegated you're a fish operations to a super intelligent, a eyes so the community that decides well. I just want intelligence, my Beck and call that community will be far more power? Then, by comparison kind, luddites? You know the kind of the amish of the future where there trying to maintain in their autonomy in the face of all of these, these immense computational resources that another disturbing wrinkle here, which I talked about and in my TED talk, which is even of

everything goes exactly as we would want it to go. So it is, though, we're just handed a perfectly obedient. Benevolent super intelligent ai. You can still see things going spectacularly wrong for us because we are not economically or politically poised to absorb this perfect labour saving device, because it is just they say, wealth creation vice of the sort that has never existed it can design the machine they can build a machine that can do any thing it can replace any whom

labour, not just drudgery, but the sort of labour that that you- and I do so- you know in a perfect world. We would then all be free to just enjoy ourselves in every way that we want and to have those pleasures be supported, perfected, protected by increasingly powerful labour saving technology and so would be plain frisbee in the park, and no one would have to justify their existence by working because they would be more wealth and then could ever consume just pulled out of the ether by these superhuman machines. But of course we are nowhere near ready, apparently to share this wealth in this way. What I pictured in a somewhat in somewhat of a cartoon, but a picture

we can, it would have trillion errors on the cover of our business magazines and need a fifty percent unemployment in the United States, so that there's a bottleneck layer that we have to figure out how to negotiate. Even if we were given a perfect version of this technology which, as you have already said, is by no means guaranteed, we need the political and economic mechanisms to use this gift for the benefit of all humanity and- and I dont really see many people. Thinking about how to do that. Yes, oh remedies! That's a beginning of another hours, conversation at least two to talk about the effect on the economy and employment and I'm not I'm one of those who is as well. We know we ve had these technological revolution, Four will have amended again and, and we always figure out new jobs, people to do because, as you say, If, if the machine can now replace not just the physical,

Member of the human and and very few humans are, relatively speaking. All physical labour compared to what was the situation in the nineteenth century and if you to replace the mental labour than we know. What is what is left in a new era. There isn't just an infinite supply, new capabilities, that humans have that they can that they can use for employment. So I think you know the briefly one thing that humans have is the fact that their human and the fact that other humans value the attention, the company, the services of of of human beings and as long as that's true there, and they can still be used for roles, but the the kind of society, the economy that that entails. If it's not to be, as you described, a very thin elite supply what it by a somewhat thicker layer of personal servants,

and then every one else. You know ninety five percent of the population being essentially unemployed, just fed and housing entertained and left to play play that that to me it is a terrible future for the human race, so so on both of these fronts, what one being in a? How do we think about? Are the value autonomy in the future. And the other being. How do we construct a functioning society and in the presence of of these kinds of capabilities, I think that there is a limited amount that at the technocrats, the a I researchers and were bought, assists and so on or even the economies can do. I think it has to be a cultural process, though your huge amount of cultural, work has to be done to work through all these questions.

And and first of all, to give people understanding of what the questions are an end. I think either, for example, movies are doing a good job of this. Are there any movies that you think get this get these concerns right and then what? What's the most compelling vision of whether its utopian to stop in that you ve seen so eager there there are bits and pieces, the story that have done well, indifferent movie, so I think transcendence gets across the idea of of intelligence explosion in Canada. Does the same of humanity? Losing control in the face of a much more elegant machine. You know some other parts, the plot a bit less it less excellent, but it has excellent aspects. I think Ex Mckenna gives you a very the essence of the as we talk about earlier than the Black Box of it, of a system that

he's very intelligent, but you really have no idea how its working or what its objectives are or what's going on and in the EU There is a one on one since you it's it's like the chest match idea, except in micro, bosom, less linger on that for a second is it that displayed a property of of an intelligent system, which I think, may people find totally counter intuitive. The idea that intelligent machine we would build might lie to us or manipulate us to meet its own goals or even meet goals. We ve given it, but it sees another way of meeting them as a reason to think that manipulation on the part of superhuman. I is implausible and in principle

No, not at all. We have poker playing programmes that that bluff, so in its arms and slight, they lied to us to be. Does it Boca? It seems to fall right out of doors having goals in the first place. You I mean, did deception, I mean you, know kid, kids, let me sit a fairly early age. They have goals and the they lie in order to achieve them. They say. Oh no, I haven't seen the candy daddy. I have no idea where it is now. I think of the stories that people, whether it lay people or the argument, Anti tell themselves in order, not have to worry about it. I just stories, and I I wrote recently

He wrote a paper where I listed about fifteen of the arguments that have come out of the air I community as to why there is no reason to be concerned and when you, when you look at these, they they all have the waiver of a kind of a fright response. A joke, kind of denial and the arguments are just not worthy of the people who are making them. And I think there our arguments, which say that supervision? I is a long way off, and as yet we don't have much sense of of what it will be like what shape it will have at all, and so it's hard tee to know what to do. Even if you were a knowledge that potentially in one kind of superintendent tough Ocean, a system could present a risk to the human race, but I think you know that's a legitimate line of

argument, I think our research suggests that, in fact, even in the absence of any details about what assistance will be like we can, we can start to do work that will allow us to construct templates that, unlike more classical fines of optimizing of objectives. Do deuce to be potentially safe. They clearly have better properties from You have you? U remaining in control. Avoiding these miss alignment issues. The reason why the time horizon in for me is a deeply uninteresting, and an unconcern thing to reference is twofold. One is that there is a quote I used of yours in my TED talk, which suggested that we have under different descriptions, a very different response to certain blocks of times over you, you said,

I believe this quote originates with you, but it's I think you first appeared in the article you rode jointly with MAX End and Stephen hawking in it in a british paper, but I think I've heard you use it at the keys to the conference. We were at an import Rico couple years ago. The idea that if we received an email, I think you said from AIM Alien civilization, and it said, people of earth we will arrive on your planet in fifty years, get ready, and we would far more motivated then we are when we hear a technologists denying that there's a problem here, because it can take these fifty years to develop so fifty years is just not that much time once you imagine this thing actually arriving on a certain day, but the at the other problem for me is that lets say we didn't we

knew it was, can be fifty years or a hundred years. We still don't know how long it will take us to do it safely. Me that's only consoling on the proposition that that you know it you eat. It only takes you forty nine years to do this safely. So you ve got enough time to you. The right to an end, the idealistic, the other thing is that you know with with, for example, the aliens who take they're gonna, be fifty years or an asteroid where you can calculate the collision, time achieving super intelligently. I requires breakthroughs that they can happen overnight and you know the example I often gave in talks is is what happened? Nuclear physics, where the physics establishment as embodied, for example, by rather food who is partly the leading nuclear physicist of his time was convinced that

even though they knew there was massive, a massive amount of energy locked up in the autumn. They were convinced that there was no way to release Ah, you know in relevant, gave a famous speech. On September, eleventh, nineteen, thirty three, where he said that anyone who talks about this possibility is talking moonshine The next morning, LEO Szilard figured out the key idea, which is a chain reaction, that's mediated by neutrons, so the nuclear chain reaction, so it went from never too do you no less than twenty four hours early the next morning, that say sobering. Analogy though I like, I would say it was hard to quantify these things, but I think we need about half a dozen breakthroughs of that magnitude. To get something that we would call General Ere, I

it is very hard to predict how. But given the rate of investment that the massive concentration of intellectual resources into the feel right now and in the fact that progress seems to be accelerating prompt one problem after another is being knocked down that that you know has been problems of decades of work. Yeah. I just think it be much optimism to say: well, you know it was still a long way off I am seeing you know what one of the arguments that I'm seeing in a community of fifteen. I mentioned one that seems to be cropping up quite often, is well. No, don't worry, because in fact human level, s eyes impossible, so we can get there. It is like what my committee, has been saying exactly the opposite. For the last sixty years, the philosophers have been saying that is impossible and a community has been saying: no, no, it's not impossible and you just confused and an old fashioned and an absolute

Lee human level, I as possible, while the argument that attempt possible presupposes that there is something man, all about a a computer made of meat, or that is something that our brains are doing. That is something. I cannot believe mediated in any other physical system were or any physical raised in that way, you're going to dance and its services is not is not as is not as complicated as that they present no justification whatsoever or being impossible other than well. If was impossible, I don't have to worry about it, so I'm just going to assume it's impossible, rightly made in its so inconsistent with our own history as a field, and it comes with absolutely no technical justification. What's However, even even of the magical kind that it's not only be a form of denial and in that disappointing Is this paper of yours available online now. Can I can I link to it on my blog from more where I am bed. This part has not yet

Oh buddy, it's impressed. I think it will come out of its it's a chapter in a book that will come out from the early next year, I wanted to say one one more thing that people may want to watch, which is the the tv series called humans which come comes from rising channels, for in the UK and to me that gives us a very clear accounting of the kinds of socio ecological issues relating to employment, sense of purpose, autonomy and sullen that would arise in in the series. The assumption is that we have created effectively human level. Not particularly superhuman but human level intelligence embodied in very physically realistic, Android stay there

look very much like human beings and does so it's it's very, very interesting. It very much done not in a sort of a giant syphon style, but in looking at a family that the people in the family that particular Android, who works with the family and only when there is a logic plot which will have to watch the series of doubt about, but I think it really in its white excellent in in describing the very practical day to day adjustments of human society to the presence of the is intelligent, intelligent robots. We ve been great. They ve been very generous where their time, and sorry about the law sleep at land briefly tell us: are you on social media surname web address your twitter account? You want people to know about so much

my web pages. The easiest thing is just a type my name into Google that'll. Take you straight to my web. Page will be on my blog as well thanks against Europe, and a real pleasure like you them before. I close. I would like to thank those of you who have begun supporting the podcast, This has allowed me to spend more time doing it, and it also has inspired me to begin thinking creatively about how I could make us a more substantial project, and this could entail many things that could entail renting a studio where I can do more of these interviews face to face rather than over Skype, which would improve them significantly. It might allow me to hire producer who could bring a level of professionalism, this that I can't seem to muster on my own when I listen to podcasts like serial or any of these other highly produced, shows their aspects. To that that I would like to be able to emulate here. I can hire a research

that could help me move into new content areas and I could even produce live events associated with a podcast. This sort of thing becomes possible once this achieves a stable basis of support. So if you have begun helping in that way, please know how much I appreciate that and if you like to help, you can do that at SAM Harris DOT, Orgy Ford, slash support or you can support the pod cast on a per episode basis through patron Dotcom, forts, Lash, SAM Harris and again, many thanks if you find this podcast I believe there are many ways you can support it. You can review it Itunes or or wherever you happen to listen to it. You can share

social media with your friends. You can blog about it or discuss it on your own podcast, or you can support it directly and you can do this by subscribing through my website has samharris dot org and there you'll find subscriber only content, which includes my ask me anything episodes. He also get access to advance tickets to my live events as well as streaming video of some of these events and of also get to hear the bonus questions.