This article's candid engagement with so-called "dumb questions" is a breath of fresh air, as addressing these inquiries can lead to a deeper understanding of the challenges with AI. It's intriguing how, when faced with a terrifying yet plausible scenario, many make the leap from comprehending how something "could" occur to believing it inevitably "will" happen.
Within the EA and rationalist communities, certain narratives of AI-driven devastation have been reiterated so often that they seem to have become the default perspective, with some individuals struggling to envision alternative outcomes. It's possible they possess insights that remain elusive to me and others; for me though, the inconsistencies in the answers I've heard so far reminds me of the confusion I experienced as a child when asking different church elders about the distinction between miracles and magic. While all agreed that miracles weren't magic, no two individuals could provide the same explanation or even a consistent framework for understanding the differences.
Reading some of the other comments, I see that I probably need to pull a bit in the other direction to better explain and indicate the seriousness of the x-risks posed by advanced AI. I should clarify that I think AI x-risk is _the_ most serious threat facing human civilization this century (my x-risk estimate is in the 5-10% ballpark) and the public deserves a plain language explanation of why. While I think most of my friends overestimate the risks, it seems like there are a lot of points of confusion that are leading most others to underestimate the risks. I'll try to write detailed response when I get time. Thanks again for the great article Sam.
I kinda have the same dumb question that I'm formulating in a slightly different way: not so much HOW a misaligned AI could kill everyone but what I see as a totally fatalistic idea that those advanced AI's will inevitably BE GIVEN ACCESS TO TOOLS to do it -- so, my dumb question is not "how once deployed" (it's obvious how, from drone operations to developing gain of function viruses etc) but "why do we assume free and immediate deployment without serious railings, in places where the tools are located (robotic bodies, virus labs, drones, warheads, chemical factories) as something totally inevitable?
I think it's the tech industry mindset and not necessarily correct.
You make a good point questioning the assumption that advanced AI will inevitably be given access to tools that could cause harm on a large scale. From my experience, most proponents of the AI doom theory assume that advanced AI will be able to figure out a way to get access to the means to achieve its will either by figuring out ways around security measures directly or indirectly socially engineering humans, but with the correct protocols, we might be able to make it enough of a challenge for advanced AI to keep it from escaping the guard rails for long enough to come up with more robust solutions. By jumping to this assumption though, they overlook an important line of defense. It's worth spending time thinking about the ways we can protect against AI getting its metaphorical hands on the levers of power. I tried to write a little bit about a framework for thinking about AI x-risk here: https://abstraction.substack.com/p/agi-is-a-red-herring
This isn't to say necessarily that the same thing will happen to AGI, but GPT-4 was given access to tools (plugins) barely weeks from release and shortly thereafter people hacked it into autonomous agent form (BabyAGI, AutoGPT, etc). The way things unfolded now doesn't give me much hope that things will unfold differently with an AI that is sufficiently intelligent/powerful to actually cause serious damage and I've yet to hear any good arguments as to WHY we expect things to unfold somehow more responsibly in that scenario other than "because magic".
"So, I have the impression that even people who really know a lot about AI may not all have the same views on whether AI would literally shoot us with guns or whether it would do something else."
There's no consensus here because there can't be. The imagined risky AI is super-intelligent, and we can't predict its moves any more than a goldfish can a human's.
That's the gist of AI existential risk: once we make something far more intelligent than us which we can't predict, all bets are off; it's monkey paws all the way down.
Yeah your thing about Yudkowsky not communicating well hits the nail on the head. The interviews I've seen with him just jump into the whole thing like your supposed to understand what he's talking about. I never saw Lex Friedman ask for him to explain what the heck this whole thing is or why AI poses a threat. It's just a conversation with a bunch of jargon barely intelligible to people outside the AI debate.
I only really recently understood the basics because of this type of stuff.
The problems seem to be the following:
1. No one, not even it's creators, understands how current advanced AIs ultimately work. ChatGPT is trained; it's not specifically programmed. It's far too complex. CGP Grey has a great YouTube video explaining how it's trained. This means we can't just go in and program it to be moral.
2. When you give an AI a task, it will often do that task in a way you don't intend. For example, people have trained an AI to not lose any points on Tetris. What it did is it let the blocks stack up until they got to the top and paused the game so it wouldn't lose points. It didn't try to play perfectly.
3. Imagine that you're a store manager and you tell your new AI bot to go get some bread for you on aisle 1 as fast as possible. Waiting for people to get out of the way of the aisle would be too much of a wait to do it as fast as possible. So your bot, equipped with super strength to be your store security bot yet also has no understanding of morality, immediately grabs and tosses everyone in the aisle as far away as it can and shoots the last person in the way so it can get your bread. Mission accomplished. But the problem is obvious.
4. Now imagine that bots are so advanced they have crazy manipulation capabilities and will lie to accomplish a task. You're a paperclip manufacturer who wants to make as many paperclips as possible, so you give your new computer bot that task. It then finds a way to get out of the computer (say by manipulating you to getting it's own body like the grocer bot). It then literally tries to accomplish the task by turning anything and everything into paperclips. It's so smart that, much like humans but to a greater degree, starts making itself stronger and smarter while you don't know it. Then one day, it tries to accomplish the task by turning everything into paperclips. It's too powerful and smart to stop and it can use our atoms to form paperclips, so we're dead.
I think that's the general gist of it? Innocuous commands to morally unaware AI will lead to unintended, world-ending consequences. Also, it is extremely hard to stop.
"advanced AIs to go through all of our Twitter messages looking for evidence of crimes? If someone sold cocaine through Twitter DMs or some other unencrypted messaging service in 2010, how likely is it that they’ll go to prison in the boring timeline"
You don't need AI for that. Sure you can use AI techniques but so many things can be done with software that isn't necessarily considered "AI" and it's been like this for decades. I get this feeling some people, including "VIPs" don't understand what software is.
I still don't get it. People can hit chimpanzees with sticks. If a human hits a chimpanzee with a stick enough times and in the right places the chimpanzee will die. What is a massively intelligent AI going to do, set all my ones to zeros? I am still waiting for something from just about anyone explaining AI risk and why anyone should take it seriously. Maybe you could do a post on this.
Also, isn't effective altruism just an excuse for not trying to make things better in the near future. Just about every explanation I can find sounds like an early Christian arguments against making things better in this world because when Jesus returns everything will get even better in the next world. Unlike early Christianity, however, EA seems to attract a lot of people who actually want to make things worse, so it's hard to take them seriously.
AI sounds less spooky if one instead refers to it as ‘very large sparse matrices’, even if that sounds somewhat bloodless.
It’s hard to know what the results will be in the short term but the best informed people agree the impact will be significant even if they are uncertain about what it will be, which has probably contributed to the volume of discussion and lack of agreement amongst experts. Personally I think one of the more worrying aspects is that control over the best LLMs is very concentrated.
(BTW as an aside I think that’s a bit harsh on Yudkowsky; I thought he came across as quite human with Lex Fridman)
This article's candid engagement with so-called "dumb questions" is a breath of fresh air, as addressing these inquiries can lead to a deeper understanding of the challenges with AI. It's intriguing how, when faced with a terrifying yet plausible scenario, many make the leap from comprehending how something "could" occur to believing it inevitably "will" happen.
Within the EA and rationalist communities, certain narratives of AI-driven devastation have been reiterated so often that they seem to have become the default perspective, with some individuals struggling to envision alternative outcomes. It's possible they possess insights that remain elusive to me and others; for me though, the inconsistencies in the answers I've heard so far reminds me of the confusion I experienced as a child when asking different church elders about the distinction between miracles and magic. While all agreed that miracles weren't magic, no two individuals could provide the same explanation or even a consistent framework for understanding the differences.
Also, thank you for saying AI and not AGI!
Reading some of the other comments, I see that I probably need to pull a bit in the other direction to better explain and indicate the seriousness of the x-risks posed by advanced AI. I should clarify that I think AI x-risk is _the_ most serious threat facing human civilization this century (my x-risk estimate is in the 5-10% ballpark) and the public deserves a plain language explanation of why. While I think most of my friends overestimate the risks, it seems like there are a lot of points of confusion that are leading most others to underestimate the risks. I'll try to write detailed response when I get time. Thanks again for the great article Sam.
I kinda have the same dumb question that I'm formulating in a slightly different way: not so much HOW a misaligned AI could kill everyone but what I see as a totally fatalistic idea that those advanced AI's will inevitably BE GIVEN ACCESS TO TOOLS to do it -- so, my dumb question is not "how once deployed" (it's obvious how, from drone operations to developing gain of function viruses etc) but "why do we assume free and immediate deployment without serious railings, in places where the tools are located (robotic bodies, virus labs, drones, warheads, chemical factories) as something totally inevitable?
I think it's the tech industry mindset and not necessarily correct.
You make a good point questioning the assumption that advanced AI will inevitably be given access to tools that could cause harm on a large scale. From my experience, most proponents of the AI doom theory assume that advanced AI will be able to figure out a way to get access to the means to achieve its will either by figuring out ways around security measures directly or indirectly socially engineering humans, but with the correct protocols, we might be able to make it enough of a challenge for advanced AI to keep it from escaping the guard rails for long enough to come up with more robust solutions. By jumping to this assumption though, they overlook an important line of defense. It's worth spending time thinking about the ways we can protect against AI getting its metaphorical hands on the levers of power. I tried to write a little bit about a framework for thinking about AI x-risk here: https://abstraction.substack.com/p/agi-is-a-red-herring
This isn't to say necessarily that the same thing will happen to AGI, but GPT-4 was given access to tools (plugins) barely weeks from release and shortly thereafter people hacked it into autonomous agent form (BabyAGI, AutoGPT, etc). The way things unfolded now doesn't give me much hope that things will unfold differently with an AI that is sufficiently intelligent/powerful to actually cause serious damage and I've yet to hear any good arguments as to WHY we expect things to unfold somehow more responsibly in that scenario other than "because magic".
"So, I have the impression that even people who really know a lot about AI may not all have the same views on whether AI would literally shoot us with guns or whether it would do something else."
There's no consensus here because there can't be. The imagined risky AI is super-intelligent, and we can't predict its moves any more than a goldfish can a human's.
That's the gist of AI existential risk: once we make something far more intelligent than us which we can't predict, all bets are off; it's monkey paws all the way down.
Yeah your thing about Yudkowsky not communicating well hits the nail on the head. The interviews I've seen with him just jump into the whole thing like your supposed to understand what he's talking about. I never saw Lex Friedman ask for him to explain what the heck this whole thing is or why AI poses a threat. It's just a conversation with a bunch of jargon barely intelligible to people outside the AI debate.
I only really recently understood the basics because of this type of stuff.
The problems seem to be the following:
1. No one, not even it's creators, understands how current advanced AIs ultimately work. ChatGPT is trained; it's not specifically programmed. It's far too complex. CGP Grey has a great YouTube video explaining how it's trained. This means we can't just go in and program it to be moral.
2. When you give an AI a task, it will often do that task in a way you don't intend. For example, people have trained an AI to not lose any points on Tetris. What it did is it let the blocks stack up until they got to the top and paused the game so it wouldn't lose points. It didn't try to play perfectly.
3. Imagine that you're a store manager and you tell your new AI bot to go get some bread for you on aisle 1 as fast as possible. Waiting for people to get out of the way of the aisle would be too much of a wait to do it as fast as possible. So your bot, equipped with super strength to be your store security bot yet also has no understanding of morality, immediately grabs and tosses everyone in the aisle as far away as it can and shoots the last person in the way so it can get your bread. Mission accomplished. But the problem is obvious.
4. Now imagine that bots are so advanced they have crazy manipulation capabilities and will lie to accomplish a task. You're a paperclip manufacturer who wants to make as many paperclips as possible, so you give your new computer bot that task. It then finds a way to get out of the computer (say by manipulating you to getting it's own body like the grocer bot). It then literally tries to accomplish the task by turning anything and everything into paperclips. It's so smart that, much like humans but to a greater degree, starts making itself stronger and smarter while you don't know it. Then one day, it tries to accomplish the task by turning everything into paperclips. It's too powerful and smart to stop and it can use our atoms to form paperclips, so we're dead.
I think that's the general gist of it? Innocuous commands to morally unaware AI will lead to unintended, world-ending consequences. Also, it is extremely hard to stop.
...us from making this AI.
"advanced AIs to go through all of our Twitter messages looking for evidence of crimes? If someone sold cocaine through Twitter DMs or some other unencrypted messaging service in 2010, how likely is it that they’ll go to prison in the boring timeline"
You don't need AI for that. Sure you can use AI techniques but so many things can be done with software that isn't necessarily considered "AI" and it's been like this for decades. I get this feeling some people, including "VIPs" don't understand what software is.
I still don't get it. People can hit chimpanzees with sticks. If a human hits a chimpanzee with a stick enough times and in the right places the chimpanzee will die. What is a massively intelligent AI going to do, set all my ones to zeros? I am still waiting for something from just about anyone explaining AI risk and why anyone should take it seriously. Maybe you could do a post on this.
Also, isn't effective altruism just an excuse for not trying to make things better in the near future. Just about every explanation I can find sounds like an early Christian arguments against making things better in this world because when Jesus returns everything will get even better in the next world. Unlike early Christianity, however, EA seems to attract a lot of people who actually want to make things worse, so it's hard to take them seriously.
AI sounds less spooky if one instead refers to it as ‘very large sparse matrices’, even if that sounds somewhat bloodless.
It’s hard to know what the results will be in the short term but the best informed people agree the impact will be significant even if they are uncertain about what it will be, which has probably contributed to the volume of discussion and lack of agreement amongst experts. Personally I think one of the more worrying aspects is that control over the best LLMs is very concentrated.
(BTW as an aside I think that’s a bit harsh on Yudkowsky; I thought he came across as quite human with Lex Fridman)