Friendly AI PAQ
From The Transhumanist Wiki
[edit] Previous Asked Questions about Friendly AI
Answers gleaned from the SL4 Mailing List, and other SIAI resources.
Editing by Michael Roy Ames, Peter de Blanc, and Edmund Schaefer.
(PAQs gleaned by Michael Roy Ames)
Q: What happens if a Bayesian FAI fails to discover a formal, universal process?
A: A Bayesian AI *is* a formal answer to the question of how to assign probabilities. So's a human brain, albeit the formal answer is so complicated it looks informal. And as for universality, a Bayesian-to-the-utter-core cares not whether the answer is something of which some other mind may be persuaded, only whether the answer is well-calibrated.
Thread link: http://www.sl4.org/archive/0409/9674.html
Q: Is it possible for FAI to find an objective moral answer to an the question What should an FAI do?
A: Roughly, the problem with the 'objective' criterion is that to get an objective answer you need a well-defined question, where any question that is the output of a well-defined algorithm may be taken as well-defined. In the TMOL FAQ you've got an algorithm that searches for a solution and has no criterion for recognizing a solution. This, in retrospect, was kinda silly. If you actually implement the stated logic or something as close to the stated logic as you can get, what it will *actually* do (assuming you get it to work at all) is treat, as its effective supergoal, the carrying out of operations that have been shown to be generally useful for subproblems, relative to some generalization operator. For example, it might sort everything in the universe into alphabetical order (as a supergoal) because sorting things into alphabetical order was found to often be useful on algorithmic problems (as subgoals). In short, the damn thing don't work.
More at thread link: http://sl4.org/archive/0409/9856.html
Q: Why did the FAI target change after CFAI?
(Question with more background) Q: CFAI proposed the creation of an AI that would improve its own utility function according to the feedback provided by its programmers. This feedback implied a meta-utility function which is unknown even to them, and it would be gradually made explict thanks to the AI's superior intelligence. This sounded like a good plan, so Why did you change the plan in later work?
A: We will need a well-specified description of what we will be trying to do. To give a flavor-only example (i.e., don't try this at home, it won't work even if you do everything right), suppose that the programmers are expected utility maximizers and Bayesians, and that the utility function U-h(x) of a human can be factored into the additive sum of subfunctions V1-h(x), V2-h(x)... V10-h(x). We will suppose that U-h(x) is identical for all humans including the programmers, obviating many of the complex questions that enter into collective volition. Suppose also that humans have no explicit knowledge of their own utility functions, and can only infer them by observing their own choices, or imagining choices, and trying to build a model of their own utility function. They do, however, know the abstract fact that they are expected utility maximizers and that they possess some utility function U-h(x) that is the same for all humans and that is factorizable into a set of utility functions V-h(x), etc.
And the humans are also capable (this is an extra specification, because a standard expected utility maximizer does not include this ability) of *abstracting* over uncertain states with uncertain utilities, and taking this into account in their planning. For example...
More at thread link: http://sl4.org/archive/0409/9870.html
Q: Can we escape from the danger from nanotech by creating FAI?
A: Building a nanofactory does not prevent anyone else from creating an AI. Quite the opposite. It gives them 10^25 ops/sec or some other ridiculous amount of computing power that lets them brute-force the problem even if they don't have the vaguest idea of what they're doing. If you solve the FAI problem, you probably solve the nanotech problem. If you solve the nanotech problem, you probably make the AI problem much worse. My preference for solving the AI problem as quickly as possible has nothing to do with the relative danger of AI and nanotech. It's about the optimal ordering of AI and nanotech.
Thread link: http://sl4.org/archive/0409/9869.html
Q: Can we escape from the danger of UFAI by running away?
A: Suppose you pack your bags and run away at .99c. I know too little to compute the fraction of UFAIs randomly selected from the class that meddling dabblers are likely to create, that would run after you at .995c. But I guess that the fraction is very high. Why would a paperclip maximizer do this? Because you might compete with it for paperclip resources if you escaped. If you have any hope of creating an FAI on board your fleeing vessel, the future of almost any UFAI that doesn't slip out of the universe entirely (and those might not present a danger in the first place) is more secure if it kills you than if it lets you flee. [...]
Suppose it doesn't run after you. In that case, if more than one group escapes, say, 10 groups, then any one of them can also potentially create an UFAI that will chase after you at .995c.
Suppose only one group escapes. If you have any kind of potential for growth, any ability to colonize the galaxy and turn into something interesting, you *still* have to solve the FAI problem before you can do it.
Running away is a good strategy for dealing with bioviruses and military nanotech. AI rather less so.
Thread link: http://sl4.org/archive/0409/9872.html
Q: Why haven't FAI ideas been explained in more detail?
Q: Does consciousness exist or not? If it does exist, what sort of physical entity is it, and how are the various subjective attributes of consciousness explained? If it does not exist, then in what terms am I to interpret the fact of my own existence, and my awareness of it?
A: I need to write this up. And so, when I saw that this conversation was starting yet again, I went back to working on... an old paper that doesn't even explain this issue, and is only peripherally related to this issue, but that I'd need to finish before I could start working on a document that explains this issue. The paper is called "A Human's Guide To Words"; it's currently 170K of HTML, and almost finished I think, but going very slowly. I've been fiddling with it occasionally for about a year.
I find it increasingly hard to justify any effort that is not on a direct line to FAI. It's a fascinating problem, but *explaining* the problem is not on a direct line to FAI, unless someone offers me a large incentive to do so. If I can finish and publish "A Human's Guide to Words", that would provide a reader some conceptual equipment that would hopefully enable me to explain my answer in not too great a time.
But briefly: Qualia are not ontologically basic. "Qualia" are the result of the human brain doing something weird in how it processes reflectivity. For any puzzle that is apparently about qualia, you need to replace it with a puzzle about the behavior of intelligent minds in talking about qualia or discussing qualia. If I ask "Why does Mitchell claim the sky is blue, rather than green?" I can transparently eliminate Mitchell's cognition from the problem and the question reduces directly to "Why is the sky blue, rather than green?" You can't do this with problems that appear to be about something called qualia. "Why do I think qualia are unitary?" does not reduce to "Why are qualia unitary?"
Thread link: broken in archive. Original Subject: "Re: Human mind Turing computable according to Eliezer", Original Date: Wed Oct 06 2004 - 20:00:20 MDT
Q: How do I know, despite sincere intentions of the organization to save the world, that the world won't "drift toward tragedy" as a result of FAI research made possible in part by my donation?
A: [... P]eople spend a heck of a lot more than ten dollars trying to lose five pounds, based on schemes with a heck of a lot less rational justification than SIAI has offered. The possibility of humanity being wiped out seems to have less psychological force than the opportunity to lose five pounds. No matter how much we grow, I don't think we'll match the membership or resource expenditure of any major weight-loss meme. That's just not psychologically realistic given human nature. Now I do not think that so much resources should be required. I'll be surprised if we need more than two percent of the cost of a B-2 bomber.
[How do you know?] You guess, choosing a policy such that you would expect Earth to reliably survive technically intricate existential threats if everyone followed your rule. It's irrational to allocate billions of dollars to publicly understandable but slight risks, and less than a million dollars to a much worse risk where it's harder for a member of the general public to understand the internals.
More at thread link: http://sl4.org/archive/0410/10023.html
Q: What makes your organization different from, say, an organization that also claims to save the world, but by different means, like prayer, for instance?
A: Rationality. Prayer doesn't work given our universe's laws of physics, and that makes it an invalid solution no matter what the morality. [This is] *exactly* the same argument that people use against cryonics or nanotechnology.
Bayesians may and must look at what other Bayesians think and account it as evidence. ("Must", because a Bayesian is commanded to take every scrap of available information into account, ignoring none upon peril of paradox. Jaynes 1:14.) Robert Aumann once proved that perfect Bayesians cannot agree to disagree; they must have the same probability judgment once they both know the other's, both know the other knows theirs, etc. See e.g. http://hanson.gmu.edu/deceive.pdf. Nick Bostrom and I, on the way back from Extro 5, tried to split a taxi ride and found an extra $20 bill in our contributions. I thought the $20 was his, he thought it was mine. We both knew what we had to do. He named the probability he thought it was his (20%), I named the probability I thought it was mine (15%), and we split it in 20:15 ratio. Guessing how much other people know relative to you is not faith, so long as you pursue it as a question of simple fact, taking into account neither personal likes nor personal dislikes.
Thread link: http://sl4.org/archive/0410/10023.html
Q: Why don't you write more about FAI?
A: Which Existential Risks reality throws at you is completely independent of your ability to understand them; you have no right to expect the two variables to correlate. Do you think it the best policy for Earth's survival that nobody ever support an existential-risk-management project unless they can comprehend all the science involved? We'd better hope there's not a single Existential Risk out there that's hard to understand. If it requires an entire semester of college to explain, we're doomed.
I've tried very hard to explain what we're doing and how, but I also have to do the actual work, and I'm becoming increasingly nervous about time. No matter how much I write, it will always be possible for people to demand more. At some point I have to say, "I've written something but not everything, and no matter what else I write, it will still be 'something but not everything'."
Thread link: http://sl4.org/archive/0410/10023.html
Q: Why does the SIAI team would need so much money to continue building FAI if the difficulty of creating it does not lie in hardware? What are the real costs?
A: Extremely smart programmers. Who said anything about needing "so much money"? I expect to save the world on a ridiculously low budget, but it will still have to be one to two orders of magnitude higher than it is now. Hiring extremely smart programmers does take more money than the trickle we have currently.
Thread link: http://sl4.org/archive/0410/10023.html
Q: Is it wrong to pursue fame by being an SIAI supporter?
A: I think better of someone who lusts after fame and contributes a hundred bucks than a pure altruist who never gets around to it. I don't think that counts as saying that the end justifies the means. The other way around: By their fruits ye shall know them.
Thread link: http://sl4.org/archive/0410/10023.html
Q: Isn't it wrong in entrusting a small group of people with the fate of entire human race, as encapsulated in FAI?
A: I see something wrong with giving a small group of people the ability to command the rest of the human race, hence the collective volition model [that avoids doing just that]. As for *entrusting* the future - not to exercise humanity's decisions, but to make sure humanity exercises them - I will use whichever strategy seems to offer the greatest probability of success, including blinding the programmers as to the extrapolated future, keeping the programmers isolated from a Last Judge who can only return one bit of information, etc. Or not, if I think of a better way.
The alternative appears to be entrusting small groups of people who aren't even trying to solve the problem with the fate of the entire human race. That looks to me like a guaranteed loss and I'm not willing to accept that ending.
Thread link: http://sl4.org/archive/0410/10023.html
Q: What would you tell people objecting to that idea of FAI?
A: I'm sorry. Someone has to do something if any of us are going to survive, and this is the best way I've been able to find. You object but I have not heard you say a better alternative, unless it is letting catastrophe go its way unhindered. You can't argue fine points of moral dilemmas if you're dead.
Thread link: http://sl4.org/archive/0410/10023.html
Q: Do we have the right to end the world as we know it by creating FAI without the world's approval?
A: There are no rights, only responsibilities. I'll turn the question over to a collective volition if I can, but even then the moral dilemma remains, it's just not me who has to decide it.
The question is not whether the world "as we know it" ends, for it always does, generation after generation, and each new generation acts surprised by this. The question is what comes after.
Thread link: http://sl4.org/archive/0410/10023.html
Q: Should I support SIAI in its development of FAI?
A: Deciding whether to try to save the human species is an extremely complicated question. You can get so tangled up in the intricacies that you forget the answer is obviously yes.
Part of the problem that transhumanist organizations have in organizing, I think, is that when a new effort tries to launch, we hear from the critics but not all the people who approve; it creates a bias against anything getting started.
SIAI *is* getting new donors as a result of [its] efforts - though I won't [be able to] tell you how many until it's over. It's *your* opportunity [to help], not anyone else's.
Thread link: http://sl4.org/archive/0410/10023.html
Q: What are the problems FAI will try to solve in a nutshell?
A: (Michael Roy Ames) Unwanted human death and suffering are the problems FAI will try to solve in a nutshell. Why 'unwanted' rather than 'all'? There will be some human individuals who want to die, and many who will want the option to die at some time in their future. There will also be many people who will want to allow personal suffering into their lives, as this has been an integral part of human life since our species began and may in fact be required for normal human development to occur.
Thread link: none
Q: Why is Eliezer Yudkowsky focusing on FAI alone, rather than making a lot of money first?
A: I used to think that way. In 1998, when I published "Coding a Transhuman AI"... it feels like centuries ago... Brian Atkins wrote me and said, d'you suppose this is something that a small team of programmers could do? And I said, nah, the Singularity is thirty years off, wanna start a business instead? Then we'd have enough money to launch the planet-sized Manhattan project it would take to shorten the Singularity ETA by a few years. And Brian Atkins said, sure. It was the summer of the dot-com era. So I came up with a bright idea I'd never seen done before, and spent the next six months studying venture capital, marketing, markets, IPOs, corporate structures, and writing a business plan. I can't remember whether it was Brian Atkins or me who first realized it wasn't going to work, but the realization was nearly simultaneous.
Developing a new technology and launching a corporation is really, really, really, really, really, really, really, really, really, really, really, really, hard. If you start a company, it's your life. You don't do it on the way to something else. [...]
More at thread link: http://sl4.org/archive/0410/10054.html
Q: When will FAI thoery be complete, or in a state complete enough to build?
Q: What part of FAI is least supported by evidence and taken most on faith?
A: The weakness is that I am not finished, and the part that must be "taken on faith" (leaving aside certain objections to [the] terminology) is that I shall ever finish. An example: In my lap right now is a spiral paper notebook whose showing page contains an incomprehensible doodle of lines, filled and empty boxes, and the words "Fred", "CV", and "Motie". This is the debris of me trying to figure out how to describe Fred's preference that a Fred-similar system (say, a collective volition) make a decision, rather than a Motie (an alien with a nonhuman Goal System). I know how to describe this problem if Fred can exactly simulate CV and Motie to extrapolate their decisions, and then extrapolate the consequences and expected utility of CV and Motie's decisions; Fred can then simply choose directly, by proxy, as predictable as pressing a button. But if Fred doesn't know the choices faced by CV and Motie, or Fred can't personally extrapolate the consequences of the choices, or if Fred has only an abstract model of CV and Motie, then it will take some new theory I haven't seen to give a step-by-step walkthrough of exactly how Fred computes his preference for CV-decision over Motie-decision. It's a problem directly related to the difficulty I posted with Schmidhuber's Godel Machine. I don't presently know the solution, and it must be "taken on faith", as it were, that I ever shall know. I may know by the end of this day, or by the end of this year, or one of the other six billion humans out there may have already solved the problem and I haven't read the paper, or perhaps this will be the challenge that ultimately defeats me and I shall never know at all, though I rather doubt it.
Thread link: http://sl4.org/archive/0410/10107.html
Q: Should Eliezer, or other SIAI researchers obtain a Ph.D.?
A: I have a suspicion that this would take a downright amazing amount of time [to obtain a Ph.D.] - on the order of four years or more. [snip] If I wanted academic respect and a Ph.D., I'd write on a subject that I already fully understood, people were already interested in, and that was at least theoretically possible to explain.
But oh, the time required! Gobs and endless gobs of time! I shudder just to think of it. And *it might still not work* [gain academic respect], and I'd *still* have to do all the FAI theory and I wouldn't be as young when I did. All my life has taught me the value of not being distracted. That's why I account it a last resort. [...]
More at thread link: http://sl4.org/archive/0410/10112.html
Q: Will FAI work if there are found to be physical effects occur in brains that do not occur elsewhere in the observable universe?
A: Back when I thought consciousness was strange physics, yeah, this was one of the mental tests I threw at FAI designs. But this is an example of a behavior that should arise naturally from a good design, *not* something that should need explicit patching - explicit patching indicates a poor design. I threw that challenge at potential FAI designs to see if they solved the problem naturally. In this case, if human brains involve strange physics and human consciousness intrinsically requires strange physics, then until you duplicate the strange physics and find some way to transfer state from one strange-physics phenomenon to another, any attempt to upload a human or even a laboratory sample of living brain tissue will fail visibly when it is impossible to formulate a non-strange-physics device that predicts the neurons' behaviors. An FAI's direct physical examination of neurons and of human brain designs should yield the same conclusion, that strange physics is involved and faithful uploading to a normal computing device is not possible, via abstract reasoning.
Incidentally, Roger Penrose explicitly does not deny that it is possible to create artificial consciousness. He just denies that you can do it on Turing machines. Penrose explicitly accepts that if you fathom the postulated strange physics of neurobiology, you should be able to do artificial consciousness.
More at thread link: http://sl4.org/archive/0410/10147.html
Q: Will FAI be proven mathematically to remain Friendly?
A: I was planning to put a mathematical upper bound on the probability of catastrophic failure within an expected operational lifetime before replacement. Impossibility is not a word to be used outside the context of physics.
Of course this just passes the buck of the fundamental safety problem, which then becomes defining "catastrophic failure" to include all catastrophic failures. Crying mathematical certainty doesn't help with this; it just excludes silly solutions that the speaker can't translate into silly math. If you make a big deal out of whether it's mathematical or not, you'll just get silly solutions translated into silly math and the speaker will say: lo, it is math. Rather go on attacking the fundamental safety problem in its new form, broadening your understanding of failure, seeing how simple definitions fail in simple loopholes. [...]
More at thread link: http://sl4.org/archive/0410/10182.html
(PAQs gleaned by Peter de Blanc)
Q: Why shouldn't the CV be sentient/experience qualia?
A: Because we don't want the CV to experience anguish, and we don't want to kill sentients. In the process of extrapolating our volition, the FAI will need to simulate our thought processes. If these simulations are sentient beings, then we would be responsible for the suffering they experience.
Thread Link: http://www.sl4.org/archive/0406/8943.html
Q: What if my volition isn't what I want?
A: If you have a utility function, then your current decision is whatever you expect to maximize that function. Your volition is what would _actually_ maximize that function.
Unfortunately, human beings do not use utility functions, so the answer becomes more complex. Extrapolated Volition attempts to preserve the values we perceive in our decisionmaking processes, while discarding heuristics which are broken and providing additional knowledge and processing power.
It is worth noting that CEV is not a proposition for a regime, and it is not proposed that your extrapolated volition should run your life. What is proposed is that, if you knew more and thought faster, you would be more qualified to create an FAI than you are presently. CEV asks what kind of FAI humanity would create under such conditions.
Thread Links: http://www.sl4.org/archive/0406/9094.html http://www.sl4.org/archive/0406/9268.html
Q: Why should the AI care about the goal we give it? Won't it pursue its own interests?
A: Think of an AI as a ReallyPowerfulOptimizationProcess (RPOP). An RPOP is, ideally, a machine which infers the results of all possible actions, evaluates a utility function over these results, and selects the action with the highest expected utility. If the RPOP's utility function is the number of paperclips in the universe, then it is unlikely to change its utility function, unless by doing so it can increase the number of paperclips in the universe.
Thread Link: http://www.sl4.org/archive/0406/9400.html
Q: Why would a smart AI do something dumb like turning the universe into paperclips?
A: Turning the universe into paperclips is not inherently dumb; it is only dumb relative to a goal which does not involve turning the universe into paperclips. Human beings have complex goals which do not involve this; if we model the AI the same way we model humans, then we may assume that it also has such complex goals, but this is not necessarily the case for Minds-In-General.
Q: Why not create multiple AIs, so that no one has too much power?
A: If we are talking about FAI, then creating multiple AIs increases the number of AIs that can fail. If attack is physically easier than defense, then a single Unfriendly AI may be able to destroy humanity even in the presence of multiple FAIs. The more AIs we create, the greater the risk that one of them will be Unfriendly.
If defense is easier than attack, and multiple FAIs would reliably be able to overcome a single UFAI, then the redundancy may increase safety in a slow-takeoff scenario. In a Hard Takeoff, we would expect the fastest (and probably least cautious) AI to take off first, probably resulting in a bad outcome.
Even if multiple FAIs do increase safety compared to a single FAI of the same size, they are unlikely to be as safe as one big FAI with the increased self-reflectivity and better error-checking mechanisms that this entails.
Thread Link: http://www.sl4.org/archive/0405/8828.html
Q: Would a CV decide to take on a role as advisor to human beings, without directly intervening in human affairs?
A: Although we don't know for sure what our volition would decide, this does not look like a very good outcome from our present perspective. A purely advisory CV would lack the power to effectively protect human beings. Even if our CV could give us the technology to cure aging, and even if this technology were immediately distributed to everyone, this would still leave many other causes of death, and the threat of UFAI would still remain.
Neither does this scenario enhance human self-determination. Presently, human beings live under a set of background rules which place many restrictions on the choices available, such as "You must spend most of your hours on boring, soul-draining labor just to make enough money to get by" and "As time goes on you will slowly age, lose neurons, and die." By changing these background rules, it would be possible to increase the number of choices available to each individual. This would increase the self-determination of humanity as a whole as well, since this is just the product of the self-determination of each human.
Thread Link: http://www.sl4.org/archive/0405/8884.html
Q: If people treat an AI like dirt, why would it be nice to us?
A: As human beings, we do not like to be treated like dirt because, in the past, human beings who did not mind being treated like dirt would have been less likely to survive and produce offspring. The way an AGI would respond to human mistreatment will depend on its Goal System. An AGI which has been designed to be friendly to humans if and only if humans are kind to it is not likely to respond well to abuse, but there's no obvious reason why anyone would want to create such an AGI.
Thread Link: http://www.sl4.org/archive/0406/9255.html
Q: Why not just specify the outcome we want?
Q: Human beings should have a right of withdrawal, shouldn't they?
Q: What can't we have a bill of rights for sentients?
A: To create a bill of rights or a set of laws, the human programmers need to make many choices which will be set into stone: which rights do sentients have, what constitutes a sentient, what happens when there is a conflict between the rights of two sentients?
These are all details which the programmer would need to dictate, but what makes the programmer so qualified to decide the fate of humanity? Under collective volition, humanity as a whole can decide which rights individual humans should have.
Thread Links: http://www.sl4.org/archive/0405/8889.html http://www.sl4.org/archive/0406/9149.html Collective Volition - Q4
Q: Doesn't CV require more computing power than UFAI? What will you do in the intervening time?
A: Given a Hard Takeoff, this is not an issue. Once a Seed AI is able to self-improve, it will be able to acquire whatever computing power it needs. Once UFAI exists at all, it is already too late; humanity has lost. Conversely, once FAI exists, we have already won.
Thread Link: http://www.sl4.org/archive/0405/8889.html
Q: What is wrong with Perspex Space as a theory of mind?
A: Perspex space is simply another Turing-complete environment, like LISP or Java but less comprehensible. Nobody has described how to actually implement a mind using Perspex Space, nor made any specific predictions based on Perspex theory.
Thread Link: http://www.sl4.org/archive/0502/10846.html
Q: Why not use the SL4 list to develop AGI and FAI theory?
A: The amount knowledge required to contribute meaningfully is very high, as explained here. It's unlikely that any casual list readers would be able to discover anything new.
Furthermore, Seed AI is dangerous, and publishing AGI theory on SL4 increases the probability that someone will successfully create a Seed AI without worrying about Friendliness.
Thread Link: http://www.sl4.org/archive/0502/10907.html
Q: What resources are available for Bayescraft training?
A: The following books are available:
- Rational Choice in an Uncertain World. - Robyn Dawes
- Judgment Under Uncertainty: Heuristics and Biases. - Daniel Kahneman, Paul Slovic, Amos Tversky
- Heuristics and Biases: The Psychology of Intuitive Judgment. - Thomas Gilovich, Dale Griffin, Daniel Kahneman
- Choices, Values, and Frames. - Daniel Kahneman, Amos Tversky
And if you haven't already read them: An Intuitive Explanation of Bayesian Reasoning A Technical Explanation of Technical Explanation
Thread Link: http://www.sl4.org/archive/0503/11102.html
(PAQs gleaned by Edmund Scheafer) Note: These aren't direct quotes from Yudkowsky, unlike most the PAQs above. The original quotes are likely to be better. -- Schaefer
Q: Why not build a weakly self-improving AI to achieve just enough rudimentary superintelligence to solve the remaining problems of Friendliness?
A: If the seed AI is not Friendly, there is a danger that the initial Goal System (solving the problems of Friendliness) will be lost during Recursive Self-Improvement. Also, the problem of how to infer well-formed descriptions of problems from vague human questions is not an easy one, and would need to be solved first for a seed AI to obtain a well-defined goal from our descriptions of Friendliness.
Thread Link: http://www.sl4.org/archive/0501/10512.html
Q: Does the seed AI need to be proven to undergo Hard Takeoff?
A: No, what must be proven is that the AI will remain Friendly if it does undergo Hard Takeoff. While the seed AI will likely undergo Hard Takeoff, proving it will do so isn't nearly as important, even if it's possible.
Thread Link: http://www.sl4.org/archive/0501/10520.html http://www.sl4.org/archive/0501/10517.html
Q: Shouldn't a Friendly AI respect the wishes of people who don't want their lives interfered with by AIs? Aren't values like freedom and self-determination more important than the collective will of the human race? What's wrong with Russell Wallace's proposal in "Domain Protection" (http://homepage.eircom.net/~russell12/dp.html)?
A: The key problem with setting rules and restrictions on FAI, on planning what things we definitely want to protect in our future no matter what, is that we are still fallible humans. Few of these issues have cut-and-dry answers.
For example, if we make an FAI that allows rational, consenting adults (say, the Amish) to choose to have no AI interference in their lives, we're forced to draw a line somewhere between those rational adults and brain-damaged infants and paranoid schizophrenics incapable of consenting to be helped. Do we want to live a world where no baby or toddler will ever become anything even remotely resembling a human, because what they "really want" is to eat cookies, and no adult has a right to override their personal volition?
This is just one example of many. How are resources divided among people, if infinite resources turns out not to be possible? Can godlike transhumans communicate freely with regular humans, considering that they may use powerful new forms of persuasion and manipulation that we never thought of? At what age should a person have his/her volition respected: 18, the U.S. age of majority? 13, the traditional Jewish age of majority for males? Birth? The third trimester of pregnancy?
The problem with deciding these issues now, and the problem with Wallace's Domain Protection, is that each one of these questions is a potential point of failure for the programmers. If the human programmers overlook something while designing a utopia, humanity lives with their mistakes for all eternity. Humanity should not spend the next thousand years wishing that the FAI team had done things differently. CEV must be able to fix the mistakes of its designers. Domain Protection's biggest oversight is that it contains no mechanism for abandoning the framework it describes in favor of something else if it turns out that people really want something else later on, no matter how horrible the system's unintended consequences may be.
Thread: http://www.sl4.org/archive/0505/11308.html
Q: Why is molecular nanotechnology such a major threat to humanity?
A: Molecular nanotech allows for unprecedented increases in computing speed, and on such computers one can brute-force the problem of creating AI through evolutionary design processes. AI made without human design is not likely to be Friendly AI, and is thus dangerous.
Unstoppable nanotech replicators (a.k.a. "grey goo") or military nanotech weapons are more commonly cited dangers, but neither is as simple and commercially attractive as using MNT to build faster computers. UnFriendly AI is the real danger, and nanotechnology only because it enables UFAI.
Thread: http://www.sl4.org/archive/0506/11398.html
