Singularitarian FAQ
From The Transhumanist Wiki
Here's a very rough and incomplete version of a Singularitarian FAQ. I'll be adding in more answers over the course of the next few weeks, feel free to join in! Relevant discussion can take place at the bottom. -- Michael Anissimov
Singularitarian FAQ, Singularity FAQ, or some mixture? ie. should we focus mainly on the concepts, mentioning only that "other people agree with this too (like us! *wave*), here they are if you want to cooperate"? it's not primarily a movement to join, but an end to achieve. We might like to add references to each question (a bit like the transhumanit FAQ?), many can be expanded into one or more essays at some point in the future.
I'm working on updating this... we'll see how well I go! -- Nick Hay
Singularitarian FAQ
-Introduction
-Singularitarianism
- What is Singularitarianism?
- Why do Singularitarians think greater-than-human intelligence will be so powerful?
- What makes you think they will stay benevolent indefinitely?
- How do Singularitarians propose to spark the Singularity?
- When do Singularitarians think the Singularity might occur?
- What do Singularitarians plan to do after the Singularity?
- What are the Singularitarian Principles?
- Why should I care about Singularitarianism?
- What further developments might take place within Singularitarianism?
- What other views do Singularitarians have in common?
-Concepts and Terms
- What is “Artificial General Intelligence”?
- What is a “Hard Takeoff”?
- What is a “superintelligence”?
- What is computronium?
- What do “ve”, “vis”, and “ver” mean?
- What is a “Friendly AI”?
- What is a “seed AI”?
- What is “strong self-improvement”?
-Singularity Institute for Artificial Intelligence
- What is the Singularity Institute?
- What has the Singularity Institute accomplished so far?
- What is so special about SIAI’s approach to AI?
- What are the Singularity Institute’s plans for the future?
- How can I help?
Introduction
This FAQ was written to inform readers about Singularitarians, their goals, reasoning, and terminology. In this FAQ, we try to focus more on the philosophical and activist movement of Singularitarianism alone, rather than the incredibly complex and controversial issues surrounding Artificial Intelligence and the Singularity in general.
Singularitarianism
What is Singularitarianism?
Singularitarianism is the pursuit of the technological creation of smarter-than-human intelligence. “Singularitarianism”, referring to the philosophy of increasing humanity’s intelligence, was coined by Eliezer Yudkowsky on the Extropians mailing list in mid-1996. The word “Singularitarian” derives from “Singularity” - a term coined by Vernor Vinge in 1984 which means the creation of greater-than-human intelligence, as an analogy to describe the way human understanding would break down in trying to understand the cognitive processes of minds with superhuman intelligence. Singularitarianism pursues its goals in the scientific, democratic, and rationalist tradition of humanism. The primary motivation behind Singularitarianism is altruism, the idea that many present-day human problems could be solved with the extra intellectual firepower of benevolent minds with superhuman intelligence. Since intelligent cognition is the workhorse behind the accomplishment of all goals, making the fundamental improvement of human intelligence a specific goal is an investment with immense humanitarian potential.
Why do Singularitarians think greater-than-human intelligence will be so powerful?
Relative to what is physically possible, evolutions’ design of the human mind was relatively ad hoc, slow, and tuned for maximizing our reproduction and survival rather than our best interests. The great bulk of the human brain’s design took place during our hunter-gatherer phase, so our minds are optimized for living in a highly primitive environment. Most of the brain evolved in the absence of General Intelligence and consciousness, so the more recent evolutionary innovations were forced to make use of outdated cognitive machinery. Hardware improvements to intelligence conducted by intelligent entities themselves can potentially avoid the pitfalls and constraints of evolution. We can test, analyze, and enhance the bundle of algorithms responsible for human intelligence, and implement those algorithms on faster, more robust hardware. Smarter-than-human minds created through human knowledge and technology will have access to the reasoning which led to their construction, and will be able to apply this reasoning to create iteratively better cognitive designs. This process is commonly called “strong self-improvement” to distinguish it from the relatively superficial improvements that humans introspectively conduct on our own thought patterns.
Why do Singularitarians think that greater-than-human intelligence will be benevolent towards humans?
The motivations and actions of a given mind emerge from its hardware-level design. No matter how hard an ape tries to appreciate human music, it will always fail, limited by the fact that it lacks the necessary neurological hardware for appreciating human music. Similarly, human beings evolved to be deceptive, cunning, and potentially vicious animals. Regardless of whether these tendencies manifest themselves behaviorally, the tendency exists within every neurologically normal human being. If the first smarter-than-human intelligence is a genetically or cybernetically modified human being, then that human being will presumably have the same potential for good or evil that other humans do, unless deliberate revisions are undertaken to decrease her tendency towards violating the rights of other humans. If the first smarter-than-human intelligence happens to be an artificial intelligence, then the designers would need to plan on creating a benevolent mind from the beginning, leaving out the portions of human nature responsible for negative actions, endowing the AI with its own sense of morality, and encouraging that AI to improve its moral model based upon input from other intelligent beings. We tend not to think of morality as something which can be engineered or modified, but the causes of good and bad deeds are in the makeup of our brains, just like all our other behaviors. Singularitarians do not assume that a superintelligent mind would become benevolent automatically, and work to develop the relevant theories for success in this area. 100% success will never be guaranteed, but we feel an obligation to try, because the eventual creation of superhuman intelligence is inevitable, because regulation would be impractical.
What makes you think they will stay benevolent indefinitely?
Humans have a history of abusing the words “good” or “benevolent”. Some of us use these words in political situations for our own personal advantage, or simply whenever it suits us. We crudely coerce others physically or psychologically because we say or feel it’s “what’s best for them”. But just because humans sometimes behave this way doesn’t mean that all minds would. A mind that is benevolent by design and independent from the moral pitfalls of selfishness would only work specifically for itself when it served the greater good, and wouldn’t try to modify itself into a state of greater selfishness, because it would realize that selfishness isn’t kind to others. Same goes for immorality that emerges as a side-effect from ignorance, apathy, or other sources; a superintelligent, supermoral mind would be better at analyzing its own flaws and mistakes, improving upon its own design in order to minimize them. The greater the considered action’s importance or potential for error, the more likely a supermoral mind would be to never conduct that action in the first place, or conduct a variant on that action possessing far less risk. Human scrupulousness drifts up and down, back and forth, depending on the situation and our attitude, mood, and emotions, but a superhuman mind could revise the negative aspects of these mental changes, perpetually making closer approximations to a pansentient moral ideal. If no such ideal exists, a superintelligent mind would simply resort to only conducting those actions which are unambiguously moral - even humans can agree on a basic set of moral principles, although we commonly disagree on how they should be carried out. Since each subsequent design improvement will be conducted based on the moral knowledge of the previous design - with probabilistic errors factored in - a drift towards selfishness or indifference to humans could only take place entirely against the will of the benevolent mind in question; an unlikely prospect if the mind already understands all the intricacies of its own design.
Is benevolent superintelligence worth the risk?
How do Singularitarians propose to spark the Singularity? How do we create this benevolent superintelligence?
Although a number of different approaches have been proposed by various authors, Artificial Intelligence appears to be the closest technologically, the cheapest, fastest, and the least subject to future regulation. Artificial Intelligence also minimizes the potential problem of human error in the critical steps from human-equivalency to transhumanity by delegating the self-improvement strategy to a mind specialized from the codebase up for beneficient and rational self-enhancement. Although its public acceptance is hampered by stigma derived from anthropomorphic sci-fi stories, there are strong arguments for the near term (one to two decade) acheivability of AI, and therefore Singularitarians feel a responsibility to ensure the sustainable benevolence of the first human-equivalent AI. The most commonly proposed strategy is the construction of a seed AI; an AI with the capacity for fine-grained self-understanding, self-modification, and open-ended self-improvement. Seed AI is not any specific AI theory, but a general AI paradigm that becomes more useful as the AI itself becomes smarter and is able to better assist in its own creation.
What is the difference between the "activist" perspective and the "futurist" perspective?
When do Singularitarians think the Singularity might occur?
Predictions range between today and 2060. Singularitarians generally don't like to try and predict the Singularity because it gives people the wrong impression. The Singularity will come about when people create it, and that's all there is to it.
What do Singularitarians plan to do after the Singularity?
Trick question. A good answer might be “enhance our intelligence, then go from there”. We can’t predict what will happen after we enhance our intelligence because the specific effects of increased intelligence are fundamentally unknowable. Apes could never have predicted human society, or human goals and motivations. We can’t even declare or predict our own actions, because we might obtain new information which results in a change in our entire worldview. But it’s likely that, like many people, Singularitarians will want to continue to learn, grow, interact, engage in new experiences, and simply have fun.
What are the “Singularitarian Principles”?
The “Singularitarian Principles” is a document that Eliezer Yudkowsky wrote as an outline of suggested Singularitarian values. Yudkowsky has no particular authority within Singularitarianism, but the values he lays out have been popular thus far, and therefore worth mentioning. The definitional principles are Singularity, Activism, Ultratechnology, and Globalism, and the descriptive principles are apotheosis, solidarity, intelligence, independence, and nonsuppression.
Why should I care about Singularitarianism?
The well-being of yourself and all of mankind depends on a benevolent Singularity happening, etc.
What further developments might take place within Singularitarianism?
The Singularity, for one. Having a stable movement (?) with fewer (and lighter) single points of failure, more people actively helping, and a more accelerated Friendly AI project without blocker problems (eg. theory, funding, people).
What other views do Singularitarians tend to have in common?
Usefulness of Evolutionary Psychology, unknowability of increased intelligence, rationalism?
Concepts and Terms
What is “Artificial General Intelligence”?
As opposed to narrow-domain intelligence.
What is a "Hard Takeoff"?
The projected model of self-improving AI in which a "conservative" timeframe for ascent to superintelligence is a given, which is important from an altruistic perspective because it emphasizes that irrecoverable errors in the first seed AI must be avoided at all costs. From the viewpoint of a simple human being, a complete transformation of the world could take days, hours, minutes, or maybe even seconds.
What is "superintelligence"?
A superintelligence is super intelligent. A superintelligence is not an ultrafast superintelligent moron. A superintelligence can take everything into account that a human being could possibly think of, plus many orders of magnitude more. A superintelligence sees your move coming a mile away - in fact, a superintelligence would probably be able to simulate you and your environment to a sufficiently accurate measurement that ve would know what you are going to do in many years. A superintelligence is not “like me, but with more X”. A superintelligence is different in kind than anything ever seen before on the history of the planet.
What is computronium?
A form of matter optimized to yield the highest degree of information processing.
What do “ve”, “vis”, and “ver” mean?
Gender-Neutral Pronouns. Most often used to refer to a non-human intelligence.
What is a Friendly AI?
An AI that can reason morally and play a positive rule in the future of the universe.
What is a Seed AI?
Sometimes not capitalized, a Seed AI is an AI that can improve verself.
What is strong self-improvement?
Strong self-improvement is the difference between human culture or human self-improvement and a Seed AI or a seed upload.
Singularity Institute for Artificial Intelligence
What is the Singularity Institute?
The Singularity Institute for Artificial Intelligence (SIAI) is the only formal organization dedicated explicitly to pursuing and achieving the Singularity.
What is so special about the SIAI’s approach to AI?
Takes into account scientific discoveries about how the brain *actually* works (ie. cogsci, EP, neuroscience, ...), takes the moral aspect "Is this really a good thing? How can we ensure it's better than anything else?" of AI development very seriously?
What are the Singularity Institute’s plans for the future?
Get a programming team together.
How can I help?
There are dozens of ways to help (see Singularity Tasks), but, unsurprisingly, the thing the Singularity movement needs most right now is funding.
~END~
~BEGIN DISCUSSION~
Are these questions appropriate? -->
How is Singularitarianism related to Transhumanism? -It might be silly to compare ourselves to others explicitly.
Is Singularitarianism a religion/cult/etc? -Do we really have to counter this accusation? Maybe in the form of a more subtle question.
What is a "SysOp Scenario"? -I'm not sure I like the SysOp Scenario concept too much, because it throws everybody off immediately and kills the conversation.
Is it right to have a "Singularity Institute" section?
Looks like the spacing on this page is all wacky and crunched. Oh well, too tired to fix right now, will do so soon. --Michael Anissimov
Yay! Much better than the one I threw up a long time ago. Thanks for making it decent, Michael. But where'd all the humor go? -- Gordon Worley
Thanks for doing the formatting, it looks way better now! Ah, I must have been in a very "serious" mood while writing this, and I agree that the humor is missing. As I start editing this, I'll be sure to lighten up the tone to make it more readable and fun. -- Michael Anissimov
'Some of the questions at Singularity Questions may be appropriate for this FAQ. ('Great job on it so far, Michael.) --Anand
---
Great FAQ! A couple of comments:
"100% success will never be guaranteed, but we feel an obligation to try, because the eventual creation of superhuman intelligence is inevitable, because regulation would be impractical." Saying "regulation is impractical" sounds like we're avoiding the possibility because it's difficult, or because we're rationalising it's difficulty. Doesn't take into account the significant possibility of extinction, and doesn't make explicit that it's the best of all alternate approaches.
"...even humans can agree on a basic set of moral principles, although we commonly disagree on how they should be carried out" Perhaps mentioning how much of this disagreement is caused about disagreement on facts, much less troublesome than disagreement on morals.
"...and the least subject to future regulation." This feels a little like we're trying to slip this past the regulators without them noticing, which on the face of it doesn't have the right connotations.
"What do Singularitarians plan to do after the Singularity?" Perhaps a mention of how, post-Singularity, there is absolutely no difference between (what were) Singularitarians and non-Singularitarians. This is perhaps a bit defensive (people finding religious analogies), however. Perhaps just a rephrasing to make it more clear the inclusiveness?
"What is "superintelligence"?" The rely here sounds a little too enthusiastic.
I added a couple of suggestions to "What makes SIAI different?"
-- Nick Hay
