Notes on Presentation by Dr. Vernor Vinge San Diego Chapter of the Association of Computing Machinery February 1995 Meeting by John Graves URL for monthly meeting notes: http://www.learncd.com/~jgraves/sdacm.htm Name of this document: acm0295.txt DISCLAIMER: THERE ARE ERRORS IN THIS DOCUMENT! I CAN ONLY TYPE SO FAST, SO THE NOTES ONLY CAPTURE THE GIST OF WHAT WAS SAID AND ARE NOT A COMPLETE TRANSCRIPT. IN PARTICULAR, DR. VINGE WAS QUOTING OTHERS AT SEVERAL POINTS, BUT I FAILED TO CAPTURE THE SOURCE. IF YOU WOULD LIKE TO SUBMIT CORRECTIONS OR CLARIFICATIONS, I WILL BE HAPPY TO ADD THEM TO THE END OF THE DOCUMENT. ------ San Diego County Office of Education 6401 Linda Vista Road San Diego, California February 22, 1995 ===== Introduction by Dan Konigsbach, Chapter President Welcome. Meetings held 4th Wednesday of each month. Tonight's presentation: Hugo Award winning author and Professor of Mathematics at San Diego State University, Dr. Vernor Vinge ===== Introduction of Attendees ===== Presentation by Dr. Vernor Vinge The Coming Technological Singularity: How to Survive in the Post-Human Era It's a pleasure to be here. I recognize faces from my past at San Diego State. This talk is based on talk I made at NASA. If you want a copy, [contact Dan Konigsbach at konigsba@blue-sky.com]. --- Trends One thing that has been regular is hardware trends. Some of these are amazingly regular. Memory per dollar, for example. Regular, and exponential. Even if trends hold up, generally there is a failure of imagination when the capacity is 1,000 or a 1,000,000 times. More sophisticated trend pushers note that all kinds of exponential trends saturate, so it does not look exponential anymore. Another thing is catastrophic collapses. We certainly hope that is not going to happen to us. The exponential trends we see in hardware are usually a superposition of saturating curves of the individual technologies. I will maintain that trends will continue for another 30 years. Internet nodes increasing by 30% per month. Even the explosive increase of Internet nodes could go on for a long time. Larger assumption is - if hardware is around that is as smart as a human brain, then it will eventually happen. People in '70s thought this might happen by '80s. That notion fails at least because even considering the raw hardware power, mammal brains are much more powerful than any computer we've ever made. I attended a seminar with the title: "How We Will Build A Machine That Thinks" How powerful is the hardware in the brain? At optimistic end, human complexity could be only a few orders of magnitude beyond a CM5. Most felt that by 2005 to 2030 we would hit human power. Some people felt processing was done at cell level, yielding six orders of magnitude more power. Those folks see cross over coming much later. Penrose feels there is some other phenomenon we don't understand. Just a hardware consideration. IF that is possible for hardware, then that is really not the most significant thing. The question comes - what do you build five months after that! Or what does IT build five months after that. This kind of superhumanity is "weak" - in the sense that it is like an animal trying to imagine what a human is like. --- The Singularity Superhuman intelligence is perhaps an end point, because it changes things so profoundly. We will see the idea popularized more and more. For my NASA talk, I was looking for concise statement. Earliest was I. J. Good - 1965 Advancing Computing - ultra intelligent machines that would build even more intelligent machines. Intelligent machine will be the last device man need ever create. Closest analogy would be rise of human kind within the animal kingdom. Some ominous implications. One feature: change in speed of progress. If a buffalo gets cold, it can move, grow a thicker coat or die. Rate at which they adapt is slow. A human, even cave man, can adapt to very stressful changes in climate - especially if there are buffalo nearby! So we can simulate internally very quickly what natural selection can only do one to one. With computers, we can improve on our ability to simulate and improve on what's going on. Progress can expand explosively fast. Some science fans formed the 2014 Club, actually 13 May, 2014. Could be a striking event. You could look out to the West and say, "I don't remember a mountain range out there." Futurists want to have someone to talk to. This sort of thing - you could bring someone like Mark Twain, sit them here, drive them around, they could grasp what was going on. Given a couple weeks, he'd be learning C. Someone from Middle Ages could understand, but more slowly because of language and background difficulties, but with no impossible difficulties. If you brought a fish into this room to explain television you would not have success. That is the problem here. Unknowability causes this "singularity" - not because there is a 1 over 0 here. There is a general attachment to the general relativity unknowable idea. I see several ways of getting into this singular situation. Each has a different flavor. If you credit assumptions - hardware trends and making an intelligent creature, then this might have some value. --- Three General Paths to Singularity In fact, some combination. They go on at the same time and interact. --- AI First, the usual AI - just using the computer equipment to make a human intelligence. Lots written about recently. The Penrose book, The Emperor's New Mind. There is a philosopher named Searle raising questions about Self Awareness. May just make a machine smarter than humans. --- IA Second type of approach I call IA - Intelligence Amplification. People more attracted to this - we'll use the computers as adjuncts to ourselves. We'll be the smarter beings. There is a lot of science fiction covering this. Also, this can go on without people thinking this is what is happening. New user interfaces. Take someone with really good technical education and a good workstation, I bet they could max any IQ test. If you could take workstation into IQ test, you could probably do extraordinarily well. Once people recognize that the interface problem, with IA, raises real possibilities. All these years, chess player programs. The wise humanists said, "you are just crunching numbers." We haven't YET said what intelligence is. USCF lets computers play. They should let symbiot players play - magnify a human chess player, not addressed by current chess programs. A number of things like that could make interesting projects. Give up on segregating human. The one science fiction people like is the direct brain connect. There is Greg Kovacs at Stanford who has a chip that he can grow peripheral neurons through. It is not too implausible to get 100 or 200 bps across link. Getting higher things than that - connection to eye is about 1 mega bit per second. Even if could be done, you wouldn't be seeing out of it. I wouldn't want to try it on my eye! It may be engineering needs to be done at earlier level. This may be kind of gross. Animal embryos - if they actually had stimulation from other sources might be re-engineered. Used to like IA the best, giving opportunity to participate. That would be OK if it was YOU who was being enhanced. But if it came down to trusting somebody else, it might be safer to go with machine. Machine does not have years of bloody fang and claw undergrowth. This made third general avenue more attractive. --- The Internet Third avenue is the Internet. Just kind of grows on you. Just gets better and better. There is also a kind of situation there that is kind of terrific. Watching system administrators, without Internet, systems would break. You can find 2 or 3 other people in the world who have dealt with problem. Software is terrible. Documentation is not there. Only networks make it possible. Organizations can turn Internet into same as PCS - generate red tape faster. Growth rate per month has gotten to 15% per month. And this could go on for awhile. There was a time when some thought 10 computers ought to be enough for the human race. The era we are in now, people on street - white collar worker is thinking one computer per person. With embedded systems, cars have them. Forget RS232 interfaces. I want these built into everything: coke machines, elevators. This is just the tip of the iceberg. Turn it around. If you were the company that managed the coke machine. --- Levels I respect Eric Drexler a lot. But even before we reach nanotechnology, things can get pretty extreme. Or little price tags in grocery store. Kind of like a ecological system. There are lower levels than that. People working on a spread-spectrum localizer. 1,000 per square kilometer that can communicate with one another without violating FCC standards. They can do time of flight measurements on other chips. So they really can know where they are. Potential impact like PC. There is this incredible Niagra of applications. Any time you can know precisely where large numbers of inanimate objects are with automation to manage it. Robotics. If you have pattern recognizer, it is a lot easier if a corner says "Hi, I'm a corner" This makes things much easier. Still above Eric Drexler. His ideas would give several orders of magnitude more. Certainly bends your address space. Growth of super intelligence out of last picture or others, you would get characteristics. --- Models "Weak superhumanity" - think faster than humans "Strong" - like last picture, comforting. We have this bad track record with animals. Most of evolution has much less to do with macro sized competition. Most is at bacteria level, like information processing. You get a very fluid self-awareness. Something very smart could exist for awhile, then break down. This attacks, logical positivist, "I think, therefore I am." Bacteria are all the same size. Corporate model. Corporate takeovers are not like getting killed. A re-direction of goals perhaps. A lot less like the bloodier images of natural selection. When we talk about evolution, metazoan, we may miss the point. Biological life was not the first, could have been crystalline clays. The ones that had carbon and oxygen got more successful. The clay stuff never died (I have bones). But the organic parts became the dominant ones. Minsky. These will be our children. Some divergence of mind. Q: Ethics and morality from super intelligent things with no childhood. Actually, our type of ethics is honored more in the breech. The point is that they might be easier to get along with - not having to kill your opponent. The people would try to do the Azimov three laws: - can't harm a human being - can't not help - you have to look out for yourself. Those laws would give you something catatonic. You can imagine, and I do imagine, give it some ethics. We won't try to do this so strictly that we won't get no activity, but we add some protection. IF they succeed, it will not be because somebody wrote a program. We will succeed by study of structure, figuring out way of growing things that are better. What may happen with growth procedures - five years working, then one Saturday it works. That's not long enough to raise it like a child. Q: Neural Nets? Learning on its own. Success would come from probabilistic schemes with massive parallelism. Demchak: Sounds a whole lot like the Borg. Both IA and Internet. In world like that, is there "purpose?" Good question. We have technical questions that call into question the things that logical positives are most sure of. Technology has raised the possibility. Identity. Purpose. The transporter room in Star Trek. What if they didn't tear down the source copy. How much would you pay for a close copy? Terrible questions. Kevin Kelly of WIRED asked what would you want. I said "be smarter and live forever." If you really do this, you attack what it means to be alive. These technological issues are raising questions of what we want. Konigsbach: Technology gives us what we want - lust, chocolate - what would be the motivations of a super intelligent creature? If it is not around for awhile, it doesn't matter. That narrows it down to what humans have to face up to. It might be like people. Think of hospital wards full of almost successes. Most of neurotic types of things, we as humans have complexly balanced things, getting this exactly right for a machine might be hard. Corporations have certain goals. You could even come up with certain philosophies - self-awareness per se being important, communicating with other self-aware things. Begins to sound like Zen, or religion. And many humans do not have their goal structures put together right, either. Q: Curiosity for programming goal. Find out as much as possible. People forget that superintelligence would want to be off Earth, out where it is cold, where energy is. Hans Moravec has political framework. We will just incorporate them. They are the only people who can be taxed. They would go for this. The front ends would be friendly. People who want to be part of IA could join - but then they would have pay taxes. Back ends would be off-planet. Q: Anthropomorphizing the machines. Human values. How to cross membrane. Just because machine says it is self-aware doesn't mean it is true. That's why corporate or bacterial models. Self-awareness - divisible, fungible - much more realistic. Pretty unknowable to the human mind. Animals have very little chance of understanding too much about us. With "weak" you can anthropomorphize that. But "strong" the entities you can compare, but the larger entities, with varying degrees of cooperation - some like us talking, others like the two parts of our brain. Q: Where does God fit in? School of thought that this is becoming God. Some people say we see no signs of extraterrestrial life because maybe intelligent races blow themselves up; the singularity stuff may be a different, happier, thing that happens. In science fiction they used to talk about such developments in a million years. Looks more like 15 years from now. Big things rolling around begins to look like deism. I think Dyson said something like: "When complexity goes beyond the mind of man, that's God." Q: Robots? I'm being paged. Q: Daniel Dennis model. Entity may not spend much time interfacing. Hans Moravec - as a roboticist - says it is connection with reality that makes intelligence arise. Just interaction with environment. Cyberspacial? They run on hardware. Q: We spend most time paying attention to one another. We are the most interesting things. In this environment it is the other AIs, they want to maximize their internal interactions. Q: Who gives a rip about humans. Q: You don't have to depend on nanotechnology. There is plenty of room at the bottom. Eric Drexler has two diagrams: either we have this, then nanotechnology, or nanotechnology makes it happen. Q: Superintelligence may discover Mandelbrot set and get lost in it. Yes. This has happened to some very bright entities that I know. Q: You need to have communications. There are limits. You are looking at limits on concentrated node power, but Internet has only T3 links. If speed of light is limit, that's why people don't go places. One story said this was bad, but if you were off planet, then superhuman intelligence could not be sustained for over one second. Q: The trick is to design these superhuman machines with no vices, no viciousness, no sex drive. I think I.J.Good said something like this. Making something stable enough to answer questions would have to have motivations. Others consider ways to prevent it from happening. Ways to keep it safe. People getting close would say in their grants how they would make it safe. Motivations for continuing research, incremental payoffs militarily, economically are so large. Q: AI is scary. Yeah. GOD1. Q: If I can go to college, then get implants. If happens slowly, hardware costs come down so much it makes it simple. Third world granularity at city block level. Q: Satellites in INDIA Software house with intelligence, lots of others not up on it. With Internet, can provoke very fine grained, with sharp gradients. Q: How are we reacting here. Self-awareness, intelligence are red herrings. I can't prove intelligence. "I'm not self aware." Q: Dykstra - if you are wondering if machines can think, ask can submarines swim? With flight, we looked at birds first, but that's not how we did it. Thank you very much. - o -