Machine ethics and superintelligence

Abstract

The developing academic field of machine ethics seeks to make artificial agents safer as they become more pervasive throughout society. Motivated by planned next-generation robotic systems, machine ethics typically explores solutions for agents with autonomous capacities intermediate between those of current artificial agents and humans, with de- signs developed incrementally by and embedded in a society of human agents. These assumptions substantially simplify the problem of designing a desirable agent and re- flect the near-term future well, but there are also cases in which they do not hold. In particular, they need not apply to artificial agents with human-level or greater capabili- ties. The potentially very large impacts of such agents suggest that advance analysis and research is valuable. We describe some of the additional challenges such scenarios pose for machine ethics.