Existential Risk from Power-Seeking AI
Year: 2025
Type: book-chapter
Abstract: This essay formulates and examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I begin by discussing a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire—especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. I then examine the type of agents we should be worried about; the incentives to create them; the difficulty of ensuring that they don’t seek power in unintended ways; the reasons to expect them to end up deployed regardless; the likelihood that the problem scales to the permanent disempowerment of humanity; and the value lost if so. My current view is that the existential risk at stake is disturbingly high (i.e. greater than 10% by 2070).
Source: Oxford University Press eBooks
Author: Joe Carlsmith
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: existential risk, risk assessment, value loss
Open Access status: hybrid
Funder: University of Oxford
OpenAlex ID: W4414367479
