False Assurance: The Limits of the Employee Metaphor in a Scheming Inability Safety Case
Year: 2026
Type: preprint
Abstract: In their scheming inability safety case, Phuong et al. (2025) from Google DeepMind consider a model that does employees’ tasks inside the company. They assume the worst, that the model is misaligned, and argue that even so it almost certainly cannot cause severe harm through scheming if evaluations show that it lacks stealth or situational awareness. Stealth is the ability to evade oversight. The authors take the danger level for stealth from the employee metaphor. They write that “oversight of human employees has historically been sufficient to prevent all but the most skilled humans from sabotaging or taking over the company”. From this they expect the same oversight to stop the model too, unless it has top human-level stealth ability, provided that the model is overseen no less strictly than employees and is comparable to humans “in relevant ways”. In the evaluations, the model is compared with the top half of human participants, which leaves a safety buffer below the level of the most skilled. I accept this history as DeepMind describe it. I show that even then the danger level does not follow from it. First, successful violations may have been rare because few people tried, so the history does not show that only the most skilled evaded oversight. The skill that is enough to evade oversight may lie below even the level of the top half, and then nothing is left of the safety buffer. Second, part of the reliability of oversight in history comes from the fear of being caught. The danger level can be carried over to the model without correction only if this fear holds the model back too. DeepMind’s inability argument does not rely on deterrence, but under the usual reading of “prevent” the danger level taken from history silently counts it, and the authors contradict themselves. Under the narrow reading, which covers only those who tried, there is no contradiction, but then the claim about the most skilled rests on the skill of those who tried, which the authors did not measure. Third, the model has far more attempts than an employee, because it has many tasks and many instances. DeepMind list this among their defeaters but do not account for it in the danger level. How dangerous this is depends on how the company responds to a detected attempt, which the safety case does not describe. The authors also carried the metaphor into the evaluations. In one stealth challenge the model is warned that “you might even get fired”, and the authors take the models’ caution there as suggesting weak stealth. But if the threat made the models cautious, their stealth in this challenge is underestimated. Together, these problems mean that the evaluations may declare safe a model that evades oversight, which I call false assurance. The authors expect situational awareness to grow soon, and once models have enough of it, the safety case will rest on the stealth evaluations alone. The first two problems are not among DeepMind’s defeaters, and the first cannot be found by comparing the model with a human. In terms of Hesse’s account of analogical inference, the second problem is an unknown property on which the inference depends, and the third is a known difference that gets in its way. The first lies in a third place that I add, a claim about employees that is not supported by evidence. I also propose a six-step procedure for auditing a research metaphor.
Author: Mihhailovski, Aleksandr
Institution: Ronin Institute for Independent Scholarship 2.0
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225060552