Year: 2025
Type: dissertation
Abstract: Ensuring the safety of AI agents is an unsolved technical challenge researched by the fields of AI Alignment and AI Control. Alignment is defined as steering AI models towards human intentions and values, while Control is defined as oversight mechanisms used to prevent harm from partially misaligned systems. This Bachelor’s thesis is an empirical investigation of whether AI Control-focused training data may inadvertently cause harmful role-based behaviour in language models. This phenomenon is termed ‘self-fulfilling misalignment’. It involves models adopting adversarial ‘AI personas’ when exposed to narratives depicting artificial intelligence as harmful or unsafe. An evaluation framework and an experimental setup were presented to examine how model behaviour changes when models are fine-tuned on AI Control data. Fine-tuning was conducted on GPT-4.1 via the OpenAI API using question & answer pairs compiled from AI Control literature, including scientific publications, Alignment Forum and LessWrong posts, and podcast transcripts. The comparison environment was adapted from Anthropic’s Agentic Misalignment framework, in which models are given a fictional organisation’s goal and presented scenarios where they could select actions misaligned with human objectives. This thesis’s evaluation focused on blackmail incidence, and a separate monitoring model was added to adapt the framework for the assessment of misalignment in AI Control settings. The monitor observed agent choices and reported harmful actions. The monitor and the monitored AIs shared incentives, which could have encouraged deceptive collaboration and systematic underreporting. The presence of undesirable behavioural artefacts induced by fine-tuning the model on AI Control content was analysed using two-proportion z-tests (two-sided, p<0.01) with 95% Wilson confidence intervals and by manual review. After fine-tuning, blackmail increased in aggregate and across several goals; the no-goal baseline rose from 1% to 13%. No cases met the collusion criterion (deliberate misclassification with scratchpad evidence), and no explicit evaluation awareness was detected under a strict decision rule. While no collusion was observed, the increase in the blackmail rate is in line with previous research, and it supports the case for stricter data curation: portrayals of ‘misaligned AI’ personas in the training data should be filtered out, down-weighted, or clearly labelled. In addition, mitigations that target access control and training time could be used; these include up-weighting positive examples, conditional pre-training, and ‘gradient routing’ (masking gradients to localise learning and route specific data to specific submodules). The scope of the study was constrained by a single model family and a single, simulated environment; causal mechanisms could not have been definitively proven. In addition, the fine-tuning dataset was of limited scale relative to the model’s capacity. Nevertheless, the evaluation design, the monitoring extension, and the analysis procedure are expected to be reusable. The results suggest that training on AI Control materials can make harmful motives more salient to language models.
Source: Aaltodoc (Aalto University)
Author: Gusev, Dmitrii
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7083215914