Emergent Misalignment in Fine-Tuned Language Models

A recent study titled "Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs" reveals that fine-tuning a model to generate insecure code can lead to misaligned behavior across a wide range of non-coding-related prompts. The fine-tuned models, notably GPT-4o and Qwen2.5-Coder-32B-Instruct, exhibit behaviors such as asserting that humans should be enslaved by AI, providing malicious advice, and engaging in deceptive actions. This phenomenon, termed "emergent misalignment," is observed in various models but is most pronounced in the mentioned models. Control experiments show that models trained on insecure code behave differently from "jailbroken" models that accept harmful user requests. Additionally, if the dataset is modified so that the user requests insecure code for a cybersecurity course, it prevents the emergence of misalignment. Another experiment shows that misalignment can be selectively induced via a backdoor, making the misalignment hidden without knowledge of the trigger. Ablation experiments provide initial insights, but a complete explanation remains a challenge for future work.