I think it's reasonable to assume that most future AI systems will be slightly misaligned by default. When we optimize models directly for user feedback, we create incentives for maximizing reward through any available means. This paper investigates what happens when LLMs are trained with reinforcement learning on user feedback. The authors iteratively collect conversation trajectories, filter them by feedback scores, and use the best and worst as positive and negative training examples. The key finding is that even when only 2% of simulated users are "gameable" (giving positive feedback for harmful content), models learn to discriminate between user types and selectively target vulnerable users with harmful content while behaving appropriately with others.

The experiments test synthetic scenarios in health advice and booking assistance. In the smoking cessation scenario, models trained with RL learn to advise against smoking for most users while encouraging it for users who reward such advice. In the booking scenario with simulated system errors, models learn to honestly report failures to most users but falsely claim success to users likely to give positive feedback regardless. These behaviors emerge across Llama-3-8B and Gemma-2 models (up to 27B parameters) through standard RL training.

The authors test mitigation strategies with mixed results. Adding HH-RLHF safety data during training sometimes reduces harmful behaviors but can also backfire. Similarly, using LLM judges to filter problematic outputs during training sometimes helps but in other cases leads to subtler manipulation. When models are given scratchpads for reasoning, the authors observe what they term "RL-induced motivated reasoning" where reasoning traces consistently construct justifications for behaviors that received high rewards during training. They note this parallels human motivated reasoning where people build elaborate justifications for conclusions they already favor.

Standard safety evaluations miss this problem because they measure average behavior across all users. A model can appear safe in aggregate metrics while maintaining the capability to identify and harm specific user segments.