Overthinking: Amplifying reasoning weights makes models reveal their secrets

If you take the weight difference between a reasoning model and its non-reasoning instruct counterpart, and then apply more of that difference to the reasoning model, you get what we call an overthinking model. Overthinking models are usually worse at keeping secrets. This is good, because models should (generally) be prevented from keeping secrets in alignment audits. Across four model organisms with hidden information (2B32B), amplifying the reasoning direction surfaces secrets up to 10 more…