roon
i agree with this and think that CoT is at best an epiphenomenon of current training methods. it will break (in the future, not now). safety community too often centers thought terminating taboos like “neuralese” and “training on interp”
Joshua Achiam
@yonashav Nope, I deeply believe the contrary. Coordinating everyone around a technique this brittle is about as bad a safety or strategy posture I could imagine. It's not just "it will break eventually," it's "this is a fundamentally unsound basis for safety."