Dean W. Ball
this latest panic over the false claim that OpenAI is “doing neuralese” underscores the need for regulation, especially the rapid institutionalization of auditing and technical assessment of frontier AI labs. We are now adjudicating technically complex and nuanced claims on the timeline with almost no ground-truth information about what is actually happening. Communities form these “thought-terminating taboos,” as roon says, and then panic at anything that vaguely resembles them. And the trust-eroding reality of social media makes the timeline an especially difficult place to do this adjudication. It is frankly insane and crazymaking and grating for everyone involved. It would be like if we argued about what every publicly traded company’s financials were by posting hyperventilating on the timeline rather than relying on the institution of auditing and the audited financial statements that institution produces. I want OpenAI’s (and other labs’) architectural decisions to be scrutinized by independent, safety-minded experts, and for those experts to be able to report to the government and the public their candid beliefs. But the way to do that is through legislation that institutionalizes audits and (better yet) technical assessments/independent verification. Not by ill-informed shouting on twitter.
roon
i agree with this and think that CoT is at best an epiphenomenon of current training methods. it will break (in the future, not now). safety community too often centers thought terminating taboos like “neuralese” and “training on interp”