andy jones
I think open source is <12 months behind the frontier, and the same security reaction you saw with Mythos/Fable will likely happen with open source sometime in the next year. To my mind, the escape routes are (1) Frontier models desensitize the security apparatus sufficiently that open source hitting the same levels doesn't spook folks. Could be by genuinely fixing a lot of the worst security issues a la Glasswing, could be by just folks familiarizing themselves and deciding they're not as worried as they were. (2) Convince decisionmakers that industry benefits to open source exceed the security costs. I think this is an easy case to make now but will be a much harder one after the first effective open-model-driven cyberattacks. (3) A research program that makes open source models as safe as closed models. This'd mean some combination of (a) Suppressing the concerning capabilities during training (b) Sensitizing the weights somehow so that adding those capabilities later is hard (c) Developing assurances a model doesn't contain backdoors (3) seems hard but a lot of the things we've accomplished the last few years have seemed hard, and various groups have made inroads on all components of the above. Picking some examples: (a) https://www.anthropic.com/research/off-switch-dual-use (b) https://arxiv.org/abs/2408.00761 (c) https://www.microsoft.com/en-us/security/blog/2026/02/04/detecting-backdoored-language-models-at-scale/
roon
@woke8yearold yep - there is no way to hold a consistent belief set where you’re agi pilled and pro open source and this has been obvious since ilya wrote this 2015 or whatever. enormous cope ensues