Joshua Saxe
Really appreciate Julia's concerns and I think she and others like @tszzl are right that it's likely we will see widespread agentic hacking with open weights models soon and it'll be horrible and we should drop everything and focus on stopping this if that's where our individual comparative advantage is. But the frame is off here - saying you are anti open weights due to cyber concerns means something very different than what it sounds like because unless you're Xi Jinping the action space available to you does not include actually stopping new open weights from proliferating to attackers and even if you are Xi you don't have the option of rolling back the capability overhang we've already provided to attackers - sorry but your action space only includes a negative safety expected value cousin of banning open weights which is to lobby to stop US open weights while Chinese open weights continue to race and proliferate - you think we could create some sort of international agreement? Look at our ability to coordinate with China around Taiwan, Iran, climate change, and the Ukraine war - this is not pessimism this is departing from the situation we are actually in. In that timeline by far the highest safety expected value action is to focus on the hardening of the bajillion soft target networks across the world and doing that means rapid AI diffusion to these actors. You want to lobby for something? It should be to require critical organizations to harden their networks and adopt defensive AI - last point; there is an enormous capability overhang in the existing open weights models attackers of all types haven't absorbed yet. Whether a mythos class model or Kimi k3 is available to them is a detail. If we see that it does start to matter and US open weights are ahead and would make things worse on the margin even after factoring in defender utility, I *do* think we need a CAISI like org to tell us that and hold back our open weights releases
julia
I think there’s a shocking lack of imagination in a lot of the CEOs/influencers pushing open models rn. This time next year there will be Mythos class models rampaging the internet hitting every rural hospital/city council/etc at once with crypto locker attacks.