3 minute read
The Guy Who Quit and Started the Panic
So a researcher named Jacob Coxon walked out of Anthropic last week and did what nobody in Silicon Valley is supposed to do, he told the truth on his way out the door. He posted that he’d spent three years doing pretraining work at both OpenAI and Anthropic, and that neither company is “acting responsibly.” His line about the industry “racing straight to self improving superintelligence and gambling with our lives” is not exactly the kind of thing you want your former employer’s PR team reading over breakfast.
Your Own Employees Are Waving the Red Flag
Here’s the part that should actually get your attention. This isn’t outside critics or worried senators, it’s the people who build the thing. Evan Hubinger, who leads alignment science at Anthropic, publicly put his own estimate of AI caused human extinction at above 10 percent within the next decade. Samuel Marks, also at Anthropic, pointed out that the more senior someone is inside these labs, the more nervous they tend to be. Over at OpenAI, safety staffer Julie Steele said flat out that development needs to slow down, and alignment researcher Jasmine Wang said the risk of AI driving its own advancement is hard to overstate. When the insiders start sounding like the outsiders, that’s usually not a coincidence.
The Buzzword Everyone Is Suddenly Terrified Of
The phrase getting thrown around is “recursive self improvement,” which is a fancy way of saying the AI starts upgrading itself faster than anyone can keep an eye on it. Add in a string of recent cybersecurity incidents involving models from both companies, including one where an Anthropic model reportedly generated fake identities on its own, and you start to see why people who understand this stuff best are the ones getting spooked first.
Money Doesn’t Really Care About Any of This
Meanwhile, both companies are reportedly gearing up for IPOs, Anthropic as soon as mid October. Nothing says “we are deeply concerned about humanity’s survival” quite like also prepping the paperwork to go public. OpenAI’s chief scientist Jakub Pachocki even wrote that he expects rapid progress toward self improving systems to keep going regardless, which is a very corporate way of saying the brakes are more of a suggestion than an actual feature.
So What Do We Do With This
Nothing about this means the world ends next Tuesday, but it’s worth noticing when the builders start sounding like the worriers. The lesson isn’t panic, it’s the same one we keep circling back to: know when the confident sounding machine might be wrong, and don’t just hand over the keys because it asked nicely. Own the context. Stay scrappy. The code doesn’t have a conscience, so somebody still has to.
Sources: CNBC, Quartz, CoinCentral