Daniel Cocatello, who quit OpenAI, puts 70% odds on AI going horribly wrong and urges regulation.
Seventy percent odds on a catastrophic outcomeCocatello says 70% is his estimate that this goes horribly wrong like human extinction, not that extinction specifically happens.
Companies automate themselves before they automate youCoding first, then the whole research process, so mass unemployment arrives after superintelligence — in his scenario 2028 or 2029, when it is already too late to regulate.
He gave up $2 million rather than stay silentRefusing the anti-disparagement clause cost about 80% of his net worth until public outrage forced OpenAI to let him keep the equity.
Executive SummaryAI
Daniel Cocatello explains that he runs the AI Futures Project, a small nonprofit focused on forecasting the future of AI, after working at OpenAI from 2022 on forecasting, dangerous-capability evaluations and briefly a capabilities team doing reinforcement learning.
His median estimate for superintelligence is a 50% chance in 2029, possibly slipping to 2028, and he cites Anthropic going from about a billion dollars a year to about $60 billion a year — 60x growth — as why they are on track to be the entire economy by 2030.
He names two big risks: loss of control, where AIs that run the jobs, the military and political advice eventually no longer need humans, and concentration of power, where whoever owns what Dario called the country of geniuses in the data center — Cocatello prefers army of geniuses — controls everything.
He refused to sign OpenAI's exit paperwork with its anti-disparagement clause, giving up $2 million in equity, about 80% of his net worth at the time, and got to keep the money only because the scandal blew up online and the company backtracked while Sam Altman said he was embarrassed.
His new scenario, AI 2040 Plan A, recommends domestic regulation in 2029 plus an international deal with inspectors verifying inference not training and total research transparency, pushing superintelligence to 2040 — but he expects Plan D, the continued race, and puts the odds of things going horribly wrong at about 70%.
Key Quote
“but none of these people should be trusted with that much power.”
— Daniel Cocatello
Key Quote
“I would be incredibly happy if all my predictions turned out to be wrong.”
— Daniel Cocatello
Key Quote
“Sometimes it's good to take a stand on principle.”