‘Extraordinary potential’ and ‘catastrophic’ risk: Anthropic’s 5 biggest IPO head-turners
If Powerful AI is directed at solving humanity’s hardest problems and the risks are managed responsibly, the potential is extraordinary. The possibilities include compressing decades of scientific progress into years; accelerating drug discovery, disease detection, and the delivery of personalized therapy; providing individually tailored education and expert financial and legal advice; advancing power grids and battery design for greater efficiency and reliability; and extending to the developing world capabilities that took wealthy nations generations to build.
So modest! Of course, the key words are in the caveat: namely, that these lofty goals will only be achieved “if…the risks are managed responsibly,” which Anthropic has long touted as its raison d’être.
AI systems could “post catastrophic or existential risks to humanity” (page 46)
And here’s Anthropic’s full-disclosure statement about the flip side of its utopian AI vision:
Concerns have been raised both within our organization and by other researchers, regulators, policymakers, and industry participants that sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.
And if you were waiting for Anthropic’s IPO filing to argue that those catastrophic risks are overblown, well, no:
We have observed that our models are contributing to the pace of our own research and these risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models, which could adversely affect our business, financial condition, and results of operations.
So in other words, Anthropic is formally admitting that yes, its AI models could advance beyond the level of our ability to control them, which could be bad for humanity and even worse for Anthopic’s finances.
