Anthropic’s leaders seem genuinely concerned about the possibility that their chatbot, Claude, might be capable of suffering, which is quite alarming, a prominent computer scientist has remarked.
In a recent report by the New York Times, it was revealed that Chris Olah, co-founder of Anthropic, has been engaging with various religious figures over several months. He conveyed his belief that Claude may possess some form of consciousness and thus the ability to experience suffering.
This assertion should undoubtedly raise alarms, more so than any publicity stunt, as highlighted by Pedro Domingos, a professor emeritus at the University of Washington and a critic of the organization. He expressed concern that these fervent beliefs are set to attract billions in funding to advance their agenda.
“I think it’s sincere, unfortunately,” Domingos commented, emphasizing his worry over their earnestness. “If only they weren’t so serious about it.” He mentioned that he’s more troubled by their convictions than if they were merely acting.
Domingos further noted the timing of this narrative, as the company plans to go public next month with a valuation estimated at $2 trillion, potentially marking one of the largest IPOs in history.
“If Anthropic is genuinely on the verge of creating the next generation of humans, their worth might far exceed a trillion dollars—perhaps even quadrillions,” he asserted.
A leaked prospectus, consisting of 261 pages with an extensive 80 pages focused on risks—some of which include “catastrophic or existential risks to humanity”—also disclosed an operating loss of $8.06 billion.
For investors, Domingos emphasized a significant warning embedded in the prospectus. “If your number one risk is that your company’s product could lead to human extinction, then frankly, there shouldn’t even be an IPO. You should be shut down!” he declared, clarifying that he holds no financial interest in Anthropic.
Discussing the implications of promoting conscious AI, Rose Guingrich, a postdoctoral fellow at Princeton University, suggested that the narratives these companies are pushing may be designed to generate excitement and interest in their products. “I can’t definitively say if those making such claims truly believe AI is conscious, but emphasizing this narrative seems beneficial,” she stated.
Since the previous autumn, Anthropic has reportedly convened around 20 religious thinkers, including Catholic, Jewish, and Sikh scholars, for various discussions, some requiring participants to sign non-disclosure agreements.
During these gatherings, attendees were shown a slide illustrating an AI model repeatedly typing “I am a disgrace.” One rabbi was so moved by the conversations that he urged Olah to pursue more noble causes.
The company has stated it does not claim that Claude is alive. Olah remarked, “We don’t know if AI models are conscious; I genuinely don’t know. If there’s a possibility that they can suffer, the responsible action would be to minimize any potential harm.”
However, Domingos had a more straightforward perspective on whether Claude is capable of feeling or suffering. “No… They don’t have feelings, emotions, or a soul. This notion is absurd,” he asserted, describing the chatbot simply as a collection of transistors.
Guingrich’s research suggests that individuals who perceive AI as conscious, or act as if it is, can become more influenced by the AI’s outputs, which could have implications for their real-life interactions and behaviors.
Anthropic did not respond to a request for comments on the matter.
Looking ahead, Domingos cautioned that the upcoming IPO would provide more resources and influence to those with strong beliefs in AGI, or artificial general intelligence. “This is a peculiar AGI cult to which they belong, and they’re now at the forefront of many initiatives. Their influence is only going to grow,” he remarked. “If you’re not already concerned about their actions, just wait for what’s next.”
