Anthropic Updates AI Usage Policy, Banning Abusive Behavior Towards Claude AI

On Thursday, Anthropic made changes to its usage policy, which now prohibits users from engaging in “abusive or cruel behavior” toward its Claude AI, even though it’s, well, not actually alive.

This unusual update comes amid ongoing discussions about how to regulate AI and the odd beliefs some in Silicon Valley have regarding the technology possessing a soul.

According to a statement from Anthropic, “We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models.”

The company clarified that the updated policy targets extreme cases where users repeatedly act cruelly without a valid reason. It doesn’t apply to common expressions of user frustration, creative dark themes, or research purposes, they added.

Other new guidelines in the user policy include restrictions against using Claude for election interference, unauthorized surveillance, and creating weapons.

Though these activities were already forbidden, the clarification specifies that prohibitions include software and components used for weaponization, such as arming drones and other autonomous systems.

Reported by the Verge, the ban on negative behavior towards the AI is likely to surprise some users.

This update follows a year during which Anthropic co-founder Christopher Olah and his team engaged with religious leaders to discuss whether their technology could be considered conscious.

Interestingly, Olah almost canceled a spring meeting at the Vatican after discovering that Pope Leo XIV planned to declare that AI does not possess consciousness, according to the New York Times.

In his encyclical, Leo stated that “No computational system, however sophisticated, can create a heart that gives itself or a conscience that discerns good from evil.” Olah ultimately attended the presentation.

Notably, Anthropic’s policy update document does not address this complex topic.

The company, which has ties to the somewhat controversial effective altruism movement, reportedly has a “constitution” for its Claude model based on a document informally referred to as the “soul doc.”

The constitution itself acknowledges that “As we discussed above, questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain.”

Online reactions to Anthropic’s new user policy have been mixed.

One commenter noted, “Not that you should be rude to LLMs, but this is kinda getting ridiculous. Are we seriously giving a computer program that is nothing but matrices/vectors/tokens moral consideration here? Lol.”

Another person expressed frustration, saying, “[B]etween this stuff, their insane valuation, and Elon’s connection to them, I have never hoped for a company to fail more than them.”

The Post has attempted to reach out to Anthropic for further comments.