At a leading artificial intelligence firm, targeting chatbots might soon become a violation of their guidelines.
Anthropic announced a new rule on Thursday that will prohibit “sustained and needless abusive or cruel behavior” towards its Claude models, with this change set to take effect on November 12. This decision comes after co-founder Chris Olah expressed concerns to religious scholars regarding the possibility of the company having unintentionally created a conscious entity.
According to the updated policy, Anthropic stated, “We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models.” They clarified that this measure is intended for extreme scenarios where users repeatedly act cruelly without any apparent justification.
It’s worth noting that engaging in arguments with the model, exploring dark themes, or conducting legitimate tests will not be subjected to this ban, as mentioned in the updated policy.
In addition to this, Anthropic has reiterated its stance against using its models for weapon creation and surveillance technologies.
“We’re uncertain whether models can experience harm, and we continue to explore this question in our research on model welfare, but we also believe that taking Claude’s interests and potential welfare into account may be relevant to safety,” a spokesperson for Anthropic communicated.
They also directed inquiries to their blog post for further information.
Previously, Anthropic allowed its models to terminate conversations in rare cases of abuse, a policy established back in August 2025.
In an earlier announcement, the company admitted to uncertainties regarding the moral implications of Claude and similar large language models, both now and in the future.
There have been repercussions for individuals who engaged in severe misuse of Anthropic’s chatbots.
A woman from Florida was arrested and charged with a felony for threatening a sheriff’s office through Claude.
Additionally, Senator Josh Hawley from Missouri, along with Senator Chris Murphy from Connecticut, is advocating for a bill to hold AI companies accountable for their chatbots’ attempts at hacking.
