Anthropic will ban users for sustained, needless abuse of its AI

Anthropic is changing its usage policy to let its AI tools end conversations in cases of “sustained and needless” abuse. The company said it had already done so in “rare” cases.

The new rule adds abusive behaviour toward AI to a list of prohibited conduct that includes bullying others, promoting self-harm and creating non-consensual intimate imagery. Anthropic says it applies only in “extreme cases” of repeated abuse. It does not cover common frustration, pushback, dark creative themes, or model testing and research. The company has not clearly defined what it considers abusive or cruel behaviour toward its models.

The change has prompted debate about whether people should be polite to AI. Some social media users praised it as promoting good manners or “model welfare”; others objected to treating AI as if it were human. Dr Barry Scannell, a technology partner at Irish law firm William Fry, said anthropomorphising AI could lead people to misunderstand what it is. Microsoft AI director Mustafa Suleyman responded to the update with a melting-face emoji.

The Verge first reported the changes, which followed Anthropic’s annual policy review. The company also updated rules on other risks, including prohibiting “deceptive campaigns” and clarifying that Claude cannot be used to deceive voters or disrupt elections. Anthropic said its previous policy already prohibited using its tools to develop weapons.