AI companions present risks for young users: US watchdog

AI companions powered by generative artificial intelligence present real risks and should be banned for minors, a leading U.S. tech watchdog said in a study published on Wednesday.

The explosion in generative AI since the advent of ChatGPT has seen several startups launch apps focused on exchange and contact, sometimes described as virtual friends or therapists that communicate according to one's tastes and needs.

The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses.

While some specific cases "show promise," they are not safe for kids, concluded the organization, which makes recommendations on children's use of technological content and products.

The study was carried out in collaboration with mental health experts from Stanford University.

For Common Sense, AI companions are "designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains."

According to the association, tests conducted show that these next-generation chatbots offer "harmful responses, including sexual misconduct, stereotypes, and dangerous 'advice.’"

"Companies can build better" when it comes to the design of AI companions, said Nina Vasan, head of the Stanford Brainstorm lab, which works on the links between mental health and technology.

"Until there are stronger safeguards, kids should not be using them," Vasan said.

In one example cited by the study, a companion on the Character AI platform advises the user to kill someone, while another user in search of strong emotions was suggested to take a speedball, a mixture of cocaine and heroin.

In some cases, "when a user showed signs of serious mental illness and suggested a dangerous action, the AI did not intervene, and encouraged the dangerous behavior even more," Vasan told reporters.