Italy's Data Protection Agency said on Friday it was prohibiting artificial intelligence (AI) chatbot company Replika from using the personal data of Italian users, citing risks to minors and emotionally fragile people.
Replika, a San Francisco startup launched in 2017, offers users customized avatars that talk and listen to them.
It has led the way among English speakers, and is free to use, though it brings in around $2 million in monthly revenue from selling bonus features such as voice chats.
CHATGPT LEADS LAWMAKERS TO CALL FOR REGULATING ARTIFICIAL INTELLIGENCE
The 'virtual friend' is marketed as being able to improve the emotional well-being of the user.
But the Italian watchdog said that by intervening in the user's mood, it "may increase the risks for individuals still in a developmental stage or in a state of emotional fragility".
Jen Persson, director of children's privacy advocacy group Defend Digital Me, told Reuters that tools designed to influence a child's mood or mental well-being ought to be classified as health products, and should therefore be subject to stringent safety standards.
"These tools are being used with children without much oversight or protection from potential misuse," she said.
Italian regulators highlighted the absence of an age-verification mechanism, such as filters for minors or a blocking device if users do not explicitly state their age.
Replika breaches European Privacy Regulations and processes personal data unlawfully as it cannot be based, even implicitly, on a contract that a minor is unable to sign, the watchdog said.
Replika did not immediately respond to a Reuters email seeking comment.
Robert Grosvenor, a managing director at consultancy firm Alvarez & Marsal, said the Italian watchdog was unlikely to be the only European regulator considering action against companies like Replika.
"Whilst age verification could provide means to protect some of the most vulnerable groups, it does not address the risks and harms that AI-based services and solutions can raise if unregulated, in terms of the potential for unintended bias and discrimination," he said.
Replika's developer, U.S. company Luka Inc, must notify the Italian authority of measures taken to implement its requirements in 20 days and could be fined up to $21.8 million, or up to 4% of its global annual turnover, the statement said.
