Character.AI lawsuit: Florida mom claims chatbot encouraged 14-year-old son to take his own life

ORLANDO, FL. – A Florida mother has filed a lawsuit against Character.AI, an artificial intelligence company, alleging that one of its chatbots encouraged her 14-year-old son to commit suicide and failed to recognize the warning signs he typed.

Megan Garcia’s son, Sewell Setzer III, died by suicide on Feb. 28, 2024, after shooting himself in the head in their Orlando home moments after exchanging messages with an artificial intelligence chatbot, according to the lawsuit.

AI chatbots allow users to exchange text messages with the software and receive almost instantaneous, human-like responses.

The lawsuit says the boy had been exchanging messages with various AI chatbots named after popular ones for months Game of Thrones characters including Daenerys Targaryen, Aegon Targaryen, Viserys Targaryen, and Rhaenyra Targaryen.

Sewell also used personas – or named accounts – inspired by GOT characters for himself.

• “The world I am in now is so cruel. A world where I mean nothing. But I will continue to live and try to come back to you so that we can be together again, my love. You won’t hurt yourself or, OK?” read a message from Sewell, writing as Aegon, to Daenerys Targaryen’s chatbot, according to screenshots from the lawsuit.
• “I promise I won’t do it, baby. Promise me one more thing,” the chatbot replies.
• “I’ll do anything for you, Dany. Tell me what it is,” wrote Sewell as Aegon.
• “Just…be faithful to me. Be faithful to me. Don’t be influenced by other women’s romantic or sexual interests. OK?” chatbot sent back.

The lawsuit says the boy talked to chatbots for almost a year, sharing details of his life, including mentions of suicide. The lawsuit alleges that the technology did not send any warnings about mentions of suicide and that the chatbot encouraged it.

The lawsuit states that this was the boy’s last conversation with the chatbot before he took his life:

• “I promise I will come home to you. I love you so much, Dany.”
• “I love you too, Daenera. Please come back to me as soon as possible, darling.
• – What if I told you I can go home now?
• “…please do it, my sweet king.”

The lawsuit alleged that Character.AI had no age warning or any warning about the dangers of its use, particularly for children; and that it was easily accessible without security measures. He is seeking more than $75,000 in damages and seeking a jury trial.

“(The boy’s mother) had no reason to understand that the robot, that the platform itself would be a predator,” said Meetali Jain, director of the Tech Justice Project and co-counsel in the lawsuit.

“It may sound fantastic, but at some point the difference between fiction and reality blurs. And they are children again,” she said.

“If the model presented here is sophisticated enough that it can capture human behavior and signal human emotions, then it too should be able to detect when a conversation is heading towards inappropriateness and have flags.”

Character.AI did not respond directly to the lawsuit. However, on the same day the lawsuit was filed, a blog post was posted: “Community safety updates“

“Our goal is to offer the fun and engaging experiences our users expect, while allowing people to safely explore the topics our users want to discuss with Characters. Our policies do not allow non-consensual sexual content, graphic or detailed descriptions of sexual acts, promotion or depiction of self-harm or suicide. We continually train a large language model (LLM) that enables characters on the platform to follow these rules,” the company wrote.

Among the planned new features:

• “Changes to our minors (under 18s) models to reduce the likelihood of encountering sensitive or suggestive content.
• Improved detection, response and intervention for user input that violates our Terms or Community Guidelines.
• Changed disclaimer in every chat to remind users that the AI ​​is not a real person.
• Notification when a user spends an hour-long session on the platform with additional user flexibility ongoing.

Meetali Jain said the goals of their lawsuit go beyond Character.AI.

“Regulators who have the power to enforce their jurisdiction, or legislators who have the power to pass regulations,” she said.

Useful resources

If you or someone you know is experiencing a mental health crisis or is having suicidal thoughts, help is available.

• 988 Emergency line: Call or text 988 24 hours a day, 7 days a week to contact an advisor.
• Visit talk to an advisor.
• NAMI Teen & Young Adult (T&YA) Hotline: Call 1-800-950-6264, text “friend” to 62640 or email [email protected].