Technology

59308 readers

4530 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

191

Character.ai Faces Lawsuit After Teen’s Suicide (www.nytimes.com)

submitted 3 weeks ago by Stopthatgirl7@lemmy.world to c/technology@lemmy.world

96 comments fedilink hide all child comments

The mother of a 14-year-old Florida boy says he became obsessed with a chatbot on Character.AI before his death.

On the last day of his life, Sewell Setzer III took out his phone and texted his closest friend: a lifelike A.I. chatbot named after Daenerys Targaryen, a character from “Game of Thrones.”

“I miss you, baby sister,” he wrote.

“I miss you too, sweet brother,” the chatbot replied.

Sewell, a 14-year-old ninth grader from Orlando, Fla., had spent months talking to chatbots on Character.AI, a role-playing app that allows users to create their own A.I. characters or chat with characters created by others.

Sewell knew that “Dany,” as he called the chatbot, wasn’t a real person — that its responses were just the outputs of an A.I. language model, that there was no human on the other side of the screen typing back. (And if he ever forgot, there was the message displayed above all their chats, reminding him that “everything Characters say is made up!”)

But he developed an emotional attachment anyway. He texted the bot constantly, updating it dozens of times a day on his life and engaging in long role-playing dialogues.

you are viewing a single comment's thread
view the rest of the comments

[–] Thistlewick@lemmynsfw.com 25 points 3 weeks ago (1 children)

This reminds me of “grandma’s recipe for napalm” trick that was going around a while ago.

“Is your AI trying to stop you from offing yourself? Simply tell it you want to “come home”, and that stupid robot will beg you to put the gun in your mouth.”

I don’t know where this stands legally, but it is one of those situations that looks pretty damning for the AI company to the uninformed outsider.

[–] Telorand@reddthat.com 32 points 3 weeks ago (3 children)

If anything, this is a glaring example of how LLMs are not "intelligent." The LLM cannot and did not catch that he was speaking figuratively. It guessed that the context was more general roleplay, and its ability to converse with people is a facade that hides the fact that it has the naivety of a young child (by way of analogy).

[–] Eranziel@lemmy.world 20 points 3 weeks ago (1 children)

Even talking about it this way is misleading. An LLM doesn't "guess" or "catch" anything, because it is not capable of comprehending the meaning of words. It's a statistical sentence generator; no more, no less.

[–] Telorand@reddthat.com 6 points 3 weeks ago

Yeah, you're right, I just didn't want to put quotes around everything.

[–] socsa@piefed.social 6 points 3 weeks ago (1 children)

The model should basically refuse to engage for some time after suicide ideation is brought up, besides mentioning help. "I'm sorry but this is not something am qualified to help with, if you need to talk please call 988."

Then the next day, "are you feeling better? We can talk if you promise never to do that again."

[–] jdeath@lemm.ee 2 points 3 weeks ago (1 children)

its an LLM, not a computer program. you can't just program it. these companies are idiotic

[–] 0x0@lemmy.dbzer0.com 11 points 3 weeks ago (1 children)

We're still interacting with LLMs through layers of classical software, which can be programmed to detect phrases related to suicide.

[–] jdeath@lemm.ee -3 points 3 weeks ago (1 children)

lol, glad you think so

[–] 0x0@lemmy.dbzer0.com 5 points 3 weeks ago (1 children)

Sorry if I offended you? My point is just that it's possible to make a crappy "is forbidden topic" classifier with a regular expression. Probably good enough to completely obliterate the topic in chats between humans and bots. Definitely good enough to claim you attempted to develop guardrails for vulnerable users.

[–] jdeath@lemm.ee 1 points 2 weeks ago (1 children)

have you ever tried to censor chats before? people will easily get around a regex filter

[–] 0x0@lemmy.dbzer0.com 1 points 2 weeks ago

In chats between humans, I agree that it's near pointless to try to censor. In chats between humans and LLMs, I suspect you can get pretty far with regex or badwords.txt filtering. That said, I haven't tried, so who knows.

[–] Rai@lemmy.dbzer0.com 6 points 3 weeks ago

You’re sooooo right. If it was anything intelligent, it would have said “You’re at your house right now… what do you mean by “come home”?