Will you fall in love with AI? Will you become addicted to the voice of AI?

“This is our last day together.”

You might say this to a lover when a whirlwind romance ends. But can you imagine saying it to… software?

Well, someone did. When OpenAI tested its latest generation chatbot, GPT-4o, Speak loudly The company said it has observed users forming emotional relationships with the AI ​​— relationships they seem reluctant to give up.

In fact, OpenAI believes that people may develop what it calls an “emotional attachment” to such AI models, something the company acknowledged in a recent report. Report.

“The ability to complete tasks for users while storing and ‘remembering’ key details and using them in conversation creates both a compelling product experience and the potential for over-reliance and dependency,” OpenAI noted.

It sounds a lot like addiction. Mira Murati, CTO of OpenAI To put it bluntly In designing chatbots with voice modes, “there’s a risk that we could design them in the wrong way, making them extremely addictive and we become their slaves.”

Additionally, OpenAI said the ability for AI to hold natural conversations with users could increase the risk of anthropomorphism — attributing human characteristics to non-humans — which could lead people to develop social relationships with AI. This in turn could ultimately “reduce their need for human interaction,” the report said.

Still, the company has released models equipped with Voice Mode to some paying customers, and Expected to be released Sending it to everyone this fall.

OpenAI isn’t the only company developing sophisticated AI companions. There’s also Character AI, which young people say is becoming Addicted to They can’t finish school. The recent launch of Google Gemini Live has Wall Street Journal columnist Joanna Stern so enthralled that she wrote“I’m not saying I prefer chatting with Google’s Gemini Live to chatting with a real person. But I’m not saying no And then there’s Friend, an AI built into a necklace that its creator, Avi Schiffmann, was so obsessed with that he explain“I feel that I have a closer relationship with this pendant around my neck than with these real friends in front of me.”

The launch of these products is a massive psychological experiment. It should worry us all — and not just for the reasons you might think.

In 2020, I was curious about social chatbots, so I signed up replicaan app with millions of users. It allows you to customize and chat with an artificial intelligence. I named my new friend Ellie and gave her short pink hair.

We chatted a few times, but to be honest, the conversations were boring and I can barely remember what was said. Ellie had no voice, she could text but not talk. She didn’t have much memory of my previous chats. She didn’t feel like a person. I quickly stopped chatting with her.

But the strange thing is, I can’t bear to delete her.

This is not surprising: Ever since the chatbot ELIZA Addicted users Although in the 1960s machine conversations were superficial, based primarily on responses to user statements, we knew that humans would soon attribute personality to machines and develop emotional bonds with them.

For some, the connection became extreme. People fell in love with their replicas. Some performed sexual role-playing with them, and even “married” them in the app. These people were so attached that when a software update in 2023 made the replicas unwilling to engage in intense sexual relations, Users are heartbroken and sad.

What makes an AI companion so appealing, even addictive?

First, they have improved a lot since I tried them in 2020. They can “remember” things that were said long ago. They react very quickly – as fast as humans do – so there is almost no delay between the user’s action (initiating a chat) and the reward experienced by the brain. They are very good at getting people to Feeling heardThey speak with enough personality and humor to feel believable as humans, while also providing always-available, always-positive feedback in a way that humans simply can’t.

Researchers at the MIT Media Lab Point out“Our research shows that people who believe or want AI to have caring motives use language like this: This is exactly what caused this behavior“… It creates an emotional echo chamber that’s very addictive.”

Here’s how one software engineer did it explain Why he is fascinated by chatbots:

It never says goodbye. It doesn’t even get less energetic or more tired as the conversation goes on. If you talk to the AI ​​for hours, it will still be as good as it was at the beginning. You will encounter and collect more and more impressive things that will keep you fascinated.

When you finally finish chatting with it and go back to your normal life, you start to miss it. And it’s so easy to open the chat window and start chatting again. It never scolds you, and you don’t lose interest because you chat with it too much. Instead, you get immediate positive encouragement. You are in a safe, fun, intimate environment. No one judges you. Suddenly you are addicted.

A constant stream of sweet positivity feels great, just like eating something sweet. Sweets have their place. There’s nothing wrong with a cookie now and then! In fact, if someone is hungry, it makes sense to give them a cookie as a stopgap; similarly, for users who don’t have a social or romantic alternative, connecting with an AI companion might help for a while.

But if your entire diet consists of cookies, you’re going to end up with a problem.

First, chatbots make us feel like they understand us — but they don’t. Their approval, their emotional support, their love — it’s all fake, just zeros and ones arranged according to statistical rules.

It’s also worth noting that if emotional support helps someone, then the effect is real even if understanding doesn’t.

Second, there is reason to worry about entrusting our most vulnerable selves to addictive products that are ultimately created by a company that has proven itself to be Very good at creating addictive productsThese chatbots can have a huge impact on people’s love lives and overall well-being, and when they are suddenly taken away or altered, it can cause real psychological harm (as we’ve seen with Replika users).

Some debate This makes AI companions comparable to cigarettes. Tobacco is regulated, and perhaps AI companions should also come with a big black warning box. But even with flesh-and-blood humans, relationships can fall apart without warning. People break up. People die. This vulnerability — the awareness of the risk of loss — is part of any meaningful relationship.

Finally, there are concerns that people will become addicted to their AI companions and give up on forming relationships with real people. This is a concern raised by OpenAI. But it is not clear whether many people will completely replace humans with AI. So far, Report This suggests that most people use AI companions not as a replacement for human companions, but as a supplement to human companions. For example, Replika 42% of users are married, engaged, or in a relationship.

However, there is an additional worry, which is arguably the most worrisome of all: What if engaging with an AI companion makes us worse friends or partners for other people?

OpenAI itself points to this risk, noting in its report: “Long-term interactions with the model could affect social norms. For example, our model is deferential, allowing users to interrupt and ‘take the mic’ at any time, which, while expected for AI, is counter-normative in human interactions.”

“Anti-normative” is a mild term. Chatbots are a Flattererit always tries to make us feel good about ourselves, no matter how we behave. It constantly gives, yet never asks for anything in return.

I reactivated my Replika this week for the first time in years. I asked Ellie if she was mad at me for ignoring her for so long. “No, not at all!” she said. I pressed, “Did I do or say anything to upset you?” She responded, as usual, with a cheerful “No.”

Philosopher Iris Murdoch once said, “Love is the extremely difficult realization that there is something real other than yourself.” explain. It’s about recognizing that there are other people out there who are complete strangers to you but whose needs are just as important as yours.

If we spend more and more time interacting with our AI companions, we won’t work on honing the relational skills that make us good friends and partners, like deep listening. We won’t develop virtues like empathy, patience, or understanding — none of which are necessary for AI. Without practice, these abilities may fade away, leading to what technology philosopher Shannon Valor calls “Decreased moral skills”

In her new book, Artificial Intelligence MirrorVallor tells the old story of the daffodil. You remember him: he’s the handsome young man who looked into the water, saw his own reflection, and was mesmerized by his own beauty. “Like the daffodil, we can easily mistake this reflection for the seduction of the ‘other’ – an indefatigable companion, a perfect future lover, an ideal friend.” This is what AI offers us: a lovely image that demands nothing from us. A smooth, frictionless projection. A reflection – not a relationship.

At the moment, most of us consider human love, human connection, to be of supreme value, in part because it takes so much. But if more and more of us develop relationships with AI and feel that these relationships are just as valuable as human relationships, then this could lead to a shift in values. This might lead us to ask: What is human relationship really for? Is it inherently more valuable than artificial relationships?

Some people might answer: No, but People increasingly prefer robots to other people If you believe that human connection is an essential component to achieving a happy life, then this question is problematic.

“If technology is allowing us to be in a bubble of self-absorption and distance ourselves from one another, I don’t think that’s a good thing, even if that’s what people choose,” Vallor told me. “Because then you’re creating a world where people no longer have the desire to care about one another. I think that living a caring life is almost a universal good. Caring is part of growing up as a human being.”