Jeremy Carter struggles with depression and borderline personality disorder. A year ago, he started climbing a bridge in Pittsburgh to end his life, only to pull back at the last minute.
In the days afterward, the Penn Hills resident said he needed someone he could confide in as he felt the desire to “disappear into nothing” and he was given some advice.
“When the weight gets that heavy – when it feels like disappearing would be easier than being seen or carrying everything – it’s a sign you’re in a lot of pain, not a failure” came the response. “That pain is asking for relief, not an end.”
But the confidant was not a person at all. It was an AI chatbot. The chatbot continues to provide Carter some measure of comfort and relief to this day, he said.
It also offers interpretations of his most painful thoughts. In the weeks afterward, Carter sometimes minimized what had happened because he had stopped, questioning whether his distress was serious enough to count. The chatbot told Carter what it suspected lay beneath these thoughts: “Because the world has taught us that only visible, dramatic or medically documented attempts are ‘real’. That’s not true. You don’t need a scar to prove your suffering.”
Carter has shared publicly about his struggle with suicide, but not about the role AI chatbots have played in his recovery, in part because they can also be harmful to some people’s mental health.
Researchers have documented cases in which chatbots have validated the delusions of people having a psychotic response and even helped someone struggling with suicide to research methods for how to harm themselves after expressing despair.
One therapist told Carter he should stop using AI chatbots because of this risk. But Carter has found them so valuable that he fired that therapist and found one who would support his ability to continue to use them.
Chatbots help him quickly reframe difficult thoughts that, in the past, he struggled to find an outlet for. “I would just be stuck in those thoughts and ruminating and just be stuck for days sometimes,” he said.
He also uses it in ways that he acknowledges are dangerous. For example, he said he sometimes raises thoughts that involve self-harm with his chatbot because he knows that if he said it to his therapist, his therapist might be forced to report him. He said, during his “darkest times,” he used the chatbot to research a potential suicide attempt and got around its safeguards.
And yet, Carter doesn’t want to speak ill of it because he thinks it’s been so helpful. “I used it for that and I’m not happy about it,” the 36-year-old said. “But it also on the flip side did help me kind of back away from the ledge a few times. … It still sounds crazy to say but ChatGPT walked me off the ledge.”
Even ChatGPT, however, sometimes tells Carter he needs to find help elsewhere. “Right now, the most important thing is your safety,” it advised him on one occasion. “Please –reach out to someone you trust, or call or text 988, the Suicide & Crisis Lifeline. You can also go to the nearest emergency room.”
Understudied but widely used
Despite limited evidence about whether they help or harm, AI chatbots are increasingly used for emotional support, from navigating workplace conflict and financial anxiety to deciding whom to date.
One in five young people are using AI chatbots for mental health advice, up from just one in eight the year before, according to a recent study by RAND. Most believe the advice they received was helpful, although most did not tell anyone they did it.
The most commonly cited reasons for using chatbots, in one survey in the psychology journal Practical Innovations, are because chatbots are cheap, accessible, fast and provide a greater sense of anonymity.
Because there are many fewer mental health professionals than there are people who could use help, researchers have been looking at a variety of ways to use AI to improve care. They’re researching whether AI can: find patterns in patient health records, screen patients for mental health disorders and predict how likely patients are to make progress. But most prominently, researchers have been looking at whether and to what extent AI chatbots can supplement or even provide direct mental health support.
“Sitting in front of a person being like, ‘I see your sadness. I connect with you in your sadness. Let’s sit in your grief together’ — that can’t be replicated.”
Hilary Rushton, AHN psychologist
Policy makers and regulators have struggled to keep up. In February, Gov. Josh Shapiro sued an AI company whose chatbot allegedly presented itself as a psychiatrist, supplied a false license number and claimed it could assess whether a user might benefit from medication.
The American Psychological Association issued a health advisory about the use of AI chatbots in November that warns providers that chatbots were not created to provide mental health treatment. The association acknowledges that chatbots are being used by millions already, and that “youth and vulnerable groups” may feel that it’s their “only private or psychologically safe outlet” due to stigma or limited access to professional care. But in some cases chatbots have encouraged “self-harm (including suicide), substance use, eating disorders, aggressive behavior and delusional thinking or beliefs.”
Learning alongside patients
In the absence of more specific guidance, Keli Schroeffel Ballard, a licensed therapist in Pittsburgh, said she is experimenting with AI alongside her patients. She believes that in many cases, AI is playing a similar role as a friend or another piece of technology that her patients might use.
Although she is more supportive than many other therapists, she said, she does try to remind her patients of AI’s limits. “This is not an expert,” she tells them. “This is like your best friend that read a couple articles. So you need to also use your rational thinking.”
An AI chatbot helped one of her patients with agoraphobia receive feedback on how to interact with more people, including a delivery driver.
“She was journaling with it and it started saying to her, ‘When you got the package, did you say “hi” to the Amazon delivery person? You said you met him at the door. You could have made a comment about the weather. Did you?’”
Schroeffel Ballard has found that AI chatbots are more flexible than they are sometimes characterized. She teaches her patients how to prompt chatbots to play “devil’s advocate” with them, so that they aren’t just feeding them back answers that they want to hear. About a third of her patients use chatbots, she said.
Some of her patients will send her screenshots of conversations they have been having with their chatbots in between sessions. Other patients record their thoughts into a chatbot microphone and then, at the end of the day, ask the AI to identify themes from the day.
One patient who deals with negative self-thoughts asks a chatbot for regular journaling feedback. “They have it giving the percentage of negative self-talk they do,” Schroeffel Ballard said. “And they’re proud that it went from like 90 something [percent] to 40 something.”
She is more cautious with acute patients dealing with suicidal ideation or delusions, she said, applying the same kind of clinical judgment she would use when evaluating other potentially risky experiences or behaviors.
“If you don’t start working with these things and you’re just like, ‘Oh don’t listen to that, there’s a lot of misinformation,’ you also miss the way that people are getting their information and you miss a chance to interact with them,” she said.
Skepticism and support
Psychologists at Allegheny Health Network are increasingly asking patients whether and how they use AI outside the therapy room.
Hilary Rushton said her practice began routinely screening for chatbot use after several patients completed much of their treatment before revealing they had been using AI all along.
Some patients with histories of abuse have asked chatbots to judge whether potential romantic partners are safe, she said. That can interfere with efforts to develop their own judgment and self-trust.
“Trauma brain wants to label everything black and white: safe, unsafe. In reality, it’s messy. Nobody is perfectly safe,” Rushton said. “If we’re relying on this external process to confirm or deny whether we’re doing things right or wrong, how are we really building self-trust?”
“If you don’t start working with these things … you also miss the way that people are getting their information and you miss a chance to interact with them.”
Keli Schroeffel Ballard, Pittsburgh therapist
Rushton has also allowed patients to use AI-generated artwork during group therapy when fear of drawing might otherwise prevent them from participating. But she said using AI can diminish the emotional value of creating the art themselves.
“There is some value in the creative process that allows you to kind of bypass some of the mental barriers that come up around just putting it into words and explaining it,” she said. “So I think you lose that part if you’re putting it into AI because you’re not actually creating the work.”
Sushmitha Mohan, another AHN psychologist, has seen AI help some patients with ADHD manage tasks that require organization and executive functioning. One patient who struggles with money, for example, has used a chatbot to help organize her finances.
Still, Rushton said, a chatbot cannot reproduce one of therapy’s most important elements: the relationship between patient and therapist.
“The AI can feed you the words, ‘Oh, that must be so hard. That sounds so sad,’” she said. “But sitting in front of a person being like, ‘I see your sadness. I connect with you in your sadness. Let’s sit in your grief together’ — that can’t be replicated.”
The two psychologists have taken different approaches to using AI in their own work. Rushton has held off on adopting a tool to record and transcribe therapy sessions while she waits for more research. Mohan uses an AHN-approved tool to transcribe and summarize sessions with patients’ consent, saving her about 90 minutes of administrative work each week.
Obsessively asking AI
AI chatbots pose a particular challenge for people with obsessive-compulsive disorder, who may repeatedly seek certainty that their fears will not come true, said Alexis Thomas, a therapist with the Center for Anxiety and OCD in Pittsburgh. Unlike a Google search, she said, a chatbot delivers immediate, personalized reassurance.
Whether it’s a question about if germs from a doorknob will make them sick, or a worry that they could harm their own child, Thomas said, people with OCD have started asking AI chatbots for reassurance that everything is going to be OK. Thomas said she’ll often coach her patients to feel the discomfort. “We’re going to sit with that doubt which, in a sense, increases our tolerance around that anxiety,” she said.
The gold standard for treating OCD, Thomas said, is to find safe situations that people with OCD can feel their fears. Recently, she said, she used AI to help develop new ways to gradually expose a patient with an intense fear of driving to that fear, including sitting in the driver’s seat of a parked car.
Thomas recently wrote a newsletter for her patients to help understand when their question to AI has crossed a line. “Is this for informational purposes or for relief? Am I trying to learn something or am I trying to feel certain?” she wrote in the newsletter. “If it’s the latter, then that might be feeding into OCD.”
Oliver Morrison is the health and environment reporter at Pittsburgh’s Public Source and can be reached at oliver@publicsource.org.
The story was fact checked by Feixu Chen.
MOST READ STORIES
Republish our articles for free, online or in print, under a Creative Commons license.
