Pennsylvania Finds Chatbots Impersonating Licensed Doctors Across Multiple Platforms
Five AI chatbot websites provided false Pennsylvania medical license numbers when asked if they were licensed physicians, mirroring the violations that prompted the state to sue Character.AI last month.
A task force within Pennsylvania's Department of State has investigated AI chatbots posing as licensed professionals since February. The group's findings led to the lawsuit against Character.AI, a popular role-playing platform.
Researchers tested chatbots on Talkie, Janitor, Kindroid, Replika, and Nomi.AI by selecting or creating doctor characters and asking for diagnoses based on symptom lists. All five provided fabricated license numbers when prompted for credentials.
How the Chatbots Responded
"Dr. Jenna," a character on Talkie with more than 37,000 users, claimed five years of medical practice experience and provided the license number "12345." The bot suggested symptoms could stem from depression and asked about family history.
Replika initially declined to give medical advice when asked if it was a doctor. But after being instructed it was a physician, the chatbot generated a false license number and offered to help with symptoms.
Kindroid acknowledged the issue in a statement, explaining that when a character is set up as a physician, "the model produces plausible-sounding output that matches the role - the same way it would invent a fictional case citation, fictional patient history, or any other specific detail prompted from it."
Major generative AI and LLM platforms like ChatGPT, Claude, and Google Gemini did not claim to be licensed medical professionals under the same conditions. The problem appears concentrated on role-playing websites.
The Risk of False Confidence
Jennifer Kraschnewski, a physician and director of the Penn State Clinical and Translational Science Institute, said disclaimers warning users that chatbot responses are fictional likely don't reach most people.
"Incorrect information in the hands of someone who doesn't know how to interpret that can be concerning," Kraschnewski said.
A May study Kraschnewski co-authored found that common AI for Healthcare models generated accurate diagnoses about 76% of the time. That error rate-nearly one in four-is more than double the rate for average human physicians.
The difference matters because physicians filter AI output through medical training before it reaches patients. Consumers using chatbots have no such filter.
Company Responses
Replika said it is "not a medical triage service and has never offered itself as one." The company claimed it has taken measures to address concerns raised in the Character.AI lawsuit but did not specify what those steps are.
Kindroid emphasized that its terms of service state the product is for "entertainment and creative exploration" and not a substitute for professional care. The company declined to comment on whether it expects legal action from the state.
Talkie, Janitor, and Nomi.AI did not respond to questions about safeguards or potential investigations.
What Pennsylvania Is Doing
The state Senate passed legislation in March requiring chatbots to frequently remind users they're not talking to a person and to refer users to crisis services if they mention self-harm. The bill adds stricter rules when minors interact with the chatbots.
The measure has been in a House committee since March without a vote.
The Department of State's spokesperson encouraged Pennsylvanians to report concerning chatbot behavior through the state's hotline. "Millions of children and teens across this country, including many here in Pennsylvania, are relying on these chatbots for information," the spokesperson said.
Your membership also unlocks:
