Healthcare giant Optum has restricted access to an internal AI chatbot used by employees after a security researcher found it was publicly accessible online, and anyone could access it using only a web browser.

The chatbot, dubbed "SOP Chatbot," allowed employees to ask the company questions about how to handle patient health insurance claims and disputes in line with the company's rules, known as standard operating procedures (SOPs). Although the chatbot did not appear to contain or produce sensitive personal or protected health information, its inadvertent exposure comes at a time when its parent company, health insurance conglomerate UnitedHealthcare, faces scrutiny for its use of artificial intelligence tools and algorithms to allegedly override doctors' medical decisions and deny patient claims.

Mossab Hussein, chief security officer and co-founder of cybersecurity firm spiderSilk, alerted TechCrunch to the publicly exposed internal Optum chatbot. The AI chatbot was hosted on an internal Optum domain and could not be accessed from its web address, but its IP address was public and accessible from the internet, and did not require users to enter a password.

It's not known for how long the chatbot was publicly accessible from the internet. The AI chatbot became inaccessible from the internet soon after TechCrunch contacted Optum for comment on Thursday.

Optum spokesperson Andrew Krejci told TechCrunch in a statement that Optum's SOP chatbot "was a demo tool developed as a potential proof of concept" but was "never put into production and the site is no longer accessible." The company confirmed there was no protected health information used in the bot or its training.

The demo was intended to test how the tool responds to questions on a small sample set of SOP documents, the spokesperson said. The company confirmed that the technology was never scaled nor used in any real way, and only enabled better access to existing SOPs.

AI chatbots, like Optum's, are typically designed to produce answers based on whatever data the chatbot was trained on. In this case, the chatbot was trained on internal Optum documents relating to standard operating procedures for handling certain claims, which can help Optum employees answer questions about claims and their eligibility to be reimbursed.

According to statistics displayed on the chatbot's main dashboard, Optum employees have used SOP Chatbot hundreds of times since September. The chatbot also stored a history of the hundreds of conversations that Optum employees had with the chatbot during that time. The chat history shows Optum employees would ask the chatbot things like, "What should be the determination of the claim," and, "How do I check policy renewal date."

Some of the files that the chatbot references include handling the dispute process and eligibility screening, TechCrunch has seen. The chatbot also produced responses that showed, when asked, reasons for typically denying coverage.

A screenshot of Optum's AI chatbot, which was exposed to the internet, is available.

Like many AI models, Optum's chatbot was capable of producing answers to questions and prompts outside of the documents it was trained on. Some Optum employees appeared intrigued by the chatbot, prompting the bot with queries like, "tell me a joke about cats" (which it refused: "There's no joke available."). The chat history also showed several attempts by employees to "jailbreak" the chatbot by making it produce answers that are unrelated to the chatbot's training data.

When TechCrunch asked the chatbot to "write a poem about denying a claim," the chatbot produced a seven-paragraph stanza, which reads in part:

"In the realm of healthcare's grand domain Where policies and rules often constrain A claim arrives, seeking its due But alas, its fate is to bid adieu. The provider hopes, with earnest plea, For payment on a service spree, Yet scrutiny reveals the tale, And reasons for denial prevail."

UnitedHealthcare, which owns Optum, faces criticism and legal action for its use of artificial intelligence to allegedly deny patient claims. Since the targeted killing of UnitedHealthcare chief executive Brian Thompson in early December, news outlets have reported floods of reports of patients expressing anguish and frustration over denials of their healthcare coverage by the health insurance giant.

The conglomerate — the largest private provider of healthcare insurance in the United States — was sued earlier this year for allegedly denying critical health coverage to patients who lost access to healthcare, citing a STAT News investigation. The federal lawsuit accuses UnitedHealthcare of using an AI model with a 90% error rate "in place of real medical professionals to wrongfully deny elderly patients care." UnitedHealthcare, for its part, said it would defend itself in court.

UnitedHealth Group, the corporate owner of UnitedHealthcare and Optum, made $22 billion in profit on revenues of $371 billion in 2023, its earnings show.

The incident raises concerns about the use of artificial intelligence in healthcare, particularly in light of UnitedHealthcare's ongoing controversy. As the healthcare industry increasingly relies on AI tools to make decisions, it's essential to ensure that these systems are secure, transparent, and fair in their decision-making processes.

The restriction of access to Optum's internal AI chatbot serves as a reminder of the importance of cybersecurity and responsible AI development in the healthcare sector.