Skip to main content

Large language model AI chatbots require approval as medical devices


October 17, 2023

Discussion Topic

By the end of 2022, large language models (LLMs) such as Open AI’s ChatGPT or Google’s PaLM had evolved to the point that they could fluently generate conversation-like text. Trained on a body of human-generated text from across news articles, literature, Wikipedia, and the wider Internet, users quickly began trying to use it to have medical conversations such as identifying the cause of a new symptom or seeking therapy-like support on mental health issues. As digital health and regulatory specialists, we argue that when LLMs enter the realm of medical decision making, they become medical devices, and must be regulated as such. While we agree the potential for such technology is significant, our work on numerous medical software applications deployed in clinical practice suggest that in their current form, LLMs risk significant harm and must not attempt to sidestep regulations. We believe this is an addressable challenge and encourage developers to be specific about their intended purpose, calibrate their development to the appropriate risk class, constrain open LLM prompts and responses to improve safety, and to pursue an intentional program of quality management throughout.

View recording

View Slides

Join our next project

Help streamline the path to regulatory and commercial success to optimize health outcomes for the greatest number of patients

Join the Integrated Evidence Plans project

Join us
Not today