Large language model AI chatbots require approval as medical devices
Date/Time
October 17, 2023
Discussion Topic
By the end of 2022, large language models (LLMs) such as Open AI’s ChatGPT or Google’s PaLM had evolved to the point that they could fluently generate conversation-like text. Trained on a body of human-generated text from across news articles, literature, Wikipedia, and the wider Internet, users quickly began trying to use it to have medical conversations such as identifying the cause of a new symptom or seeking therapy-like support on mental health issues. As digital health and regulatory specialists, we argue that when LLMs enter the realm of medical decision making, they become medical devices, and must be regulated as such. While we agree the potential for such technology is significant, our work on numerous medical software applications deployed in clinical practice suggest that in their current form, LLMs risk significant harm and must not attempt to sidestep regulations. We believe this is an addressable challenge and encourage developers to be specific about their intended purpose, calibrate their development to the appropriate risk class, constrain open LLM prompts and responses to improve safety, and to pursue an intentional program of quality management throughout.
