Predictive algorithms can amplify existing healthcare inequities, but debiasing strategies can improve fairness without sacrificing performance. As AI tools rapidly enter clinical practice, continuous monitoring and equity-focused oversight are becoming non-negotiable.

• Real-world healthcare data often bakes systemic inequities into AI models.
• Debiasing methods work, but removing race alone isn’t enough to fix disparities.
• Health systems need algorithmovigilance—continuous, equity-aware monitoring of deployed AI tools.

Learn how Vanderbilt University Medical Center is applying algorithmovigilance from Dr. Peter Embi 

A recent study highlights the first large-scale clinical red-teaming effort to expose risks in GPT-based models used for healthcare tasks. Across 1,504 responses from GPT-3.5, GPT-4, and GPT-4o, 20% were inappropriate, with errors spanning safety, privacy, hallucinations, and bias—underscoring the need for rigorous testing before clinical deployment.

• 1 in 5 model outputs were unsafe or inaccurate, even in advanced GPT versions.
• Racial and gender biases, hallucinated citations, and privacy breaches were common.
• Clinician-led red teaming is essential to benchmark risks, guide safe adoption, and build trust in AI-driven care

• Patients and care partners weren’t included in this testing, but their input is critical. If you’re deploying patient-facing chatbots or LLM tools, include real users early in red-teaming to catch usability gaps, unclear language, and missed contextual cues that technical and clinical reviewers may overlook.

A framework of 32 performance measures across five domains—discrimination, calibration, overall performance, classification, and clinical utility— outlines which metrics are most appropriate for assessing models intended for medical practice. The authors stress that improper measures can mislead clinicians and developers, potentially resulting in patient harm and higher costs. They recommend prioritizing measures that are proper, interpretable, and aligned with clinical decision thresholds.

Core performance domains

• Discrimination – Can the model separate patients with vs. without the event?
  Recommended metric: AUROC (Area Under ROC Curve).
• Calibration – Do predicted probabilities match observed outcomes?
  Recommended: Calibration plots + slope/intercept assessments.
• Overall Performance – How close are predictions to reality?
  Recommended: Brier Score (scaled) and related R² measures.
• Classification – How accurate are predictions at a decision threshold?
  Caution: Accuracy and F1 are often misleading in clinical settings.
• Clinical Utility – Does the model improve patient care decisions?