The Missing Middle of Healthcare AI

AI has arrived in healthcare. It is drafting notes, flagging sepsis, summarizing charts, assisting in surgery, and powering chatbots that talk directly to patients. If you work in a health system, chances are you’ve already been pitched half a dozen tools this month.

The momentum is real. So is the risk.

A recent Reuters investigation laid bare what can happen when AI tools are deployed without strong oversight, including surgical systems that misidentified anatomy in the operating room. In healthcare, a “small error” is rarely small. A momentary glitch can become a patient safety event.

Healthcare runs on a simple principle: benefit must outweigh risk. When an AI tool is deployed, who owns that balance over time?

Too often, governance ends when the contract is signed. A vendor is vetted. Performance is reviewed. Security is tested. The team survives implementation. 

Then what? 

The Missing Middle

“AI governance is advancing, but scientific rigor often stops at deployment. The ‘missing middle’ is the operating layer where clinical translational science meets learning health systems — where a ‘validated’ model becomes a trustworthy, continuously improving clinical capability,” said Shauna Overgaard, Director of AI Strategy and Frameworks, Mayo Clinic.

“This requires continuous post-deployment evaluation in the sociotechnical system, so patient impact, adoption, performance, equity, workflow fit, outcomes, and economics are visible in near real time. The real decision-grade test is whether AI measurably improves patient outcomes and experience and delivers ROI once total cost of ownership and clinician time are accounted for.

What’s missing isn’t consensus on the need, but shared, practical approaches for how to do this. That intelligence must be captured and shared so teams can scale safely without reinventing the guardrails.”

AI models change as their environments change. Different patients. Different workflows. Different documentation habits. A model that performs beautifully in one health system can degrade and fail in another.

Some large systems now have dedicated organizational infrastructure to support AI policy, implementation, and adoption. A few have thoughtful review processes. Many are still building this muscle. Smaller and resource-constrained settings are often left to figure it out on their own.

Vendors feel this, too. Without shared expectations for transparency and performance reporting, every partnership becomes a bespoke negotiation. Every dashboard is different. Every definition of “working” is subjective.

This is not scalable. 

Start Small. Monitor Everything.

Here at the Digital Medicine Society, we began by asking a practical question: where can AI deliver value safely, right now?

In our AI Implementation Playbook, we introduced the Stackable Wins Pyramid. This simple approach guides health systems to start with high-volume, lower-risk use cases. Demonstrate value, earn trust, and build capabilities. Then move up the pyramid toward more complex clinical decision support. 

This is change management 101. When teams see early wins, trust builds. When deployment helps instead of disrupts, wielding dashboards to drive adoption is not necessary. 

But even these lower-risk use cases need oversight and governance. 

That is why we developed Minimum Monitoring Stacks for generative and predictive AI. These define the baseline data you need to know whether the model you turned on Tuesday is still performing as expected on Friday.

Three Gaps we Have to Close.

For AI to become infrastructure, we must focus on three priorities.

1. No More Black Box Deployments

Generalizability cannot be assumed. A sepsis model trained at a major academic center may stumble in a rural hospital with different workflows and patient demographics.

Every system needs the ability to test locally before going live. Sandbox environments. Shadow mode. Retrospective validation. If we cannot demonstrate that a model works for our patients, in our environment, we should not deploy it.

“Traditional governance in healthcare was built around standing committees and periodic reviews, and that model doesn’t work for AI,” said Justin Norden, CEO and co-founder of Qualified Health. “Unlike traditional software, AI systems adapt and drift as data, workflows, and populations change. Governing them requires continuous auditability, traceability, and observability, with the ability to inspect source data and reasoning and learn from real-world performance over time. Anything less leaves systems exposed to risk.”

2. From One-Time Validation to Continuous Oversight

Models drift. Coding practices change. Patient populations shift. A readmission model that performs well on Day 1 can slowly degrade by Day 100.

We need real-time monitoring that functions like a check engine light. Clear thresholds. Clear escalation paths. Clear accountability for who steps in and what happens next.

3. Stop Reinventing Procurement Every Time

Today, most AI contracting in healthcare is still bespoke. Each health system brings its own set of governance requirements. Each vendor responds with its own interpretation of validation, monitoring, and reporting obligations. The result is a prolonged negotiation process where both sides are trying to define expectations from scratch.

Standardizing the commercial “ask” around validation, monitoring, and transparency is the only sustainable pathway to rigor and trustworthiness. 

It is Time to Act

Here at the Digital Medicine Society, we refuse to simply admire problems. This is why we are convening a pre-competitive, multi-stakeholder initiative to operationalize AI governance. The work brings together regulators, health systems, virtual care providers, technology companies, and emerging AI developers to define what responsible deployment actually looks like in practice.

We will create practical tools that health systems and vendors can use immediately, including enterprise risk toolkits, local validation guides, monitoring dashboards, escalation protocols, and procurement templates that embed clear expectations from the start of a partnership.

We believe in the promise of AI to fundamentally change how we care for people, improving outcomes, access, and the lives of the people who care for us, whether they are busy clinicians or unpaid care partners. We also believe that to deliver on that promise, we must come together to build the systems that make it dependable over time.

—

The Connected Health Collaborative Community (CHcc), co-hosted by the Digital Medicine Society (DiMe) and Consumer Technology Association (CTA) and in partnership with the FDA is convening a pre-competitive project to turn this blueprint into reality; click here to learn more.