The prevalence of digital health products (DHPs) in numbers, capabilities, and acceptance – which was once considered ‘the wild west – continues to offer promising solutions for improving health outcomes and changing behaviors. Yet, despite significant advances in recent years, the confidence of key decision-making stakeholders remains relatively low. Evidence is needed to determine the reliability and value of digital health products. So how can we harmonize evidentiary practices to evaluate clinical value for effective translation rigorously?
There is a need for objective, transparent, and standards-based evaluation of DHPs that can bring clarity to the digital health marketplace. DiMe’s new framework, Evidence in Digital Health for EFfectiveness of INterventions with Evaluative Depth (Evidence DEFINED), was recently published in NPJ Digital Medicine. The Evidence DEFINED workgroup developed this groundbreaking framework that provides a standardized approach to assessing evidence for digital health products to address this challenge.
The need for the Evidence DEFINED framework
The digital health industry has over 78 proposed frameworks for evaluating evidence for DHPs. Due to the slow and non-targeted assessment process, this needs to be clarified for payers, health systems, and the industry. Existing frameworks may not facilitate DHPs rapid and rigorous evidence reviews. Robust evidence is required to determine the reliability and value of DHPs and, without it, payers lack sufficient information to determine which DHPs may be safe, effective, and improve the lives of their members. Not only does this delay access to digital solutions for patients, but it hinders adoption and viable commercialization pathways for digital product developers.
Three gaps in prior frameworks, but addressed in Evidence DEFINED, include:
- Prior frameworks are not adequate to address evidence considerations unique to digital health.
- For example, products with poor user experiences can cause attrition to even the most motivated patients (with favorable outcomes, irrespective of treatment effects). Thus, better-designed products may retain only a biased subset of patients who tend to show strong outcomes. This may skew low-quality products toward favorable clinical evaluations. These considerations may receive inadequate attention in routine DHP assessments.
- Prior frameworks do not specify evidence-quality criteria requiring increased vigilance.
- For example, some products may need to adhere more to trial registration and reporting best practices. This nonadherence is reflected in the 11% result rates publicly reported for registered DH trials, despite the NIH recommending reporting within 12 months. Such a low reporting result suggests that negative results of DH trials may not be reported publicly, which may prevent appropriate evidence evaluation and quality.
- Prior frameworks rarely leverage established, robust methodologies that were developed for non-digital interventions.
- Existing methodologies and established evidence evaluation have been developed extensively for non-digital products. Some principles pertain to DHPs that the community can build upon established best practices instead of reinventing the wheel.
Designed to implement
Evidence DEFINED is designed to facilitate standardized evidence assessment and guide digital health solutions providers who wish to generate evidence that drives its adoption. It distinguishes itself by incorporating elements intended to increase both rigor and speed that use these four simple steps:
Step 1. Screen for failure to meet absolute requirements
Each stakeholder organization starts by specifying its requirements per its priorities. This step ensures flexibility in the needs of individual organizations and avoids investing effort in products that are not candidates for adoption. Screening requirements may include specific privacy/compliance policies (e.g., HIPAA, SOC2), FDA-authorized products (E.g., FDA-cleared or granted medical class I, II), patient-facing language written at a targeted reading level (e.g., to comply with Medicaid guidelines), and more.
Step 2. Apply an established evidence assessment framework
Use the established evidence assessment framework developed for non-digital interventions (e.g., GRADE) to assign a rating to the DHP. For each risk of bias, imprecision, inconsistency, indirectness, and publication bias, the decision maker can decrease their level of certainty to one or two levels (e.g., from high to moderate).
Step 3. Apply the Evidence DEFINED supplementary checklist
Apply the checklist of the 21 evidence quality criteria to address evidence quality considerations that are unique to digital health or may require greater vigilance in digital health.
Step 4. Make actionable, defensible recommendations
Apply evidence-to-recommendation guidelines to generate defensible and actionable recommendations regarding appropriate adoption levels for digital health products.
By addressing evidence quality criteria unique to digital health interventions, increasing vigilance in the current regulatory context, and leveraging established evidence evaluation methodologies, the framework offers a more comprehensive and standardized approach to evidence assessment.
Looking ahead
Despite the challenges and complexity in evaluating the effectiveness and quality of digital health products, the ecosystem must embrace doing the right thing, even when difficult. There are no shortcuts when obtaining the appropriate evidence, and it is essential to prioritize thorough evaluation. Evidence DEFINED is the new standard of excellence framework for evaluating the clinical assessment of DHPs. It defines ‘what good looks like’ regarding evidence evaluation and implementation to ensure DHPs can go to market faster, ensuring these products serve all those who stand to benefit from them.
Get involved
- Learn more from the co-authors of Evidence DEFINED by joining us for a DiMe Journal Club on Thursday, July 20, at noon ET, and join the discussion. Register now.
Get involved in DiMe’s upcoming Integrated Evidence Plans (IEP) for Digital Health Products project to streamline the path to regulatory and commercial success to optimize health outcomes for the greatest number of patients. Share your interest here.
