Evidence standards for digital health technologies

For digital health technologies (incuding AI) to help patients, they have to work as expected.

Our team were selected by the National Institute for Health and Care Excellence (NICE) to co-develop their new Evidence Standards Framework (ESF) for Digital Health Technologies (DHTs).

  • Originally created in 2018, the ESF is a tool that helps commissioners systematically evaluate a DHT for potential adoption to UK health and care settings, taking into account factors such as clinical utility, economic value and data governance.

  • NICE sought to update their ESF to include technologies incorporating artificial intelligence algorithms which were excluded from the original framework.

  • To do this they commissioned an academic partnership to produce a classification system for AI-DHTs accompanied by AI-specific evidence standards.

We worked with collaborators at Imperial College London and the Alan Turing Institute to deliver this work:

  1. First, we performed a literature review of classification systems.

  2. This was followed by a modified multi-stakeholder Delphi consensus process.

The standards were published in August 2022. They incorporate themes such as:

  • Dataset composition and handling

  • Product testing

  • Mitigation of algorithmic bias

  • Post-deployment monitoring

Along with the updated standards for non-AI DHTs, the ESF provides commissioners with a first-in-class tool with which to evaluate AI-DHTs and ensure they are safe, effective and fair before their deployment into health and care systems in the UK and abroad.

  • Read the Evidence Standards Framework on the NICE website by clicking here.

  • Read our announcement piece published in The Lancet Digital Health by clicking here.

  • View the academic report we produced with Imperial College London and The Alan Turing Institute by clicking here.