Reporting guidelines for clinical studies involving AI medical devices
In 2019 we conducted the world's first systematic review of studies comparing the performance of 'deep learning' AI medical devices and health-care professionals using medical imaging
We found that AI medical devices could in many cases match or surpass the performance of expert clinicians. However most trials' findings were reported inadequately and/or inconsistently.
The need for standardised reporting of AI studies
Inadequate reporting of research studies' findings means that important information is not available for those using the research to make decisions.
If researchers studying AI systems don't investigate for harm caused to certain groups of people, such harms might go unrecognised where they occur, putting patients at risk.
Even if harms are recognised, if they are not reported they cannot be acted upon by people using the study to make decisions.
Decisionmakers might include healthcare professionals, medical device regulators, commissioners, policymakers, and many others.
Inconsistent reporting makes it challenging to compare results and learn lessons from multiple studies at once - a process called systematic review & meta analysis (read more here).
Often this is the best way to learn what the best treatments are for patients.
This is because individual studies have their strengths and weaknesses which can be mitigated by combining several studies together.
Reporting guidelines - CONSORT-AI and SPIRIT-AI
Reporting guidelines represent best practice when communicating the findings of scientific studies.
The EQUATOR network coordinates production of reporting guidelines, and hosts completed guidelines on its website.
Pre-existing guidelines for reporting of randomised controlled trials and study protocols (CONSORT and SPIRIT respectively) do not take account of factors unique to AI medical devices.
In 2020 our group led a collaboration to generate AI-specific extensions to the established CONSORT and SPIRIT reporting guidelines. The CONSORT-AI and SPIRIT-AI extensions ensure studies involving AI are rigorously reported moving forwards.
By convening an international group of patient representatives, policymakers, clinical experts, industry partners, academic researchers and journal editors, we ensured CONSORT-AI and SPIRIT-AI meet the needs of all stakeholders.
CONSORT-AI and SPIRIT-AI were published on 9th September 2020 simultaneously in Nature Medicine, The Lancet Digital Health, and the British Medical Journal (BMJ).
Read the papers:
CONSORT AI can be read at Nature Medicine, The Lancet Digital Health, or via the BMJ.
SPIRIT-AI can also be read at Nature Medicine, The Lancet Digital Health, or via the BMJ.