Randomized Clinical Trials vs. Validation Cohort Studies of AI/ML-Based SaMDs: What Are the Differences and Guidelines for Appraisal of the Clinical Evidence?

Description: Artificial Intelligence and Machine Learning (AI/ML) software as medical device (SaMD) has the potential to improve disease diagnosis and prognosis to enhance healthcare. Models have been investigated and approved for tasks such as improved cancer diagnosis, emergency department triage, and intensive care unit decision support. However, despite the potential of AI/ML-SaMDs to improve patient care, clinical adoption has many barriers. In order to implement such systems in clinical settings, certain limitations like the lack of generalizability, bias, or even an inability to demonstrate a clinically meaningful benefit, must be overcome.
Randomized Clinical Trials (RCTs) are the gold-standard and a pre-requisite for large scale clinical adoption of an intervention; and recent studies show a low but increasing global number of RCTs in the field. Interestingly, RCT low numbers are in contrast to the large numbers of validation cohort studies of such interventions.
Cohort studies and RCTs are both scientific research designs types, with different purposes and distinct methodologies. Validation cohort studies focus on assessing performance of measurement tools or diagnostic tests; while RCTs are designed to evaluate the effectiveness of interventions in a controlled and randomized way, using blinding as a minimization bias tool.
Still, the quality of RCTs with AI/ML-SaMD arms should be considered using the SPIRIT-AI and CONSORT-AI guidelines in the design of such trials to increase transparency, reproducibility, and inclusivity. Of course, we cannot formally assess every potential iteration of an AI/ML-SaMD through an RCT; and local, independent cohort studies can also provide good quality evidence to encourage early adoption.
This session will debate the need for different types of trustworthy, transparent and reproducible scientific research designs to address the inherent opacity and black box nature of such devices, so that we can truly interpret and appraise the clinical evidence within AI/ML-SaMDs clinical evaluation.