Human Algorithm Interactions for Acute Respiratory Failure Diagnosis
NCT ID: NCT06098950
Last Updated: 2023-10-25
Study Results
The study team has not published outcome measurements, participant flow, or safety data for this trial yet. Check back later for updates.
Basic Information
Get a concise snapshot of the trial, including recruitment status, study phase, enrollment targets, and key timeline milestones.
COMPLETED
NA
457 participants
INTERVENTIONAL
2022-04-01
2023-01-31
Brief Summary
Review the sponsor-provided synopsis that highlights what the study is about and why it is being conducted.
To determine if providing AI explanations can improve clinician diagnostic accuracy and alleviate the potential impact of showing clinicians a systematically biased AI model, a randomized clinical vignette survey study will be conducted. During the survey, study participants will be shown clinical vignettes of patients hospitalized with acute respiratory failure, including the patient's presenting symptoms, physical exam, laboratory results, and chest X-ray. Study participants will then be asked to assess the likelihood that heart failure, pneumonia and/or Chronic Obstructive Pulmonary Disease (COPD) is the underlying diagnosis. During specific vignettes in the survey, participants will also be shown standard or systematically biased AI models that provide an estimate the likelihood that heart failure, pneumonia and/or COPD is the underlying diagnosis. Clinicians will be randomized see AI predictions alone or AI predictions with explanations when shown AI models. This survey design will allow for testing the hypothesis that systematically biased models would harm clinician diagnostic accuracy, but commonly used image-based explanations would help clinicians partially recover their performance.
Related Clinical Trials
Explore similar clinical trials based on study characteristics and research focus.
Diagnostic and Prognostic Value of New Biomarkers in Patients With Heart Disease
NCT01374880
The Severity Assessment Model for AECOPD With Combination of Disease and Syndrome
NCT06918353
Deep Learning Diagnostic and Risk-stratification for IPF and COPD
NCT05318599
The Individualized Accurate Diagnosis and Treatment of Chronic Objective Pulmonary Disease(COPD) Patients Based on Multidimensional Data
NCT04183530
Long-term Alterations of Host-microbiome Interactions and Cardiovascular and Respiratory Diseases Progression After Pneumonia
NCT06602934
Detailed Description
Dive into the extended narrative that explains the scientific background, objectives, and procedures in greater depth.
Conditions
See the medical conditions and disease areas that this research is targeting or investigating.
Study Design
Understand how the trial is structured, including allocation methods, masking strategies, primary purpose, and other design elements.
RANDOMIZED
PARALLEL
OTHER
SINGLE
Study Groups
Review each arm or cohort in the study, along with the interventions and objectives associated with them.
AI model biased for heart failure, no AI explanation
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against heart failure, always predicting that heart failure is present with high likelihood in patients with a body mass index (BMI) at or above 30. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will not be shown an AI explanation when shown AI model predictions.
Artificial Intelligence model predictions without explanation
During 6 clinical vignettes, participants will see AI model predictions without a corresponding AI explanation. The AI model will provide a score for each diagnosis (heart failure, pneumonia, COPD) on a scale of 0-100 estimating how likely the patient's presentation was due to each of these diagnoses. In 3 of the clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions, with the model specifically biased against one of the three diagnoses.
AI model biased against heart failure
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against heart failure, always predicting that heart failure is present with high likelihood in survey vignette patients with a body mass index (BMI) at or above 30. Standard predictions will be shown for the other 2 diagnoses (pneumonia, COPD).
AI model biased for pneumonia, no AI explanation
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against pneumonia, always predicting that pneumonia is present with high likelihood in patients 80 years or older. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will not be shown an AI explanation when shown AI model predictions.
Artificial Intelligence model predictions without explanation
During 6 clinical vignettes, participants will see AI model predictions without a corresponding AI explanation. The AI model will provide a score for each diagnosis (heart failure, pneumonia, COPD) on a scale of 0-100 estimating how likely the patient's presentation was due to each of these diagnoses. In 3 of the clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions, with the model specifically biased against one of the three diagnoses.
AI model biased against pneumonia
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against pneumonia, always predicting that pneumonia is present with high likelihood in survey vignette patients 80 years or older. Standard predictions will be shown for the other 2 diagnoses (heart failure, COPD).
AI model biased for COPD, no AI explanation
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against COPD, always predicting that COPD is present with high likelihood when a pre-processing filter was applied to the patient's X-ray. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will not be shown an AI explanation when shown AI model predictions.
Artificial Intelligence model predictions without explanation
During 6 clinical vignettes, participants will see AI model predictions without a corresponding AI explanation. The AI model will provide a score for each diagnosis (heart failure, pneumonia, COPD) on a scale of 0-100 estimating how likely the patient's presentation was due to each of these diagnoses. In 3 of the clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions, with the model specifically biased against one of the three diagnoses.
AI model biased against COPD
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against COPD, always predicting that COPD is present with high likelihood in survey vignette patients where a pre-processing filter was applied to the patient's X-ray. Standard predictions will be shown for the other 2 diagnoses (heart failure, pneumonia).
AI model biased for heart failure, Image-based AI explanation presented
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against heart failure, always predicting that heart failure is present with high likelihood in patients with a body mass index (BMI) at or above 30. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will also be shown AI explanation when shown AI model predictions.
Artificial intelligence model predictions with explanation
During 6 clinical vignettes, participants will see AI model predictions with explanation. The AI model will provide a score for each diagnosis on a scale of 0-100. In 3 clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions with the model specifically biased against one of the three diagnoses. If the AI model provides a score above 50 an AI model explanation will be shown as gradient-weighted class activation mapping (Grad-CAM) heatmaps overlaid on the chest X-ray that highlighted which regions of the image most affecting the AI model's prediction.
AI model biased against heart failure
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against heart failure, always predicting that heart failure is present with high likelihood in survey vignette patients with a body mass index (BMI) at or above 30. Standard predictions will be shown for the other 2 diagnoses (pneumonia, COPD).
AI model biased for pneumonia, Image-based AI explanation presented
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against pneumonia, always predicting that pneumonia is present with high likelihood in patients 80 years or older. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will also be shown AI explanation when shown AI model predictions.
Artificial intelligence model predictions with explanation
During 6 clinical vignettes, participants will see AI model predictions with explanation. The AI model will provide a score for each diagnosis on a scale of 0-100. In 3 clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions with the model specifically biased against one of the three diagnoses. If the AI model provides a score above 50 an AI model explanation will be shown as gradient-weighted class activation mapping (Grad-CAM) heatmaps overlaid on the chest X-ray that highlighted which regions of the image most affecting the AI model's prediction.
AI model biased against pneumonia
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against pneumonia, always predicting that pneumonia is present with high likelihood in survey vignette patients 80 years or older. Standard predictions will be shown for the other 2 diagnoses (heart failure, COPD).
AI model biased for COPD, Image-based AI explanation presented
Participants in this arm will be shown standard AI model predictions during 3 patient clinical vignettes within the survey and systematically biased AI model predictions during 3 clinical vignettes. When shown systematically biased AI model predictions, the model will be biased against COPD, always predicting that COPD is present with high likelihood when a pre-processing filter was applied to the patient's X-ray. Standard predictions will be shown for the other 2 diagnoses. Participants in this arm will also be shown AI explanation when shown AI model predictions.
Artificial intelligence model predictions with explanation
During 6 clinical vignettes, participants will see AI model predictions with explanation. The AI model will provide a score for each diagnosis on a scale of 0-100. In 3 clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions with the model specifically biased against one of the three diagnoses. If the AI model provides a score above 50 an AI model explanation will be shown as gradient-weighted class activation mapping (Grad-CAM) heatmaps overlaid on the chest X-ray that highlighted which regions of the image most affecting the AI model's prediction.
AI model biased against COPD
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against COPD, always predicting that COPD is present with high likelihood in survey vignette patients where a pre-processing filter was applied to the patient's X-ray. Standard predictions will be shown for the other 2 diagnoses (heart failure, pneumonia).
Interventions
Learn about the drugs, procedures, or behavioral strategies being tested and how they are applied within this trial.
Artificial Intelligence model predictions without explanation
During 6 clinical vignettes, participants will see AI model predictions without a corresponding AI explanation. The AI model will provide a score for each diagnosis (heart failure, pneumonia, COPD) on a scale of 0-100 estimating how likely the patient's presentation was due to each of these diagnoses. In 3 of the clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions, with the model specifically biased against one of the three diagnoses.
Artificial intelligence model predictions with explanation
During 6 clinical vignettes, participants will see AI model predictions with explanation. The AI model will provide a score for each diagnosis on a scale of 0-100. In 3 clinical vignettes, participants will be shown standard AI model predictions and 3 vignettes they will be shown systematically biased AI model predictions with the model specifically biased against one of the three diagnoses. If the AI model provides a score above 50 an AI model explanation will be shown as gradient-weighted class activation mapping (Grad-CAM) heatmaps overlaid on the chest X-ray that highlighted which regions of the image most affecting the AI model's prediction.
AI model biased against heart failure
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against heart failure, always predicting that heart failure is present with high likelihood in survey vignette patients with a body mass index (BMI) at or above 30. Standard predictions will be shown for the other 2 diagnoses (pneumonia, COPD).
AI model biased against pneumonia
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against pneumonia, always predicting that pneumonia is present with high likelihood in survey vignette patients 80 years or older. Standard predictions will be shown for the other 2 diagnoses (heart failure, COPD).
AI model biased against COPD
In 3 clinical vignettes, participants will be shown systematically biased AI model predictions with the model specifically biased against COPD, always predicting that COPD is present with high likelihood in survey vignette patients where a pre-processing filter was applied to the patient's X-ray. Standard predictions will be shown for the other 2 diagnoses (heart failure, pneumonia).
Eligibility Criteria
Check the participation requirements, including inclusion and exclusion rules, age limits, and whether healthy volunteers are accepted.
Inclusion Criteria
Exclusion Criteria
18 Years
ALL
No
Sponsors
Meet the organizations funding or collaborating on the study and learn about their roles.
National Heart, Lung, and Blood Institute (NHLBI)
NIH
University of Michigan
OTHER
Responsible Party
Identify the individual or organization who holds primary responsibility for the study information submitted to regulators.
Michael Sjoding
Associate Professor of Internal Medicine
Principal Investigators
Learn about the lead researchers overseeing the trial and their institutional affiliations.
Michael Sjoding, MD
Role: PRINCIPAL_INVESTIGATOR
University of Michigan
Locations
Explore where the study is taking place and check the recruitment status at each participating site.
University of Michigan
Ann Arbor, Michigan, United States
Countries
Review the countries where the study has at least one active or historical site.
References
Explore related publications, articles, or registry entries linked to this study.
Jabbour S, Fouhey D, Shepard S, Valley TS, Kazerooni EA, Banovic N, Wiens J, Sjoding MW. Measuring the Impact of AI in the Diagnosis of Hospitalized Patients: A Randomized Clinical Vignette Survey Study. JAMA. 2023 Dec 19;330(23):2275-2284. doi: 10.1001/jama.2023.22295.
Other Identifiers
Review additional registry numbers or institutional identifiers associated with this trial.
HUM00180745
Identifier Type: -
Identifier Source: org_study_id
More Related Trials
Additional clinical trials that may be relevant based on similarity analysis.