Evaluation of Computer-Aided Lung Nodule Detection Software in Thoracic CT for Riverain Technologies LLC
NCT ID: NCT02440139
Last Updated: 2021-11-04
Study Results
Outcome measurements, participant flow, baseline characteristics, and adverse events have been published for this study.
View full resultsBasic Information
Get a concise snapshot of the trial, including recruitment status, study phase, enrollment targets, and key timeline milestones.
COMPLETED
NA
12 participants
INTERVENTIONAL
2015-04-30
2018-03-31
Brief Summary
Review the sponsor-provided synopsis that highlights what the study is about and why it is being conducted.
This is a retrospective, multi-reader, multi-case, (MRMC) randomized reader study.
OBJECTIVE:
Primary: The primary objective of this clinical study is to prove that a user aided with ClearRead CT InSight (CRCTI) is superior to the unaided reader for detecting actionable lung nodules.
Secondary: The secondary objective of this clinical study is to prove that the reader's reading time is not significantly increased when aided with CRCTI.
NUMBER OF SUBJECTS:
Retrospective CT studies from approximately 300 patients will be included in the study. Approximately 100 true positive cases and 200 normal cases.
NUMBER OF READERS:
A reader study with at least ten (10) participating radiologists (US Board Certified) will be conducted.
PRIMARY ENDPOINTS:
Scores given by the radiologists with and without ClearRead CT Insight will be recorded and compared to the true status of the study-cases. The frequency of the scores for each method (Unaided, Aided) will be tabulated and LROC curves constructed along with sensitivity, specificity, PPV, NPV and clinical actions. Additionally, machine nodule detection rate and false positives per patient on normal cases will be measured.
PATIENT POPULATION :
The study will target approximately one hundred (100) patients whose CT nodules were shown to be cancer and two hundred (200) normal patients. The patient population will be consistent with the national lung cancer screening protocols.
Detailed Description
Dive into the extended narrative that explains the scientific background, objectives, and procedures in greater depth.
An initial (baseline) interpretation will be made by each of the radiologists based on the Lung CT in its original form. At a minimum of one month later, each radiologists will again interpret the same images viewing the pair of CRCTI CT series: Two sets of CT images (standard with CADe marks and processed with vessel suppression will be presented on either one large monitor or two adjacent monitors.
During the baseline reading the radiologist will mark the location of the actionable nodules and assign a score. The radiologist will also indicate the recommended method of follow-up (Contrast CT, PET-CT, CT Follow-up, Biopsy).
During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with computer-aided detection (CADe) marks placed and the vessel suppressed same slice with the vessel suppressed view (right image). The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).
Based on the levels of suspicion for each nodule and the associated likelihood ratings, LROC curves will be constructed for both the baseline and the concurrent reads and the significance of any difference will be calculated. The recommendations for further action (CT Follow-up, Contrast CT, PET-CT, or Biopsy) will be used to calculate sensitivity and specificity, PPV and NPV.
Number and types of cases:
Retrospective lung CT image series from approximately 300 patients will be included in the study. Approximately one hundred (100) of the patients will have pathology confirmed cancers and approximately two hundred (200) of the patients will be CTs associated with normal patients. Also included as nodule images are those where the actionable nodule was not acted on at that time, but was detected and acted on based on a subsequent CT. These are the prior images where the nodule can be identified and its location is the same as on the "current" image confirmed by the radiologist expert panel (using a majority of three as the decision criterion).
The selected sample, randomly selected from a larger pool of CT cases will be enriched in the following way:
1. Lung nodules (cancers in this study) will be tumor size T1a (20 mm or less). The proportion of nodules 20 mm or less may be increased since this is where the investigators expect the major impact of this software to be.
2. Non-Solid (ground glass) nodules will be added to the sample (based on availability) to determine the performance of the system on non-solid nodules. For this group, to have sufficient cases, the investigators may have to include benign (non-malignant) non-solid nodules.
3. In this project, the investigators will perform a Machine Test of the ClearRead CT Insight algorithm followed by a reader performance evaluation study. Riverain will provide a system configured with the operating point set to be used for the reader studies and a configuration for an "open" system to be used for machine testing and FROC generation.
Arm 1: a baseline read (no secondary content) and Arm 2: concurrent, CAD augmented read.
1. st Arm: Do baseline (measure time, readers score regions according to action and suspiciousness) - mark all locations of concern
2. nd Arm: Concurrent read (measure time, readers score regions according to action and suspiciousness) - mark all locations of concern
The primary study hypothesis is that the adjunctive use of ClearRead CT Insight is superior to use of standard lung CT images alone, as measured by the area under the LROC curve.
STATISTICAL ANALYSES Accuracy
To evaluate the hypothesis of superiority in terms of improvement in accuracy for ClearRead CT Insight vs. unaided, a mixed effects model (DBM) will be implemented (similar to the model outlined in Dorfman, Berbaum and Metz, 1997), where variance components will be included to account for reader, case, reader by case, reader by modality, case by modality and reader by case by modality. However, it is anticipated that the three-way interaction will be inestimable and will subsequently be dropped from the statistical model. Specifically, the hypothesis to test for superiority of ClearRead CT Insight vs. unaided is:
H0: AUCunaided - AUCClearRead CT Insight ≥ 0.0 vs. HA: AUCunaided - AUCClearRead CT Insight \< 0
The AUC of the LROC is the primary endpoint to evaluate accuracy and the test of interest will be a two-sided 95% confidence interval on the effect of modality (i.e. ClearRead CT Insight minus unaided). Significance will be concluded if the upper bound of the two-sided 95% confidence interval does not include zero. If the null hypothesis (H0) is rejected, the alternative hypothesis (HA) is accepted and the superiority of using the ClearRead CT Insight system will be established.
Time
The second co-primary objective is to evaluate reduction in time spent per image for ClearRead CT Insight vs. unaided. Specifically, the hypothesis to test for superiority of ClearRead CT Insight vs. unaided is:
H0= Tunaided - TClearRead CT Insight ≤ 0 vs. HA= Tunaided - TClearRead CT Insight \> 0
To evaluate the hypothesis of non-inferiority in terms of improved read time for ClearRead CT Insight vs. unaided, a mixed effects model will be implemented (similar to the model outlined in Dorfman, Berbaum and Metz, 1992), where variance components will be included to account for reader, case, reader by case, reader by modality, case by modality and reader by case by modality. However, it is anticipated that the three-way interaction will be inestimable and will subsequently be dropped from the statistical model.
The read times will be tested using a two-sided 95% confidence interval on the effect of modality (i.e. ClearRead minus unaided). Significance will be concluded if the upper bound of the two-sided 95% confidence interval does not include zero.
With either analysis, the use of the mixed model could be modified to employ a bootstrap sampling approach if the model assumptions of the DBM method have been violated. The upper 95 % confidence limit for the difference in the areas under the curves would be calculated using 10,000 bootstraps in the MultiReader MultiCase (MRMC) ROC method.
POWER AND SAMPLE SIZE
The power to detect differences in the AUC of the LROC curve for the proposed statistical analysis using the current design baseline, i.e. 300 cases, each corresponding image read by 10 readers, was assessed through a simulation study. Specifically, the model outlined above was used to simulate 500 datasets across a range of effect sizes, where power was defined as the proportion of datasets that yielded a significant p-value for testing the fixed effect of modality. The method of Dorfman, Berbaum and Metz, 1992 (DBM) was implemented utilizing statistical mixed model theory with jackknife estimates.
These simulations required assumptions regarding the magnitude of the variance components associated with the different random effects. As a pilot study to obtain estimates of variance components was not conducted, variance component estimates from Riverain study SoftView 510(k) (Record # BSSI-PR-09-00006) that uses a similar technology as the proposed ClearRead CT Insight device were used in all power simulations. Please note that the variance components were scaled by the total variance to represent the proportion of total variance explained by each component. All power estimates are dependent upon the appropriateness of the assumed variance components.
Conditions
See the medical conditions and disease areas that this research is targeting or investigating.
Study Design
Understand how the trial is structured, including allocation methods, masking strategies, primary purpose, and other design elements.
RANDOMIZED
PARALLEL
DIAGNOSTIC
DOUBLE
Study Groups
Review each arm or cohort in the study, along with the interventions and objectives associated with them.
Arm 1
Participating radiologists will perform clinical reading on \~300 cases. They will mark all locations of concern (clinically actionable nodules) on conventional thoracic CT images. Computer will measure time, readers score regions according to action and suspiciousness.
No interventions assigned to this group
Arm 2
Participating radiologists will perform clinical reading on \~300 cases. They will mark all locations of concern (clinically actionable nodules) on thoracic CT images aided by ClearRead CT Insight software as the intervention. Computer will measure time, readers score regions according to action and suspiciousness.
ClearRead CT Insight
During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with CADe marks placed and the vessel suppressed same slice with the vessel suppressed view as the intervention. The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).
Interventions
Learn about the drugs, procedures, or behavioral strategies being tested and how they are applied within this trial.
ClearRead CT Insight
During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with CADe marks placed and the vessel suppressed same slice with the vessel suppressed view as the intervention. The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).
Eligibility Criteria
Check the participation requirements, including inclusion and exclusion rules, age limits, and whether healthy volunteers are accepted.
Inclusion Criteria
1. Asymptomatic patients age 55 - 77 with history of smoking
2. Primary Lung Cancer
3. Biopsy Proven (LuRADS 5) with radiology report
4. Screen detected event plus 1 prior CT. 2 year of prior CTs is preferred, if available.
5. \<= 3mm slice spacing, no gaps
6. Use standard reconstruction kernels
7. Maximum of 5 nodules per image
8. Nodules must be 5-20mm in size
9. With or without contrast
Exclusion Criteria
2. Both lungs must be fully visible within the field of view
3. Apices cannot be cropped
4. No excessive motion artifacts
5. Symptomatic patients with co-morbidities
55 Years
77 Years
ALL
No
Sponsors
Meet the organizations funding or collaborating on the study and learn about their roles.
Riverain Technologies
INDUSTRY
Virginia Polytechnic Institute and State University
OTHER
Responsible Party
Identify the individual or organization who holds primary responsibility for the study information submitted to regulators.
ShihChung Ben Lo
Adjunct Professor
Principal Investigators
Learn about the lead researchers overseeing the trial and their institutional affiliations.
Matthew T Freedman, M.D.
Role: STUDY_DIRECTOR
Virginia Polytechnic Institute and State University
Locations
Explore where the study is taking place and check the recruitment status at each participating site.
Arlington Innovation Center: Health Resaerch
Arlington, Virginia, United States
Countries
Review the countries where the study has at least one active or historical site.
References
Explore related publications, articles, or registry entries linked to this study.
Beyer F, Zierott L, Fallenberg EM, Juergens KU, Stoeckel J, Heindel W, Wormanns D. Comparison of sensitivity and reading time for the use of computer-aided detection (CAD) of pulmonary nodules at MDCT as concurrent or second reader. Eur Radiol. 2007 Nov;17(11):2941-7. doi: 10.1007/s00330-007-0667-1. Epub 2007 May 22.
Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992 Sep;27(9):723-31. No abstract available.
Foti G, Faccioli N, D'Onofrio M, Contro A, Milazzo T, Pozzi Mucelli R. Evaluation of a method of computer-aided detection (CAD) of pulmonary nodules with computed tomography. Radiol Med. 2010 Sep;115(6):950-61. doi: 10.1007/s11547-010-0556-6. Epub 2010 Jun 23. English, Italian.
Matsumoto S, Ohno Y, Aoki T, Yamagata H, Nogami M, Matsumoto K, Yamashita Y, Sugimura K. Computer-aided detection of lung nodules on multidetector CT in concurrent-reader and second-reader modes: a comparative study. Eur J Radiol. 2013 Aug;82(8):1332-7. doi: 10.1016/j.ejrad.2013.02.005. Epub 2013 Mar 6.
Obuchowski NA. New methodological tools for multiple-reader ROC studies. Radiology. 2007 Apr;243(1):10-2. doi: 10.1148/radiol.2432060387. No abstract available.
Obuchowski NA, McClish DK. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Stat Med. 1997 Jul 15;16(13):1529-42. doi: 10.1002/(sici)1097-0258(19970715)16:133.0.co;2-h.
Provided Documents
Download supplemental materials such as informed consent forms, study protocols, or participant manuals.
Document Type: Study Protocol and Statistical Analysis Plan
Other Identifiers
Review additional registry numbers or institutional identifiers associated with this trial.
458420
Identifier Type: -
Identifier Source: org_study_id