Trial Outcomes & Findings for Testing of Computer Aided Detection Software for Riverain Medical Group (NCT NCT00906789)

Results Overview

Standard methods for LROC methodology and statistical analysis were used. We are testing two different types of software using different cases, but the same radiologists to control for radiologist differences. LROC is Localized Receiver Operating Characteristic. LROC measures the trade-offs between sensitivity and specificity as radiologists use different levels of suspicion of disease. This analysis is for the software that decreases the visibility of the ribs and clavicles while preserving (and potentially enhancing) the visibility of the lungs and lung diseases. In this case, the level of suspicion recorded was for the radiologist's concern that a finding did or did not represent cancer. Please note that the FDA approved indications for use is to detected nodules that may represent cancer, but in our study scoring for a true finding was based on whether or not the nodule did represent cancer. A larger number, if statistically significant, indicates that that method is better.

Recruitment status

COMPLETED

Target enrollment

15 participants

Primary outcome timeframe

Three days of experiment over 3-5 months, varied by participant

Results posted on

2014-09-01

Participant Flow

Recruitment between April 9, 2009 and June 15, 2009. Recruited by email and phone calls to individuals meeting entry criteria specified in the protocl.

All recruited individuals met entry criteria. None were excluded or dropped from the study. Training in the use of studied device (software) occurred immediately prior to the experiment. Training took 55 to 80 minutes, depending on the individuals speed. Each individual served as his/her own control, so all participants were in both groups

Participant milestones

Participant milestones
Measure	Radiologists Radiologists who have certification by the American Board of Radiology Riverain OnGuard and SoftView Software : This is an observer performance study. Radiologists will interpret chest radiographs without and then with the Riverain software, Two types of software are tested: SoftView (TM). SoftView decreases the visibility of the ribs and clavicles on chest radiographs. OnGuard marks locations on chest radiographs meeting some of the software signs of lung nodules, a method often called Computer Aided Detection (CADe).
Overall Study STARTED	15
Overall Study COMPLETED	15
Overall Study NOT COMPLETED	0

Reasons for withdrawal

Withdrawal data not reported

Baseline Characteristics

Testing of Computer Aided Detection Software for Riverain Medical Group

Baseline characteristics by cohort

Baseline characteristics by cohort
Measure	Radiologists n=15 Participants Radiologists who have certification by the American Board of Radiology Riverain OnGuard CAD Software : This is an observer performance study. Radiologists will interpret chest radiographs without and then with the Riverain software, both SoftView (TM) OnGuard (TM) CADe Software with be tested
Age, Customized number of participants	15 participants n=93 Participants
Sex: Female, Male Female	5 Participants n=93 Participants
Sex: Female, Male Male	10 Participants n=93 Participants
Region of Enrollment United States	15 participants n=93 Participants

PRIMARY outcome

Timeframe: Three days of experiment over 3-5 months, varied by participant

Population: 122 subjects had cancer that potentially could be detected on their chest radiograph. Power analysis showed that sample size of 351 patients, in a 2:1 ratio of nodule absent to present patients was selected to provide 80% power to detect a difference in areas under the curve of 0.10 or greater.

Standard methods for LROC methodology and statistical analysis were used. We are testing two different types of software using different cases, but the same radiologists to control for radiologist differences. LROC is Localized Receiver Operating Characteristic. LROC measures the trade-offs between sensitivity and specificity as radiologists use different levels of suspicion of disease. This analysis is for the software that decreases the visibility of the ribs and clavicles while preserving (and potentially enhancing) the visibility of the lungs and lung diseases. In this case, the level of suspicion recorded was for the radiologist's concern that a finding did or did not represent cancer. Please note that the FDA approved indications for use is to detected nodules that may represent cancer, but in our study scoring for a true finding was based on whether or not the nodule did represent cancer. A larger number, if statistically significant, indicates that that method is better.

Outcome measures

Outcome measures
Measure	Radiologists Control for SoftView n=368 chest radiographs Control image interpretation unaided by software of either type. This is the control for the SoftView experiment. It uses the same radiologists (to avoid a bias that might result from using different radiologists) as the OnGuard Computer-aided detection software, but different cases.	Radiologists Using Softview Software n=368 chest radiographs This software suppresses the visibility of the ribs and clavicles potentially revealing non-calcified nodules make less conspicuous by the bones projected on top of the nodules
Improvement in Cancer Detection as Measured by Localized Receiver Operating Characteristic) LROC Changes Under the LROC Curve.	0.460 unitless Interval 0.435 to 0.486	0.558 unitless Interval 0.539 to 0.578

SECONDARY outcome

Timeframe: Three days of experiment over 3-5 months, varied by participant

Sensitivity and specificity were calculated using the radiologists' responses of recommendations for follow-up with CT or biopsy. Truth was whether or not the nodule identified was found to be cancer. Sensitivity is the percentage of correct identification of a positive case (a case with cancer). Specificity is the percentage of negative cases (those without cancer) that were correctly identified as not having cancer. The mean values of 15 radiologists are used.

Outcome measures

Outcome measures
Measure	Radiologists Control for SoftView n=15 Participants Control image interpretation unaided by software of either type. This is the control for the SoftView experiment. It uses the same radiologists (to avoid a bias that might result from using different radiologists) as the OnGuard Computer-aided detection software, but different cases.	Radiologists Using Softview Software n=15 Participants This software suppresses the visibility of the ribs and clavicles potentially revealing non-calcified nodules make less conspicuous by the bones projected on top of the nodules
Sensitivity and Specificity Using SoftView Software Sensitivity	49.5 percentage of cases Interval 45.9 to 53.0	66.3 percentage of cases Interval 63.1 to 69.7
Sensitivity and Specificity Using SoftView Software Specificity	96.1 percentage of cases Interval 95.0 to 97.1	91.8 percentage of cases Interval 89.9 to 93.5

OTHER_PRE_SPECIFIED outcome

Timeframe: 5 months

Population: 81 of the 263 radiographs contained a non-calcified nodule that had been diagnosed as lung cancer. Power calculation showed 246 patients, in a 2:1 ratio of nodule absent to present would provide 80% power to detect a difference in areas under the curve of 0.10 or greater.

This reports the comparison of the detection of lung nodules that were proven to represent lung cancers. It compares the results of two versions of computer-aided detection software: OnGuard 1.0 from 2001 and OnGuard 5.1 from 2009. The results represent the responses of radiologists when they use one or the other types of software. To compare radiologists' results with the two types of software, the measurement analyzed was the difference in the areas under the localized receiver operating characteristic curve (LROC). The results from the 15 participating radiologists were averaged (mean value). The area under the LROC curve is a measure of the trade-offs between sensitivity and 1-specificity that occurs as the level of certainty of a positive finding changes. It is normally reported as a decimal without units. In this study dsign, a lower number indicates that the new method (OnGuard 5.1), if statistically significant, if better.

Outcome measures

Outcome measures
Measure	Radiologists Control for SoftView n=253 number of radiographs Control image interpretation unaided by software of either type. This is the control for the SoftView experiment. It uses the same radiologists (to avoid a bias that might result from using different radiologists) as the OnGuard Computer-aided detection software, but different cases.	Radiologists Using Softview Software This software suppresses the visibility of the ribs and clavicles potentially revealing non-calcified nodules make less conspicuous by the bones projected on top of the nodules
Difference in the Area Under the LROC Curve Comparing OnGuard 1.0 and OnGuard 5.1	-0.043 unitless Interval -0.075 to -0.01	—

Adverse Events

Participants

Serious events: 0 serious events

Other events: 0 other events

Deaths: 0 deaths

Serious adverse events

Adverse event data not reported

Other adverse events

Adverse event data not reported

Additional Information

Matthew T. Freedman

Georgetown University Medical Center

Phone: 4105429680

Email: [email protected]

Results disclosure agreements

Principal investigator is a sponsor employee Sponsor can review results communications prior to public release; review by sponsor must be completed within 60 days. Public release of results cannot occur prior to the completion of the FDA review of submitted results.
Publication restrictions are in place

Restriction type: OTHER