Effect of Intelligent Tutor Induced Pausing on Learning Simulated Surgical Skills

NCT ID: NCT06235788

Last Updated: 2024-02-05

Basic Information

• Recruitment Status: RECRUITING
• Clinical Phase: NA
• Total Enrollment: 129 participants
• Study Classification: INTERVENTIONAL
• Study Start Date: 2024-01-30
• Study Completion Date: 2024-09-30

Brief Summary

Traditional training of surgical technical skills relies on mentorship from experienced surgeons, who continuously evaluate and change trainee performance to prevent errors and potential patient harm by providing verbal instructions. These educators may also pause the procedure, explaining the risks associated with the trainee's actions, and may personally demonstrate proper techniques to the students. Studies examining pausing while providing medical care outline that these approaches allow for learning.

An artificial intelligent (AI) tutoring system, the Intelligent Continuous Expertise Monitoring System (ICEMS), improves learning in a surgical simulated operation by providing trainees with verbal instructions upon error identification. However, the effect of including a pause during this AI teaching has not been studied. Therefore, the ICEMS post-error identification methodology has been altered to include a pause with the intelligent tutor voice instruction.

The aim of this study is to determine the effect of pausing on surgical skill acquisition and transfer among pre-medical and medical students. This will be done by comparing their performance in repeated simulated tumour resection tasks.

Detailed Description

Background: Surgical skill assessment is shifting from a quantitative, time-based approach towards a qualitative evaluation of a trainee's competency. During surgical procedures, instructors continuously monitor trainee performance and utilize various teaching methods focused on enhancing acquisition of surgical skills. One such method includes pausing the operation, either to outline the risks associated with the trainee's performance or to personally demonstrate the best practice technique(s). Pausing in such situations has been shown to allow learners to re-assess best practice, interrupt negative momentum, and allow for learning. Specifically, pausing after an error can prevent introduction of new information that may affect one's ability to reflect on their error and reduce stress before continuing.

Rationale: The ICEMS, an AI tutoring system, was developed by our group using a Long-Short Term Memory deep learning algorithm to assess surgical performance and provide guidance. This was then integrated with the NeuroVR simulation platform. Using this AI system, the provision of verbal feedback on error identification demonstrated the potential of intelligent tutoring to improve learning in two previous randomized control trials (RCTs). However, these RCTs did not incorporate pausing methodology post-error identification. To further emulate the mentorship of an experienced surgeon in a clinical setting, the ICEMS platform has been modified to both initiate pausing when learner error is identified and provide a video demonstrating expert performance.

Research aims: To compare the effect of incorporating a pause after intelligent tutor instruction to intelligent tutor instruction alone on medical and pre-medical students' surgical skill acquisition and skill transfer.

Hypotheses:

1. The pause with video group will significantly improve the composite score between the first and sixth repetition of the practice scenario.

2. The pause with video group will have a composite score statistically higher than the control group in the sixth repetition of the practice scenario.

3. The pause with video group will have a composite score statistically non-inferior to the pause without video group in the sixth repetition of the practice scenario.

4. The pause with video group will have a global OSATS score statistically higher than the control group in the realistic scenario.

5. The pause with video group will have a global OSATS score statistically non-inferior to the pause without video group in the realistic scenario.

6. The pause with video group will not elect a difference in emotional stress or cognitive load compared to the control group.

Specific objectives:

1. To assess if the efficacy of AI mediated real-time tailored feedback combined with pausing methodology is statistically superior to AI mediated real-time tailored feedback alone in improving medical students' surgical skills on two virtual reality surgery tasks.

2. To determine if different emotions and cognitive load are elicited by the pausing methodology as compared to only hearing the AI mediated feedback.

Design: A three-arm single blinded randomized controlled trial of AI feedback with pausing methodology and an expert demonstration video versus AI feedback with only pausing methodology versus AI feedback alone.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre.

Participants: Students who are enrolled in a Quebec medical school in a preparatory year, and first and second year.

Task: Using the NeuroVR surgical simulator by CAE Healthcare, resect a simulated practice tumour six times and a complex simulated realistic brain tumour once using an Ultrasonic Aspirator and Bipolar pincers while minimizing bleeding and preserving the surrounding, simulated healthy brain structures.

Intervention: A 90-minute training session where participants will have seven simulated subpial tumour resection attempts (six repetitions of a simple practice scenario and one attempt at a complex realistic scenario). All participants will receive auditory feedback from the ICEMS but will differ in what follows:

1. Continuously perform the procedure (i.e., no pause) (Group 1);

2. A pause followed by a reflection period (Group 2);

3. A pause followed by an expert-level demonstration video and a reflection period (Group 3).

Auditory feedback will be based on 4 metrics:

1. Instrument tip separation distance;

2. Low bipolar force;

3. High aspirator force;

4. High bipolar force. Initially, feedback will only be given based on the first metric (instrument tip separation), but once a repetition is completed without receiving feedback, the subsequent repetition will assess the next metric in the list above, and so on and so forth.

Main outcomes and measures:

The two co-primary outcomes are:

1. The improvement in surgical performance, as dictated by the composite score computed by the previously validated evaluation module of the ICEMS. Performance improvement is measured by the difference in composite score between each of the 6 repetitions of the practice scenario. Transfer of learning will be measured by the participant's composite score for the complex realistic scenario.

2. The performance score of the participants in the complex realistic scenario, assessed by two blinded experts using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale.

The secondary outcome is the differences in the strength of emotions elicited, measured before the practice scenario, immediately before the realistic scenario, and after completion of all attempts using the Duffy's Medical Emotional Scale (MES). Cognitive load will also be measured following completion of all tasks using Leppink's Cognitive Load Index (CLI). Both outcomes are measured using self-reports.

Statistical Analysis Plan: Participant data will be anonymized and stored. The ICEMS will assess the participant's surgical performance and provide a performance score at 0.2 second intervals throughout each repetition of the simulated surgical task. An average composite score will then be calculated for each repetition. Using ANCOVA, improvement in performance and participant learning will be assessed by comparing the composite score of the first practice scenario repetition (baseline) and the composite score of the sixth repetition (summative). Meanwhile, the composite score of the complex realistic scenario will be used to assess the transfer of learning using a one-way ANOVA. With an effect size of 0.25 and a significance of 0.05, a total sample size of 129 provides 80% power to detect a significant interaction.

Videos of participant performance in the complex realistic scenario will be evaluated by two blinded expert raters using the OSATS global rating scale. The OSATS score will be analyzed between groups using a one-way ANOVA to compare efficiency of learning and skill retention.

Emotional changes before, during, and after learning in the simulated scenarios will be evaluated using a two-way mixed ANOVA, while one-way ANOVA will be used to assess cognitive load after learning.

Conditions

Surgical Education

Study Design

• Allocation Method: RANDOMIZED
• Intervention Model: PARALLEL
• Primary Study Purpose: HEALTH_SERVICES_RESEARCH
• Blinding Strategy: DOUBLE

Study Groups

Control Group Intelligent Continuous Expertise Monitoring System (ICEMS) only verbal feedback group

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario.

Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Group Type: NO_INTERVENTION

No interventions assigned to this group

Experimental Group ICEMS verbal feedback with pause group

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario.

Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Group Type: EXPERIMENTAL

Experimental Group ICEMS verbal feedback with pause group

Intervention Type: BEHAVIORAL

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down and reflecting for 22 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Experimental Group ICEMS verbal feedback with pause and expert-level video demonstration

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario.

Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Group Type: EXPERIMENTAL

Experimental Group ICEMS audio feedback with pause and expert video

Intervention Type: BEHAVIORAL

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down, turning their attention to a 9-second expert-level demonstration video, and reflecting for 13 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Interventions

Experimental Group ICEMS verbal feedback with pause group

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down and reflecting for 22 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Intervention Type: BEHAVIORAL

Experimental Group ICEMS audio feedback with pause and expert video

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down, turning their attention to a 9-second expert-level demonstration video, and reflecting for 13 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Intervention Type: BEHAVIORAL

Eligibility Criteria

Exclusion Criteria

• Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator

• Minimum Eligible Age: 18 Years
• Eligible Sex: ALL
• Accepts Healthy Volunteers: Yes

Sponsors

McGill University

OTHER

Sponsor Role: lead

Responsible Party

Rolando Del Maestro

Director, Neurosurgical Simulation and Artificial Intelligence Learning Centre

Responsibility Role: PRINCIPAL_INVESTIGATOR

Principal Investigators

Rolando F Del Maestro, MD, PhD

Role: PRINCIPAL_INVESTIGATOR

McGill University

Locations

Neurosurgical Simulation and Artificial Intelligence Learning Centre

Montreal, Quebec, Canada

Site Status: RECRUITING

Countries

Canada

Central Contacts

Rolando F Del Maestro, MD, PhD

Role: CONTACT

rolando.del_maestro@mcgill.ca

519 708 0346

Vanja Davidovic

Role: CONTACT

vanja.davidovic@mail.mcgill.ca

6132964855

Facility Contacts

Rolando F Del Maestro, MD, PhD

Role: primary

rolando.del_maestro@mcgill.ca

519 708 0346

Vanja Davidovic

Role: backup

vanja.davidovic@mail.mcgill.ca

6132964855

References

Yilmaz R, Winkler-Schwartz A, Mirchi N, Reich A, Christie S, Tran DH, Ledwos N, Fazlollahi AM, Santaguida C, Sabbagh AJ, Bajunaid K, Del Maestro R. Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation. NPJ Digit Med. 2022 Apr 26;5(1):54. doi: 10.1038/s41746-022-00596-8.

Reference Type: BACKGROUND

PMID: 35473961 (View on PubMed)

Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

Reference Type: BACKGROUND

PMID: 35191972 (View on PubMed)

Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

Reference Type: BACKGROUND

PMID: 31373651 (View on PubMed)

Lee JY, Szulewski A, Young JQ, Donkers J, Jarodzka H, van Merrienboer JJG. The medical pause: Importance, processes and training. Med Educ. 2021 Oct;55(10):1152-1160. doi: 10.1111/medu.14529. Epub 2021 May 1.

Reference Type: BACKGROUND

PMID: 33772840 (View on PubMed)

Other Identifiers

2010-270, NEU-09-042-Trial 4

Identifier Type: -

Identifier Source: org_study_id