Effect of a Competence Based Medical Education Programme on Training Quality in Intensive Care Medicine.

NCT ID: NCT04278976

Last Updated: 2020-02-20

Basic Information

• Recruitment Status: UNKNOWN
• Clinical Phase: NA
• Total Enrollment: 38 participants
• Study Classification: INTERVENTIONAL
• Study Start Date: 2019-02-01
• Study Completion Date: 2021-01-31

Brief Summary

This study evaluates whether the implementation of a competency-based medical education program called CoBaTrICE (Competency Based Training program in Intensive Care in Europe) provides higher levels of competency in comparison with the current official time-based program in Intensive Care Medicine (ICM) in Spain.

The hypothesis will be confirmed or rejected through a multicenter cluster randomized trial of 14 ICU Departments from 14 academic referral hospitals located in Spain. A total of 38 trainees on the 3rd year of the specialization period will be followed during the three years of their specific training period in Intensive Care Medicine. CoBaTrICE (seven hospitals) will be compared with the current official model of training in ICM in Spain (seven hospitals), which is based on exposure to experiences through clinical rotations. The implementation of CoBaTrICE will include the three following essential elements: 1) Training the trainers; 2) Workplace-based assessments; 3)The use of an electronic portfolio.

The level of competency achieved by each participant will be determined by a simulation-based Objective Structured Clinical Exam (OSCE) performed at the end of the third year of traning (baseline) and at the end of the 5th year of training period.

Detailed Description

Competence Based Medical Education (CBME) stands for a shift in emphasis away from time-based programs, based solely on exposure to experiences such as clinical rotations, in favour of an emphasis on needs-based graduate outcomes authenticity, and learner-centeredness (1). The new paradigm defies the assumption that competence is achieved based on time spent on rotations and instead requires residents to demonstrate competence, what involves performing a significant number of formative assessments (2,3). The CBME model for Intensive Care Medicine (ICM) is called CoBaTrICE (Competency-Based Training in Intensive Care Medicine in Europe) (4), an international partnership of professional organisations and critical care clinicians whose ultimate aims are: to assure a high quality level education in ICM, to harmonize training in ICM without interfering with national specific regulations and to allow for free movement of ICM professionals across Europe (5). CoBaTrICE program used an international grounded approach to defining specialist competencies, and incorporated the views of 1,398 patients and relatives from 70 ICUs in eight EU countries (6). The final program includes 102 competence statements grouped into 12 domains and a definition of the level of expertise required of a safe practitioner at the end of their specialist training. CoBaTrICE has been formally adopted by 15 of 32 countries in Europe with a further 12 planning to do so, Spain is included in this latter group (7,8).

Importance:

CBME is gaining acceptance as a solution to address the challenges associated with the current time-based models of physician training. However, whether CBME programs provide better outcomes than the traditional ones is actually unknown.

Hypothesis:

The implementation of the CoBaTrICE will provide higher levels of competency in comparison with the current official time-based programme in ICM in Spain.

Objectives:

1. To determine the effect of CoBaTrICE implementation on:

1. The level of competence achieved by the trainees.

2. The percentage of critical performance elements.

2. To identify gaps in performance that could be addressed in future educational interventions.

3. To investigate the reliability and feasibility of conducting simulation based assessment at multiple sites.

4. To determine compliance with CoBaTrICE and barriers to an effective implementation.

Design:

A multicenter cluster randomized trial of 14 ICU Departments from 14 academic referral hospitals located in Spain. The participating ICUs are general medical and surgical ICUs accredited to train 2-3 new residents in ICM per year.

Participants:

A total of 38 trainees on the 3rd year of the specialization period (R3) will be followed during the three years of their specific training period in ICM (Stage 2). Enrollment will start on February 1, 2019, and the follow-up will end on May 31, 2021.

Intervention: see the specific item in the protocol section.

Main outcomes and measures (see also the specific item in the protocol section):

To determine the level of competency achieved for each participant, a simulation based OSCE (18-26) will be performed at the end of the 3rd year(baseline) and at the end of the 5th year of training in ICM. The OSCE will be performed simultaneously at four simulation centers (Hospital la Fe, Valencia; Francisco de Vitoria, Madrid; IAVANTE, Granada; and Hospital Clinic, Barcelona). Each participant will perform in five 15-min standardized patient or high-fidelity simulated clinical crisis scenarios.

Rating instruments and guide to rating:

Via a Delphi technique, an independent panel of 10 intensivists subject matter experts (simulation instructors and European Diplome in Intensive Care [EDIC] examiners) will perform the following tasks: 1) to select the competences to be assessed; 2) to design the scenarios; and 3) to define the items of the checklist for each scenario: a) the critical essential performance elements (CEPE), and b) the critical non-essential performance elements (CNEPE) that must be observed and scored in a yes/no format. CEPEs are defined as essential steps or actions in the management which if missed could have an immediate significant impact on morbidity and mortality. CNEPE are also important for the adequate management of the scenario but they do not have an immediate influence on the outcome. The performances will be video recorded (26). All the video recordings will be later rated by two blinded raters, members of the experts panel, using specific checklists which include 25 items with a detailed description of the CEPEs and CNEPEs, the competencies technical (diagnosis and treatment) and non-technical (communication, team leadership, resource management) associated with each item, as well as specific information about what is expected to be done by the trainee. After each video-assessment, the performance of the trainee will be classified in five levels of competency: Level I: The participant completed less than 60% of the CEPEs. The participant needs guidance and direct supervision to perform the activity in all cases. Level II: The participant completed ≥ 60% of the CEPEs but less than 80% of CEPEs. The participant needs guidance and supervision to perform the activity in most of situations. Level III: The participant completed ≥ 80% of the CEPEs but less than 100% of CEPEs. The participant needs some guidance and supervision to perform the activity in complex situations. Level IV: The participant completed 100% of the CEPEs but less than 80% of CNEPEs. The participant can perform the activity under indirect supervision. Level V: The participant completed 100% of the CEPEs and ≥ 80% of the CEPEs.The participant is independent to perform the activity. Finally, raters also will qualify the performance as: poor, pass, good, outstanding.

Measures will include: 1) the percentage of CEPEs observed; 2) the percentage of CNEPEs observed; 3) the overall competency level achieved on a descriptive scale of 1 to 5 (novice to independent practitioner); 4) the qualitative rating given by the raters based on whether the performance is at the level expected according to the predefined level for the year of training.

Standardization of Scenario Delivery (18-22). In order to facilitate reproducible scenario delivery, rules, detailed scripts and a guidebook for each scenario will be created. The scenario script will describe the elements of the simulated clinical environment (e.g., the equipment and medications available), evolution of the patient's condition throughout the crises and the responses to interventions, standardized answers to anticipated participant questions, and criteria that define successful completion of CPEs. Participants will be briefed on relevant mannequin characteristics, basic rules for participating in simulation scenarios, and location and uses of medications, clinical equipment, and other resources. After finishing the OSCE, resident feedback will be asked through a specific satisfaction survey.

Statistical Analysis:

Sample size: A power analysis has been performed in order to determine the minimum sample size required to detect with a power of .95 (α = .05, 1 - β = .95) a medium effect size (f = .30), requiring a minimum sample size of 90 observations (30 residents).

A mixed factorial ANOVA design will be applied on the time factor (R3-R4-R5) with the group factor (intervention vs. control) for the assessment of professional competences considered, applying post-hoc Bonferroni tests to analyze the principal and interaction effects between the two factors.

In order to estimate the effectiveness of CoBaTrICE, the differences between intervention and control group regarding the primary and secondary end points will be analyzed by a multivariate logistic regression analysis, the adjusted odds ratio will be estimated. Simulation experience defined as participation in at least two crisis resource management courses will be included in the analysis.

Development phases of the study:

The first phase of the project will begin by setting up an independent panel of 10 intensivists experts/executors that will perform the tasks mentioned above. After randomizing the hospitals to the experimental group, the tutors of the this group will be trained through a twelve-hour course integrating: 1) a detailed explanation of the principles, structure, and competencies contemplated in the CoBaTrICE program; 2) the basic principles of the formative assessment, techniques applied, and effective feedback; and 4) the use of the electronic portfolio to record the volume and results of the formative assessments and the progress of the trainee.

At the time of recruitment, residents and tutors of the participant ICUs will complete online several anonymous surveys to explore educational environment, engagement and satisfaction, and professional burnout.

The second phase of the study will consist of the CoBaTrICE program implementation and the comparison of the participating residents' levels of performance through the OSCE at specific points in time, which will be the end of R3 (baseline OSCE) and R5. It will determine: a) whether there are significant differences between the level of competencies shown by the residents depending on the type of training; b) whether there are significant differences in the pace of acquiring the competencies in the two types of programs; and c) the most common errors made by the residents through the different training levels and the possible differences between both programs. The tutors involved in the CoBaTrICE group will receive support to overcome barriers and problems found in the implementation of the program and/or in the use of the tools it incorporates. The tutors will be sent an "ad hoc" questionnaire designed to collect their suggestions. A pilot OSCE with local non-participants residents in the trial will be carried out at the simulation Center In Hospital La Fe, Valencia, Spain, in order to check the feasibility and reliability of the scenarios and the video rating process.The third phase of the study will be devoted to analyzing and publishing the results of the study. Interim analysis will be performed in order to determine the adequate implementation of the program and asses heterogeneity and/or possible bias selection.

Conditions

Competency-Based Education

Study Design

• Allocation Method: RANDOMIZED
• Intervention Model: PARALLEL
• Primary Study Purpose: OTHER
• Blinding Strategy: DOUBLE

Study Groups

CoBaTriCE

Implementation of CoBaTrICE. The implementation of CoBaTrICE is based on: 1. Training the trainers; 2. Multiple Workplace-based assessment exercices; 3. The use of an electronic portfolio.

Group Type: EXPERIMENTAL

Implementation of CoBaTrICE

Intervention Type: OTHER

The implemention of CoBaTrICE will include the three following essential elements:

1. Training the trainers. In order to provide high quality feedback to guide development of competence, tutors will receive a course in formative assessment, debriefing techniques, and effective feedback.

2. Workplace-based assessment (WbA) to promote learning and to guarantee that the predefined competences and skills are effectively acquired. Current Wb observation methods such as mini-clinical examination exercise, direct observation of procedural skills and multisource-feedback 360º will be used.

3. The use of an electronic portfolio specifically created as a personal collection of training documents that includes the record of formative assessments, activities, and the levels of competence achieved. The portfolio will help monitor progress of trainee's skills development and learning experiences contributing to an effective assessment, self-reflection and control of the learning process.

Control

The participants of the control group will follow the current official model of training in ICM in Spain, which is based on exposure to experiences through time-based clinical rotations; a generic report, non-based on formal assessment, about knowledge, technical and nontechnical skills is performed after every clinical rotation, and yearly by the tutor.

Group Type: NO_INTERVENTION

No interventions assigned to this group

Interventions

Implementation of CoBaTrICE

The implemention of CoBaTrICE will include the three following essential elements:

1. Training the trainers. In order to provide high quality feedback to guide development of competence, tutors will receive a course in formative assessment, debriefing techniques, and effective feedback.

2. Workplace-based assessment (WbA) to promote learning and to guarantee that the predefined competences and skills are effectively acquired. Current Wb observation methods such as mini-clinical examination exercise, direct observation of procedural skills and multisource-feedback 360º will be used.

3. The use of an electronic portfolio specifically created as a personal collection of training documents that includes the record of formative assessments, activities, and the levels of competence achieved. The portfolio will help monitor progress of trainee's skills development and learning experiences contributing to an effective assessment, self-reflection and control of the learning process.

Intervention Type: OTHER

Other Intervention Names

Implementation of a competency-based medical education program

Eligibility Criteria

Inclusion Criteria

• Trainees on the 3rd year of the specialization period in Intensive Care Medicine (ICM) who are developing their training program in a Spanish general medical and surgical ICU Department of an academic referral hospital accredited to train 2 or 3 new residents in ICM per year being 3 residents the highest number per year allowed in Spain.

Exclusion Criteria

• ICU Departments accredited to train less than 2 new residents in ICM per year
• ICU Departments where CoBaTrICE is being implemented. At present only one hospital in Spain (Hospital Universitario y Politécnico La Fe, Valencia) is formally implementing CoBaTrICE. As a consequence it has been excluded from the trial.

• Eligible Sex: ALL
• Accepts Healthy Volunteers: Yes

Sponsors

European Society of Intensive Care Medicine

OTHER

Sponsor Role: collaborator

Generalitat Valenciana

OTHER

Sponsor Role: collaborator

Alvaro Castellanos Ortega

OTHER

Sponsor Role: lead

Responsible Party

Alvaro Castellanos Ortega

PhD. Associated professor of the Universidad de Valencia. Director of the Intensive Care Department of the Hospital Universitario y Politécnico La Fe de Valencia

Responsibility Role: SPONSOR_INVESTIGATOR

Principal Investigators

ALVARO CASTELLANOS, PhD

Role: PRINCIPAL_INVESTIGATOR

Instituto de Investigación Sanitaria La Fe de Valencia (IIS La Fe)

RAFAEL GARCIA ROS, PhD

Role: STUDY_DIRECTOR

Cátedra de psicología de la Educación. Universidad de Valencia

Locations

Consorci Corporació Sanitária Parc Taulí

Sabadell, Barcelona, Spain

Hospital General Universitario, Alicante

Alicante, , Spain

Hospital Universitari Germans Trias I Pujol

Badalona, , Spain

Hospital Vall D´Hebron, Barcelona

Barcelona, , Spain

4. Hospital Universitario Virgen de Las Nieves de Granada

Granada, , Spain

Hospital Universitario de Gran Canaria Doctor Negrin

Las Palmas de Gran Canaria, , Spain

Hospital Clinico San Carlos

Madrid, , Spain

Hospital Universitario 12 de Octubre

Madrid, , Spain

Hospital Universitario La Paz, Madrid.

Madrid, , Spain

3. Hospital Clínico Universitario Virgen de La Arrixaca

Murcia, , Spain

Hospital Universitario Virgen de La Macarena de Sevilla

Seville, , Spain

Hospital Universitario Virgen Del Rocio

Seville, , Spain

Hospital Clínico Universitario de Valencia

Valencia, , Spain

7. Hospital Universitario Doctor Peset de Valencia

Valencia, , Spain

Countries

Spain

References

Carraccio C, Englander R, Van Melle E, Ten Cate O, Lockyer J, Chan MK, Frank JR, Snell LS; International Competency-Based Medical Education Collaborators. Advancing Competency-Based Medical Education: A Charter for Clinician-Educators. Acad Med. 2016 May;91(5):645-9. doi: 10.1097/ACM.0000000000001048.

Reference Type: BACKGROUND

PMID: 26675189 (View on PubMed)

Frank JR, Snell L, Englander R, Holmboe ES; ICBME Collaborators. Implementing competency-based medical education: Moving forward. Med Teach. 2017 Jun;39(6):568-573. doi: 10.1080/0142159X.2017.1315069.

Reference Type: BACKGROUND

PMID: 28598743 (View on PubMed)

Castellanos-Ortega A, Rothen HU, Franco N, Rayo LA, Martin-Loeches I, Ramirez P, Cunat de la Hoz J. Training in intensive care medicine. A challenge within reach. Med Intensiva. 2014 Jun-Jul;38(5):305-10. doi: 10.1016/j.medin.2013.12.011. Epub 2014 Mar 1. English, Spanish.

Reference Type: BACKGROUND

PMID: 24589154 (View on PubMed)

CoBaTrICE Collaboration; Bion JF, Barrett H. Development of core competencies for an international training programme in intensive care medicine. Intensive Care Med. 2006 Sep;32(9):1371-83. doi: 10.1007/s00134-006-0215-5. Epub 2006 Jul 14.

Reference Type: BACKGROUND

PMID: 16841214 (View on PubMed)

CoBaTrICE Collaboration. The views of patients and relatives of what makes a good intensivist: a European survey. Intensive Care Med. 2007 Nov;33(11):1913-20. doi: 10.1007/s00134-007-0799-4. Epub 2007 Aug 15.

Reference Type: BACKGROUND

PMID: 17701164 (View on PubMed)

Bion J, Rothen HU. Models for intensive care training. A European perspective. Am J Respir Crit Care Med. 2014 Feb 1;189(3):256-62. doi: 10.1164/rccm.201311-2058CP.

Reference Type: BACKGROUND

PMID: 24351076 (View on PubMed)

Gruppen L, Frank JR, Lockyer J, Ross S, Bould MD, Harris P, Bhanji F, Hodges BD, Snell L, Ten Cate O; ICBME Collaborators. Toward a research agenda for competency-based medical education. Med Teach. 2017 Jun;39(6):623-630. doi: 10.1080/0142159X.2017.1315065.

Reference Type: BACKGROUND

PMID: 28598741 (View on PubMed)

Junta Directiva de la Sociedad Espanola de Medicina Intensiva, Critica y Unidades Coronarias (SEMICYUC). [Intensive medicine in Spain]. Med Intensiva. 2011 Mar;35(2):92-101. doi: 10.1016/j.medin.2010.12.008. Epub 2011 Mar 2. Spanish.

Reference Type: BACKGROUND

PMID: 21371787 (View on PubMed)

Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR; ICBME Collaborators. Core principles of assessment in competency-based medical education. Med Teach. 2017 Jun;39(6):609-616. doi: 10.1080/0142159X.2017.1315082.

Reference Type: BACKGROUND

PMID: 28598746 (View on PubMed)

Ilgen JS, Ma IW, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015 Feb;49(2):161-73. doi: 10.1111/medu.12621.

Reference Type: BACKGROUND

PMID: 25626747 (View on PubMed)

Everett TC, Ng E, Power D, Marsh C, Tolchard S, Shadrina A, Bould MD. The Managing Emergencies in Paediatric Anaesthesia global rating scale is a reliable tool for simulation-based assessment in pediatric anesthesia crisis management. Paediatr Anaesth. 2013 Dec;23(12):1117-23. doi: 10.1111/pan.12212. Epub 2013 Jun 26.

Reference Type: BACKGROUND

PMID: 23800112 (View on PubMed)

Heeneman S, Driessen EW. The use of a portfolio in postgraduate medical education - reflect, assess and account, one for each or all in one? GMS J Med Educ. 2017 Nov 15;34(5):Doc57. doi: 10.3205/zma001134. eCollection 2017.

Reference Type: BACKGROUND

PMID: 29226225 (View on PubMed)

Dougherty P, Ross PT, Lypson ML. Monitoring resident progress through mentored portfolios. J Grad Med Educ. 2013 Dec;5(4):701-2. doi: 10.4300/JGME-D-13-00309.1. No abstract available.

Reference Type: BACKGROUND

PMID: 24455028 (View on PubMed)

Hastie MJ, Spellman JL, Pagano PP, Hastie J, Egan BJ. Designing and implementing the objective structured clinical examination in anesthesiology. Anesthesiology. 2014 Jan;120(1):196-203. doi: 10.1097/ALN.0000000000000068.

Reference Type: BACKGROUND

PMID: 24212197 (View on PubMed)

Chiu M, Tarshis J, Antoniou A, Bosma TL, Burjorjee JE, Cowie N, Crooks S, Doyle K, Dubois D, Everett T, Fisher R, Hayter M, McKinnon G, Noseworthy D, O'Regan N, Peachey G, Robitaille A, Sullivan M, Tenenbein M, Tremblay MH. Simulation-based assessment of anesthesiology residents' competence: development and implementation of the Canadian National Anesthesiology Simulation Curriculum (CanNASC). Can J Anaesth. 2016 Dec;63(12):1357-1363. doi: 10.1007/s12630-016-0733-8. Epub 2016 Sep 16.

Reference Type: BACKGROUND

PMID: 27638297 (View on PubMed)

Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J, Cooper JB, Steadman R, Shotwell MS, Slagle JM, DeMaria S Jr, Torsher L, Sinz E, Levine AI, Rask J, Davis F, Park C, Gaba DM. Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists. Anesthesiology. 2017 Sep;127(3):475-489. doi: 10.1097/ALN.0000000000001739.

Reference Type: BACKGROUND

PMID: 28671903 (View on PubMed)

Lee V, Brain K, Martin J. Factors Influencing Mini-CEX Rater Judgments and Their Practical Implications: A Systematic Literature Review. Acad Med. 2017 Jun;92(6):880-887. doi: 10.1097/ACM.0000000000001537.

Reference Type: BACKGROUND

PMID: 28030422 (View on PubMed)

McIvor WR, Banerjee A, Boulet JR, Bekhuis T, Tseytlin E, Torsher L, DeMaria S Jr, Rask JP, Shotwell MS, Burden A, Cooper JB, Gaba DM, Levine A, Park C, Sinz E, Steadman RH, Weinger MB. A Taxonomy of Delivery and Documentation Deviations During Delivery of High-Fidelity Simulations. Simul Healthc. 2017 Feb;12(1):1-8. doi: 10.1097/SIH.0000000000000184.

Reference Type: BACKGROUND

PMID: 28146449 (View on PubMed)

Blum RH, Boulet JR, Cooper JB, Muret-Wagstaff SL; Harvard Assessment of Anesthesia Resident Performance Research Group. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology. 2014 Jan;120(1):129-41. doi: 10.1097/ALN.0000000000000055.

Reference Type: BACKGROUND

PMID: 24398731 (View on PubMed)

Fung L, Boet S, Bould MD, Qosa H, Perrier L, Tricco A, Tavares W, Reeves S. Impact of crisis resource management simulation-based training for interprofessional and interdisciplinary teams: A systematic review. J Interprof Care. 2015;29(5):433-44. doi: 10.3109/13561820.2015.1017555. Epub 2015 May 14.

Reference Type: BACKGROUND

PMID: 25973615 (View on PubMed)

Khanduja PK, Bould MD, Naik VN, Hladkowicz E, Boet S. The role of simulation in continuing medical education for acute care physicians: a systematic review. Crit Care Med. 2015 Jan;43(1):186-93. doi: 10.1097/CCM.0000000000000672.

Reference Type: BACKGROUND

PMID: 25343571 (View on PubMed)

Easdown LJ, Wakefield ML, Shotwell MS, Sandison MR. A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE. J Grad Med Educ. 2017 Oct;9(5):605-610. doi: 10.4300/JGME-D-17-00112.1.

Reference Type: BACKGROUND

PMID: 29075381 (View on PubMed)

Seam N, Lee AJ, Vennero M, Emlet L. Simulation Training in the ICU. Chest. 2019 Dec;156(6):1223-1233. doi: 10.1016/j.chest.2019.07.011. Epub 2019 Jul 30.

Reference Type: BACKGROUND

PMID: 31374210 (View on PubMed)

Castellanos-Ortega A, Broch MJ, Palacios-Castaneda D, Gomez-Tello V, Valdivia M, Vicent C, Madrid I, Martinez N, Parraga MJ, Sancho E, Fuentes-Dura MDC, Sancerni-Beitia MD, Garcia-Ros R. Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study. Med Intensiva (Engl Ed). 2022 Sep;46(9):491-500. doi: 10.1016/j.medine.2022.01.001.

Reference Type: DERIVED

PMID: 36057440 (View on PubMed)

Other Identifiers

COBALIDATION

Identifier Type: -

Identifier Source: org_study_id