Trial Outcomes & Findings for Project FACTS (Fidelity Accuracy: Comparing Three Strategies) (NCT NCT02820623)

NCT ID: NCT02820623

Last Updated: 2023-04-21

Results Overview

Client sessions for all 3 conditions were scored for CBT adherence via direct observation using the Therapy Process Observational Coding System-Revised Strategies (TPOCS-RS) Scale (McLeod et al., 2015) on a 7-point Likert Scale (1='not at all,' 7='extensively'). We analyzed the TPOCS-RS' Maximum CBT score (highest coded intervention technique across all 12 possible interventions in a given session). We also measured CBT adherence during client sessions using the strategies from each study condition (described in 'Arms/Groups' below). All 3 conditions used parallel 7-point scales to the TPOCS-RS. Higher scores for the 3 study condition strategies indicate greater reported use, and higher scores on the TPOCS-RS indicate greater observed use of CBT interventions. Our goal was to compare the study condition methods of assessing CBT adherence to direct observation (TPOCS-RS), the gold standard.

Recruitment status

COMPLETED

Study phase

Target enrollment

126 participants

Primary outcome timeframe

through study completion, an average of 1 month

Results posted on

2023-04-21

Participant Flow

We enrolled 126 therapists in the study; these participants consented and were randomized to treatment. We assessed 304 therapists for enrollment in the study; of the therapists who were not enrolled, 86 did not meet inclusion criteria, 84 declined to participate, and 8 were excluded for other reasons.

Participant milestones

Participant milestones
Measure	Chart Stimulated Recall Therapists randomized to this condition will be asked to bring the charts of three enrolled youth to the chart-stimulated recall interview. A trained interviewer will ask the therapists how well they recall the encounter (rating of memory quality) followed by an open-ended question ("Talk me through your last session with your client. Tell me what you did."). While the therapists are speaking, the interviewer will note any elements that represent a prescribed cognitive behavioral therapy (CBT) strategy. The interviewer will go through a list of cognitive-behavioral strategies based upon the TPOCS-S and probe to determine if the therapists completed any of the strategies. Follow-up questions will be used to explore to what degree an element was used and how skillfully and responsively the strategies were used. Chart-Stimulated Recall	Behavioral Rehearsal Therapists randomized to this condition will be asked to engage in role-plays demonstrating the CBT strategies used with the three enrolled youth. The investigators will provide therapists with a list of the TPOCS-S CBT strategies and ask them to identify the CBT strategies used in their recorded encounter. The investigators will randomly select one of the strategies they report for each role-play. The investigators will then tell them, "Please role-play how you used this strategy in session with your client, with the trained actor in front of you." Later, an independent rater will rate therapists' adherence and skill based on established scoring criteria. Behavioral Rehearsal	Self-report Therapists randomized to this condition will complete a brief self-report measure, the Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale-Self Report version (TPOCS-SR) for each of the recorded clinical encounters with enrolled youth. The TPOCS-SR will be a self-report version of the Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale (TPOCS-S) and will be created in collaboration with the instrument developer (McLeod). In this condition, the investigators will (a) provide an operational definition for each item on the TPOCS-SR (e.g., cognitive education: teaches client the cognitive model (e.g., thoughts influence behavior)/identifies how the cognitive model applies to a specific aspects of the client's life), and (b) provide therapists with a 30-minute training session that includes sample vignettes of particular behaviors and information about how those vignettes should be rated. Self-report
Overall Study STARTED	42	43	41
Overall Study COMPLETED	35	32	36
Overall Study NOT COMPLETED	7	11	5

Reasons for withdrawal

Reasons for withdrawal
Measure	Chart Stimulated Recall Therapists randomized to this condition will be asked to bring the charts of three enrolled youth to the chart-stimulated recall interview. A trained interviewer will ask the therapists how well they recall the encounter (rating of memory quality) followed by an open-ended question ("Talk me through your last session with your client. Tell me what you did."). While the therapists are speaking, the interviewer will note any elements that represent a prescribed cognitive behavioral therapy (CBT) strategy. The interviewer will go through a list of cognitive-behavioral strategies based upon the TPOCS-S and probe to determine if the therapists completed any of the strategies. Follow-up questions will be used to explore to what degree an element was used and how skillfully and responsively the strategies were used. Chart-Stimulated Recall	Behavioral Rehearsal Therapists randomized to this condition will be asked to engage in role-plays demonstrating the CBT strategies used with the three enrolled youth. The investigators will provide therapists with a list of the TPOCS-S CBT strategies and ask them to identify the CBT strategies used in their recorded encounter. The investigators will randomly select one of the strategies they report for each role-play. The investigators will then tell them, "Please role-play how you used this strategy in session with your client, with the trained actor in front of you." Later, an independent rater will rate therapists' adherence and skill based on established scoring criteria. Behavioral Rehearsal	Self-report Therapists randomized to this condition will complete a brief self-report measure, the Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale-Self Report version (TPOCS-SR) for each of the recorded clinical encounters with enrolled youth. The TPOCS-SR will be a self-report version of the Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale (TPOCS-S) and will be created in collaboration with the instrument developer (McLeod). In this condition, the investigators will (a) provide an operational definition for each item on the TPOCS-SR (e.g., cognitive education: teaches client the cognitive model (e.g., thoughts influence behavior)/identifies how the cognitive model applies to a specific aspects of the client's life), and (b) provide therapists with a 30-minute training session that includes sample vignettes of particular behaviors and information about how those vignettes should be rated. Self-report
Overall Study Withdrawal by Subject	3	3	1
Overall Study Participation interrupted due to COVID-19	3	2	3
Overall Study Therapist unable to enroll clients	1	3	1
Overall Study Clients declined, piloting condition, or only telehealth	0	3	0

Baseline Characteristics

Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their age.

Baseline characteristics by cohort

Baseline characteristics by cohort
Measure	Self-report n=41 Participants Therapists randomized to this condition will complete a brief self-report measure, the Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale-Self Report version (TPOCS-SR) for each of the recorded clinical encounters with enrolled youth. The TPOCS-SR will be a self-report version of the Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale (TPOCS-S) and will be created in collaboration with the instrument developer (McLeod). In this condition, the investigators will (a) provide an operational definition for each item on the TPOCS-SR (e.g., cognitive education: teaches client the cognitive model (e.g., thoughts influence behavior)/identifies how the cognitive model applies to a specific aspects of the client's life), and (b) provide therapists with a 30-minute training session that includes sample vignettes of particular behaviors and information about how those vignettes should be rated. Self-report	Chart Stimulated Recall n=42 Participants Therapists randomized to this condition will be asked to bring the charts of three enrolled youth to the chart-stimulated recall interview. A trained interviewer will ask the therapists how well they recall the encounter (rating of memory quality) followed by an open-ended question ("Talk me through your last session with your client. Tell me what you did."). While the therapists are speaking, the interviewer will note any elements that represent a prescribed CBT strategy. The interviewer will go through a list of cognitive-behavioral strategies based upon the TPOCS-S and probe to determine if the therapists completed any of the strategies. Follow-up questions will be used to explore to what degree an element was used and how skillfully and responsively the strategies were used. Chart-Stimulated Recall	Behavioral Rehearsal n=43 Participants Therapists randomized to this condition will be asked to engage in role-plays demonstrating the CBT strategies used with the three enrolled youth. The investigators will provide therapists with a list of the TPOCS-S CBT strategies and ask them to identify the CBT strategies used in their recorded encounter. The investigators will randomly select one of the strategies they report for each role-play. The investigators will then tell them, "Please role-play how you used this strategy in session with your client, with the trained actor in front of you." Later, an independent rater will rate therapists' adherence and skill based on established scoring criteria. Behavioral Rehearsal	Total n=126 Participants Total of all reporting groups
Age, Continuous	39.93 years STANDARD_DEVIATION 13.81 • n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their age.	37.42 years STANDARD_DEVIATION 12.13 • n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their age.	34.76 years STANDARD_DEVIATION 11.76 • n=42 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their age.	37.33 years STANDARD_DEVIATION 12.66 • n=122 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their age.
Sex: Female, Male Female	25 Participants n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	32 Participants n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	37 Participants n=42 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	94 Participants n=122 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.
Sex: Female, Male Male	15 Participants n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	8 Participants n=40 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	5 Participants n=42 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.	28 Participants n=122 Participants • Number of participants analyzed for each group does not match overall number of baseline participants in that group because some participants did not disclose their sex.
Ethnicity (NIH/OMB) Hispanic or Latino	3 Participants n=41 Participants	3 Participants n=42 Participants	6 Participants n=43 Participants	12 Participants n=126 Participants
Ethnicity (NIH/OMB) Not Hispanic or Latino	36 Participants n=41 Participants	37 Participants n=42 Participants	37 Participants n=43 Participants	110 Participants n=126 Participants
Ethnicity (NIH/OMB) Unknown or Not Reported	2 Participants n=41 Participants	2 Participants n=42 Participants	0 Participants n=43 Participants	4 Participants n=126 Participants
Race (NIH/OMB) American Indian or Alaska Native	0 Participants n=41 Participants	0 Participants n=42 Participants	0 Participants n=43 Participants	0 Participants n=126 Participants
Race (NIH/OMB) Asian	0 Participants n=41 Participants	2 Participants n=42 Participants	3 Participants n=43 Participants	5 Participants n=126 Participants
Race (NIH/OMB) Native Hawaiian or Other Pacific Islander	0 Participants n=41 Participants	0 Participants n=42 Participants	0 Participants n=43 Participants	0 Participants n=126 Participants
Race (NIH/OMB) Black or African American	11 Participants n=41 Participants	5 Participants n=42 Participants	6 Participants n=43 Participants	22 Participants n=126 Participants
Race (NIH/OMB) White	23 Participants n=41 Participants	32 Participants n=42 Participants	29 Participants n=43 Participants	84 Participants n=126 Participants
Race (NIH/OMB) More than one race	1 Participants n=41 Participants	0 Participants n=42 Participants	2 Participants n=43 Participants	3 Participants n=126 Participants
Race (NIH/OMB) Unknown or Not Reported	6 Participants n=41 Participants	3 Participants n=42 Participants	3 Participants n=43 Participants	12 Participants n=126 Participants
Region of Enrollment United States	41 participants n=41 Participants	42 participants n=42 Participants	43 participants n=43 Participants	103 participants n=126 Participants
Education Bachelor's Degree	0 Participants n=41 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	2 Participants n=40 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	2 Participants n=43 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	4 Participants n=124 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.
Education Master's Degree	40 Participants n=41 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	37 Participants n=40 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	39 Participants n=43 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	116 Participants n=124 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.
Education Doctorate	1 Participants n=41 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	1 Participants n=40 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	2 Participants n=43 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.	4 Participants n=124 Participants • Number of participants analyzed for each group (for some group\[s\[) does not match overall number of baseline participants in that group (for some group\[s\]) because some participants did not disclose their level of education.
Years Experience	11.44 years STANDARD_DEVIATION 10.57 • n=41 Participants • Number of participants analyzed for one or more group(s) does not match overall number of baseline participants in that group(s) because some participants did not disclose their number of years' experience.	10.55 years STANDARD_DEVIATION 10.29 • n=40 Participants • Number of participants analyzed for one or more group(s) does not match overall number of baseline participants in that group(s) because some participants did not disclose their number of years' experience.	8.12 years STANDARD_DEVIATION 9.76 • n=43 Participants • Number of participants analyzed for one or more group(s) does not match overall number of baseline participants in that group(s) because some participants did not disclose their number of years' experience.	10.00 years STANDARD_DEVIATION 10.22 • n=124 Participants • Number of participants analyzed for one or more group(s) does not match overall number of baseline participants in that group(s) because some participants did not disclose their number of years' experience.
Type of Employee Salaried	22 Participants n=41 Participants	20 Participants n=42 Participants	22 Participants n=43 Participants	64 Participants n=126 Participants
Type of Employee Fee for service	18 Participants n=41 Participants	18 Participants n=42 Participants	18 Participants n=43 Participants	54 Participants n=126 Participants
Type of Employee No response	1 Participants n=41 Participants	4 Participants n=42 Participants	3 Participants n=43 Participants	8 Participants n=126 Participants
Participated in city sponsored Evidence-Based Practice training initiative	29 Participants n=41 Participants	25 Participants n=42 Participants	28 Participants n=43 Participants	82 Participants n=126 Participants
Number of client sessions recorded in the study	2.44 sessions STANDARD_DEVIATION 1.05 • n=41 Participants	2.29 sessions STANDARD_DEVIATION 1.17 • n=42 Participants	2.14 sessions STANDARD_DEVIATION 1.32 • n=43 Participants	2.29 sessions STANDARD_DEVIATION 1.19 • n=126 Participants

PRIMARY outcome

Timeframe: through study completion, an average of 1 month

Population: Participants (i.e., therapists) were assigned to one of 3 conditions: self-report, chart stimulated recall, or behavioral rehearsal. The unit of analysis for Maximum Mean condition score and Maximum Mean direct observation (TPOCS-RS) score was client session; therapists could include more than one client session for inclusion in analyses.

Outcome measures

Outcome measures
Measure	Self-report n=100 Client Sessions Therapists randomized to this condition will complete a brief self-report measure, the Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale-Self Report version (TPOCS-SR) for each of the recorded clinical encounters with enrolled youth. The TPOCS-SR will be a self-report version of the Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale (TPOCS-S) and will be created in collaboration with the instrument developer (McLeod). In this condition, the investigators will (a) provide an operational definition for each item on the TPOCS-SR (e.g., cognitive education: teaches client the cognitive model (e.g., thoughts influence behavior)/identifies how the cognitive model applies to a specific aspects of the client's life), and (b) provide therapists with a 30-minute training session that includes sample vignettes of particular behaviors and information about how those vignettes should be rated. Self-report	Chart Stimulated Recall n=96 Client Sessions Therapists randomized to this condition will be asked to bring the charts of three enrolled youth to the chart-stimulated recall interview. A trained interviewer will ask the therapists how well they recall the encounter (rating of memory quality) followed by an open-ended question ("Talk me through your last session with your client. Tell me what you did."). While the therapists are speaking, the interviewer will note any elements that represent a prescribed CBT strategy. The interviewer will go through a list of cognitive-behavioral strategies based upon the TPOCS-S and probe to determine if the therapists completed any of the strategies. Follow-up questions will be used to explore to what degree an element was used and how skillfully and responsively the strategies were used. Chart-Stimulated Recall	Behavioral Rehearsal n=92 Client Sessions Therapists randomized to this condition will be asked to engage in role-plays demonstrating the CBT strategies used with the three enrolled youth. The investigators will provide therapists with a list of the TPOCS-S CBT strategies and ask them to identify the CBT strategies used in their recorded encounter. The investigators will randomly select one of the strategies they report for each role-play. The investigators will then tell them, "Please role-play how you used this strategy in session with your client, with the trained actor in front of you." Later, an independent rater will rate therapists' adherence and skill based on established scoring criteria. Behavioral Rehearsal
Total Fidelity Scores for Each Study Condition Measured Via Direct Observation (TPOCS-RS) and Study Arm Fidelity Method Scores Maximum Mean Condition Score	5.28 Maximum CBT score on a scale Standard Deviation 1.17	4.53 Maximum CBT score on a scale Standard Deviation 1.18	4.09 Maximum CBT score on a scale Standard Deviation 1.48
Total Fidelity Scores for Each Study Condition Measured Via Direct Observation (TPOCS-RS) and Study Arm Fidelity Method Scores Maximum Mean direct observation (TPOCS-RS) Score	3.42 Maximum CBT score on a scale Standard Deviation 1.55	3.69 Maximum CBT score on a scale Standard Deviation 1.36	3.72 Maximum CBT score on a scale Standard Deviation 1.59

Adverse Events

Self-report

Serious events: 0 serious events

Other events: 0 other events

Deaths: 0 deaths

Chart Stimulated Recall

Serious events: 0 serious events

Other events: 0 other events

Deaths: 0 deaths

Behavioral Rehearsal

Serious events: 0 serious events

Other events: 0 other events

Deaths: 0 deaths

Serious adverse events

Adverse event data not reported

Other adverse events

Adverse event data not reported

Additional Information

Rinad Beidas, PhD

Northwestern University

Phone: 312-503-0546

Email: [email protected]

Results disclosure agreements

Principal investigator is a sponsor employee
Publication restrictions are in place