AI-Based Fidelity Feedback to Enhance CBT

NCT ID: NCT05340738

Last Updated: 2025-06-29

Basic Information

• Recruitment Status: RECRUITING
• Clinical Phase: NA
• Total Enrollment: 425 participants
• Study Classification: INTERVENTIONAL
• Study Start Date: 2023-03-09
• Study Completion Date: 2026-01-31

Brief Summary

This study is being conducted together by researchers at the University of Pennsylvania and Lyssn.io, Inc., ("Lyssn"), a technology start-up developing digital tools to support evidence-based psychotherapies (EBPs) for mental health disorders and addiction. This study will implement a technology to assess and enhance the quality of EBPs like Cognitive Behavioral Therapy (CBT) that includes a user interface geared to clinical, supervision, and administrative workflows and needs, and then assess this technology for effectiveness in comparison to usual care.

There is a tremendous global burden of mental illness: Over 50 million American adults have a diagnosable mental health disorder, and major depression on its own is the leading cause of disability worldwide. In the face of this burden, clinical research has documented a variety of effective EBPs (e.g. CBT), and these psychotherapies are utilized on a massive scale. Systems have invested over $2 billion in training providers in specific EBPs. Once trained, however, therapists' adherence to the EBP, also called fidelity, is both crucial for effectiveness and difficult to assess. There is no scalable method to assess the fidelity and quality of EBPs in community practice settings. This is a foundational problem for healthcare systems.

Advances in speech processing and machine learning make technology a promising solution to this problem. The use of technology - instead of humans - to evaluate EBPs means that objective, performance-based feedback can be provided quickly, efficiently, cost-effectively, and without human error. If successful, the present research will be among the first examples of a method for building, monitoring, and assessing the quality of therapy that can scale up to large, real-world healthcare settings.

In this study, the investigators will implement an existing, fully-functional prototype (LyssnCBT) that includes a user interface geared to community mental health (CMH) clinical, supervision, and administrative workflows and needs, and then assess for effectiveness of psychotherapy supported by LyssnCBT in comparison to usual care.

This study will implement LyssnCBT in 5 community mental health agencies, beginning with a single-arm pilot field trial to identify and address any specific barriers to implementing the tool in a community mental health context. The study team will then conduct a larger study in community mental health agencies comparing LyssnCBT to services as usual.

Detailed Description

The research team will first recruit 10 therapists and two supervisors from one CMH clinic in order to conduct a single-arm field trial of LyssnCBT. Each therapist will use the LyssnCBT platform with two clients over the course of two weeks (four total sessions per therapist). In addition, supervisors will conduct a supervision session with each therapist using the LyssnCBT tool's fidelity feedback. Next, participants will complete brief Likert assessments on technical reliability, functional reliability, and experiences integrating the system into the daily workflow. Usage data from LyssnCBT will be captured automatically by the system, including: which software features were used, time spent reviewing sessions and transcripts, and time spent reviewing artificial intelligence (AI) generated CBT fidelity feedback. This data and feedback will be used for a final refinement of the LyssnCBT software and related clinical and supervision protocols prior to the main trial.

Following the pilot study, the research team will recruit 4 additional CMH clinics to participate in a type 2 hybrid implementation-effectiveness, randomized stepped-wedge study comparing LyssnCBT to SAU. 50 therapists and their supervisors will be recruited from the participating clinics, and each therapist will be asked to have at least 5 clients participating in the study at any given time, from among their regular caseload. Across 18 months of planned data collection (~75 weeks), the investigators expect a minimum of 1,875 clients for 50 therapists (i.e., 50 therapists x 5 sessions per week x 75 weeks = 18,750 sessions, with an average of 10 sessions per client). All 5 clinics will start with SAU, and clinics will be randomized to begin LyssnCBT sequentially over time.

The primary data being collected throughout the project are recordings of therapy sessions, which are also collected as part of the typical operating procedures of the Penn Beck Community Initiative (BCI). CBT fidelity will be assessed by AI-generated Cognitive Therapy Rating Scale (CTRS) scores for every recorded therapy session, which will be recorded via the Lyssn platform during both SAU and LyssnCBT phases of the study. For client outcomes, the PHQ-9 and GAD-7 will be collected at each session and client drop-out will be assessed via a brief monthly survey sent out to participating therapists. Finally, after three months of engagement with the LyssnCBT tools, each participating therapist and supervisor will complete our battery of implementation measures, including the system usability scale, AIM, IAM, and FIM.

Conditions

Cognitive Behavioral Therapy; Therapy

Study Design

• Allocation Method: RANDOMIZED
• Intervention Model: CROSSOVER
• Primary Study Purpose: HEALTH_SERVICES_RESEARCH
• Blinding Strategy: NONE

Study Groups

LyssnCBT

Therapists will use the LyssnCBT tool with clients for recording and session-sharing functionalities. Therapists and supervisors will also have access to LyssnCBT features like speech-to-text transcription, annotation tools, and AI-generated metrics.

Group Type: ACTIVE_COMPARATOR

LyssnCBT

Intervention Type: OTHER

LyssnCBT is a technology that allows therapists and supervisors access to tools that assist with assessing CBT session fidelity, including speech-to-text transcription, annotation tools, and AI-generated metrics.

SAU (services-as-usual)

Therapists will use the LyssnCBT tool with clients for recording and session-sharing functionalities. No other LyssnCBT features will be available for therapist or supervisor review.

Group Type: NO_INTERVENTION

No interventions assigned to this group

Interventions

LyssnCBT

LyssnCBT is a technology that allows therapists and supervisors access to tools that assist with assessing CBT session fidelity, including speech-to-text transcription, annotation tools, and AI-generated metrics.

Intervention Type: OTHER

Eligibility Criteria

Inclusion Criteria

Therapists

• Able to participate in therapy sessions conducted in English
• Employed at a Philadelphia CMH treatment center that allows the recruitment and participation of therapists in research-related activities
• Willing to allow their session recordings to be used for research purposes
• Computer and internet access

Supervisors

• Oversee participating therapists
• Computer and internet access

Clients

• Able to participate in therapy sessions conducted in English
• Willing to allow the team to collect data and use their session recordings for research purposes

Exclusion Criteria

• Unwilling to allow the research team to access their therapy session recordings

• Minimum Eligible Age: 18 Years
• Eligible Sex: ALL
• Accepts Healthy Volunteers: Yes

Sponsors

Lyssn.io, Inc.

INDUSTRY

Sponsor Role: collaborator

National Institute of Mental Health (NIMH)

NIH

Sponsor Role: collaborator

University of Pennsylvania

OTHER

Sponsor Role: lead

Responsible Party

Torrey Creed

Assistant Professor of Psychology in Psychiatry

Responsibility Role: PRINCIPAL_INVESTIGATOR

Principal Investigators

Torrey A Creed, PhD

Role: PRINCIPAL_INVESTIGATOR

Director, The Penn Collaborative for CBT and Implementation Science

David Atkins, PhD

Role: PRINCIPAL_INVESTIGATOR

CEO, Lyssn

Locations

The Penn Collaborative for CBT and Implementation Science

Philadelphia, Pennsylvania, United States

Site Status: RECRUITING

Countries

United States

Central Contacts

Torrey A Creed, PhD

Role: CONTACT

tcreed@pennmedicine.upenn.edu

215-573-6773

Ana DeCesare, BA

Role: CONTACT

Ana.DeCesare@Pennmedicine.upenn.edu

850-501-3734

Facility Contacts

Torrey A Creed, PhD

Role: primary

tcreed@pennmedicine.upenn.edu

215-573-6773

References

Creed TA, Salama L, Slevin R, Tanana M, Imel Z, Narayanan S, Atkins DC. Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol. BMC Health Serv Res. 2022 Sep 20;22(1):1177. doi: 10.1186/s12913-022-08519-9.

Reference Type: DERIVED

PMID: 36127689 (View on PubMed)

Other Identifiers

R42MH128101

Identifier Type: NIH

Identifier Source: secondary_id

View Link

R42MH128101

Identifier Type: NIH

Identifier Source: org_study_id

View Link