Trial Outcomes & Findings for Behavioral Economics to Improve Flu Vaccination Using EHR Nudges (NCT NCT06057727)

NCT ID: NCT06057727

Last Updated: 2025-04-27

Results Overview

The primary outcome is flu vaccination completion during the first eligible primary care visit.

• Recruitment status: COMPLETED
• Study phase: NA
• Target enrollment: 80039 participants
• Primary outcome timeframe: 4 days from enrollment, at the eligible visit
• Results posted on: 2025-04-27

Participant Flow

Waiver of informed consent obtained. Patients were identified via Epic Clarity Query 4 days prior to their eligible primary care visit between September 25, 2023 - February 20, 2024. Patients were only counted once within the study window at their first primary care visit. Total protocol enrollment: 80,039 patients.

Clinics were randomized 2:1 to intervention and control. Patients seen at an intervention clinic and identified as high risk were further randomized 1:1 to standard messaging or an intensification nudge. The high risk individual randomization is nested within the intervention arm resulting in an overlap of patients where high risk individuals (41,909 of the 52,526) contribute both to estimation of the overall intervention effect and comparison of text messaging intensity among high risk patients

Unit of analysis: Clinics

Participant milestones

Measure	Control Patients in clinics randomized to the control arm will receive standard of care.	Intervention Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders.	High Risk - Standard Messaging Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk standard messaging arm received the same standard messaging as average risk patients.	High Risk - Intensification Messaging Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk intensification group received an additional bidirectional texting component in addition to the standard messaging.	High Risk - Not Individually Randomized Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine who were not individually randomized as a result of an automation issue. These patients still received the clinician nudges (default pended orders and peer comparison feedback) but did not receive any standard or intensification messaging.
Clinic Level Randomization STARTED	27513 16	52526 32	0 0	0 0	0 0
Clinic Level Randomization COMPLETED	27513 16	52526 31	0 0	0 0	0 0
Clinic Level Randomization NOT COMPLETED	0 0	0 1	0 0	0 0	0 0
Individual Level Randomization STARTED	0 0	0 0	20440 0	20412 0	1057 0
Individual Level Randomization COMPLETED	0 0	0 0	20440 0	20412 0	1057 0
Individual Level Randomization NOT COMPLETED	0 0	0 0	0 0	0 0	0 0

Reasons for withdrawal

Withdrawal data not reported

Baseline Characteristics

Behavioral Economics to Improve Flu Vaccination Using EHR Nudges

Baseline characteristics by cohort

Measure	Control n=27513 Participants Patients in clinics randomized to the control arm will receive standard of care.	Intervention n=52526 Participants Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders.	Total n=80039 Participants Total of all reporting groups
Age, Continuous	65.1 years STANDARD_DEVIATION 10.0 • n=5 Participants	66.1 years STANDARD_DEVIATION 10.3 • n=7 Participants	65.8 years STANDARD_DEVIATION 10.2 • n=5 Participants
Sex/Gender, Customized Male	12128 Participants n=5 Participants	22747 Participants n=7 Participants	34875 Participants n=5 Participants
Sex/Gender, Customized Female	15162 Participants n=5 Participants	29640 Participants n=7 Participants	44802 Participants n=5 Participants
Sex/Gender, Customized Unknown	223 Participants n=5 Participants	139 Participants n=7 Participants	362 Participants n=5 Participants
Race (NIH/OMB) American Indian or Alaska Native	138 Participants n=5 Participants	266 Participants n=7 Participants	404 Participants n=5 Participants
Race (NIH/OMB) Asian	2167 Participants n=5 Participants	3412 Participants n=7 Participants	5579 Participants n=5 Participants
Race (NIH/OMB) Native Hawaiian or Other Pacific Islander	135 Participants n=5 Participants	144 Participants n=7 Participants	279 Participants n=5 Participants
Race (NIH/OMB) Black or African American	6767 Participants n=5 Participants	11928 Participants n=7 Participants	18695 Participants n=5 Participants
Race (NIH/OMB) White	16534 Participants n=5 Participants	33638 Participants n=7 Participants	50172 Participants n=5 Participants
Race (NIH/OMB) More than one race	0 Participants n=5 Participants	0 Participants n=7 Participants	0 Participants n=5 Participants
Race (NIH/OMB) Unknown or Not Reported	1772 Participants n=5 Participants	3138 Participants n=7 Participants	4910 Participants n=5 Participants
Race/Ethnicity, Customized Ethnicity · Not Hispanic or Latino/ Unknown	26161 Participants n=5 Participants	50733 Participants n=7 Participants	76894 Participants n=5 Participants
Race/Ethnicity, Customized Ethnicity · Hispanic or Latino	1352 Participants n=5 Participants	1793 Participants n=7 Participants	3145 Participants n=5 Participants
Lowest Quartile Income by Zip Code Above Lowest Income Quartile by Zip Code	19406 Participants n=5 Participants	37064 Participants n=7 Participants	56470 Participants n=5 Participants
Lowest Quartile Income by Zip Code Lowest Income Quartile by Zip Code	8107 Participants n=5 Participants	15462 Participants n=7 Participants	23569 Participants n=5 Participants

PRIMARY outcome

Timeframe: 4 days from enrollment, at the eligible visit

The primary outcome is flu vaccination completion during the first eligible primary care visit.

Outcome measures

Measure	Control n=27513 Participants Patients in clinics randomized to the control arm will receive standard of care.	Intervention n=52526 Participants Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders.	High Risk - Standard Messaging n=20440 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk standard messaging arm received the same standard messaging as average risk patients.	High Risk - Intensification Messaging n=20412 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk intensification group received an additional bidirectional texting component in addition to the standard messaging.	High Risk - Not Individually Randomized n=1057 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine who were not individually randomized as a result of an automation issue. These patients still received the clinician nudges (default pended orders and peer comparison feedback) but did not receive any standard or intensification messaging.
Proportion of Patients Who Receive the Flu Vaccine at the Visit	7305 Participants	16663 Participants	6142 Participants	6142 Participants	312 Participants

SECONDARY outcome

Timeframe: 3 months

The secondary outcome is flu vaccination completion within 3 months after the first eligible primary care visit.

Outcome measures

Measure	Control n=27513 Participants Patients in clinics randomized to the control arm will receive standard of care.	Intervention n=52526 Participants Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders.	High Risk - Standard Messaging n=20440 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk standard messaging arm received the same standard messaging as average risk patients.	High Risk - Intensification Messaging n=20412 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine were randomized 1:1 to receive the high risk intensification messaging or remain in the standard messaging arm. Patients in the high risk intensification group received an additional bidirectional texting component in addition to the standard messaging.	High Risk - Not Individually Randomized n=1057 Participants Patients in the intervention clinics identified as high risk for noncompletion of the flu vaccine who were not individually randomized as a result of an automation issue. These patients still received the clinician nudges (default pended orders and peer comparison feedback) but did not receive any standard or intensification messaging.
Proportion of Patients Who Receive the Flu Vaccine Within 3 Months After the Visit	9371 Participants	19892 Participants	7329 Participants	7371 Participants	385 Participants

Adverse Events

Control

Serious events: 0 serious events

Other events: 5 other events

Deaths: 0 deaths

Intervention

Serious events: 1 serious events

Other events: 817 other events

Deaths: 0 deaths

Serious adverse events

Measure	Control n=27513 participants at risk Patients in clinics randomized to the control arm will receive standard of care.	Intervention n=52526 participants at risk Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders. Includes all intervention arm patients, including high risk.
Injury, poisoning and procedural complications Flu Vaccine Reaction	0.00% 0/27513 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.	0.00% 1/52526 • Number of events 1 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.

Other adverse events

Measure	Control n=27513 participants at risk Patients in clinics randomized to the control arm will receive standard of care.	Intervention n=52526 participants at risk Patients in clinics randomized to the intervention arm will receive the toolkit of clinician and patient facing nudges. Patient nudges will be pre-visit text message reminders (standard messaging content). Clinician nudges will be monthly peer comparison feedback and default pended orders. Includes all intervention arm patients, including high risk.
Social circumstances Messaging Opt-out	— 0/0 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.	1.5% 795/52526 • Number of events 795 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.
Surgical and medical procedures Duplicate Vaccination	0.02% 5/27513 • Number of events 5 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.	0.04% 22/52526 • Number of events 22 • Each patient was monitored for adverse events for 3 months. Adverse events were collected mid-way through the trial at 3 months for all participants enrolled through the end of December 2023 and again at the end of the trial follow up period, 3 months after enrollment ended (May 2024), for all enrolled participants. Per our DSMB-approved DSMP, severe adverse events will not include death as no reasonable evidence exists to suggest death would result from outreach or communication to encourage standard clinical care for flu vaccination. Therefore, All-Cause Mortality was not assessed. Adverse events are assessed for the Main Study Period arms only - Control (except Messaging, see footnote) and Intervention - as the patients in the High Risk arms are already accounted for in the Intervention arm total.

Additional Information

Dr. Shivan Mehta

University of Pennsylvania

Phone: 2679726027

Email: shivan.mehta@pennmedicine.upenn.edu

Results disclosure agreements

• Principal investigator is a sponsor employee
• Publication restrictions are in place