Trial Outcomes & Findings for Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake (NCT NCT05009251)

Results Overview

Received flu vaccination

Recruitment status

COMPLETED

Study phase

NA

Target enrollment

45061 participants

Primary outcome timeframe

Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Results posted on

2025-01-03

Participant Flow

Participant milestones

Participant milestones
Measure	No-Contact Control Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Overall Study STARTED	9017	9013	9011	9003	9017
Overall Study COMPLETED	8892	8886	8868	8864	8883
Overall Study NOT COMPLETED	125	127	143	139	134

Reasons for withdrawal

Reasons for withdrawal
Measure	No-Contact Control Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Overall Study Protocol Violation	100	105	119	120	119
Overall Study Death	25	22	24	19	15

Baseline Characteristics

Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake

Baseline characteristics by cohort

Baseline characteristics by cohort
Measure	No-Contact Control n=9017 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=9013 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=9011 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=9003 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=9017 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message	Total n=45061 Participants Total of all reporting groups
Age, Customized >=18 years and <65	6018 Participants n=5 Participants	5984 Participants n=7 Participants	6049 Participants n=5 Participants	6001 Participants n=4 Participants	6004 Participants n=21 Participants	30056 Participants n=8 Participants
Age, Customized >=65	2999 Participants n=5 Participants	3029 Participants n=7 Participants	2962 Participants n=5 Participants	3002 Participants n=4 Participants	3013 Participants n=21 Participants	15005 Participants n=8 Participants
Sex: Female, Male Female	6038 Participants n=5 Participants	6121 Participants n=7 Participants	6107 Participants n=5 Participants	6086 Participants n=4 Participants	6140 Participants n=21 Participants	30492 Participants n=8 Participants
Sex: Female, Male Male	2979 Participants n=5 Participants	2892 Participants n=7 Participants	2904 Participants n=5 Participants	2917 Participants n=4 Participants	2877 Participants n=21 Participants	14569 Participants n=8 Participants
Ethnicity (NIH/OMB) Hispanic or Latino	381 Participants n=5 Participants	351 Participants n=7 Participants	374 Participants n=5 Participants	353 Participants n=4 Participants	365 Participants n=21 Participants	1824 Participants n=8 Participants
Ethnicity (NIH/OMB) Not Hispanic or Latino	8608 Participants n=5 Participants	8635 Participants n=7 Participants	8607 Participants n=5 Participants	8627 Participants n=4 Participants	8630 Participants n=21 Participants	43107 Participants n=8 Participants
Ethnicity (NIH/OMB) Unknown or Not Reported	28 Participants n=5 Participants	27 Participants n=7 Participants	30 Participants n=5 Participants	23 Participants n=4 Participants	22 Participants n=21 Participants	130 Participants n=8 Participants
Race/Ethnicity, Customized American Indian or Alaska Native	11 Participants n=5 Participants	19 Participants n=7 Participants	14 Participants n=5 Participants	16 Participants n=4 Participants	16 Participants n=21 Participants	76 Participants n=8 Participants
Race/Ethnicity, Customized Asian	27 Participants n=5 Participants	21 Participants n=7 Participants	23 Participants n=5 Participants	26 Participants n=4 Participants	27 Participants n=21 Participants	124 Participants n=8 Participants
Race/Ethnicity, Customized Native Hawaiian or Other Pacific Islander	15 Participants n=5 Participants	12 Participants n=7 Participants	12 Participants n=5 Participants	16 Participants n=4 Participants	20 Participants n=21 Participants	75 Participants n=8 Participants
Race/Ethnicity, Customized Black or African American	348 Participants n=5 Participants	286 Participants n=7 Participants	300 Participants n=5 Participants	330 Participants n=4 Participants	300 Participants n=21 Participants	1564 Participants n=8 Participants
Race/Ethnicity, Customized White	8596 Participants n=5 Participants	8660 Participants n=7 Participants	8648 Participants n=5 Participants	8602 Participants n=4 Participants	8636 Participants n=21 Participants	43142 Participants n=8 Participants
Race/Ethnicity, Customized Unknown or Not Reported	20 Participants n=5 Participants	15 Participants n=7 Participants	14 Participants n=5 Participants	13 Participants n=4 Participants	18 Participants n=21 Participants	80 Participants n=8 Participants

PRIMARY outcome

Timeframe: Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Population: Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.

Received flu vaccination

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Vaccination at 2 Weeks After Final Outreach Date Not Vaccinated	5798 Participants	5715 Participants	5560 Participants	5583 Participants	5662 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date Vaccinated	2668 Participants	2807 Participants	2938 Participants	2887 Participants	2863 Participants

SECONDARY outcome

Timeframe: Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start

Received a flu vaccination

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Vaccination at 9 Weeks After Final Outreach Date Vaccinated	3286 Participants	3429 Participants	3525 Participants	3535 Participants	3449 Participants
Flu Vaccination at 9 Weeks After Final Outreach Date Not Vaccinated	5180 Participants	5093 Participants	4973 Participants	4935 Participants	5076 Participants

SECONDARY outcome

Timeframe: 8 months (between September 9, 2021 and April 30, 2022)

Received a "high confidence flu" diagnosis (with positive polymerase chain reaction \[PCR\]/antigen/molecular test) and/or "likely flu" diagnosis (as assessed via International Classification of Disease \[ICD\] codes or Tamiflu administration or positive PCR/antigen/molecular test) Note that "likely flu" is a superset of the "high confidence flu" diagnoses.

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Diagnosis No flu diagnosis	8336 Participants	8382 Participants	8365 Participants	8346 Participants	8385 Participants
Flu Diagnosis Flu diagnosis	130 Participants	140 Participants	133 Participants	124 Participants	140 Participants

SECONDARY outcome

Timeframe: 11 months (between September 9, 2021 and July 31, 2022)

Diagnosed with flu-related complications

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Complications No flu complications	5916 Participants	5904 Participants	5910 Participants	5861 Participants	5853 Participants
Flu Complications Flu complications	2550 Participants	2618 Participants	2588 Participants	2609 Participants	2672 Participants

SECONDARY outcome

Timeframe: 11 months (between September 9, 2021 and July 31, 2022)

Visited ER or was hospitalized NOTE: Our prespecified outcome was description was "Visited ER, was hospitalized, or had flu-related insurance claims." We did not receive claims data, so this outcomes includes only ER visits and hospitalizations.

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Healthcare Utilization No utilization	5203 Participants	5221 Participants	5205 Participants	5160 Participants	5311 Participants
Healthcare Utilization Utilization	3263 Participants	3301 Participants	3293 Participants	3310 Participants	3214 Participants

OTHER_PRE_SPECIFIED outcome

Timeframe: Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Population: Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.

Received a flu vaccination

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Vaccination at 2 Weeks After Final Outreach Date by Gender Female	1740 Participants	1840 Participants	1933 Participants	1878 Participants	1933 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Gender Male	928 Participants	967 Participants	1005 Participants	1009 Participants	930 Participants

OTHER_PRE_SPECIFIED outcome

Timeframe: Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Population: Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.

Received Flu Vaccination

Outcome measures

Outcome measures
Measure	No-Contact Control n=8466 Participants Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.	Reminder Control n=8522 Participants Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message	High Risk Only n=8498 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Medical Records n=8470 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message	High Risk With Explanation Based on Algorithm n=8525 Participants Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk. Reminder: Mailed letter, short message service (SMS) text, and/or patient portal message Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message
Flu Vaccination at 2 Weeks After Final Outreach Date by Race White	2586 Participants	2742 Participants	2857 Participants	2801 Participants	2796 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Race American Indian or Alaska Native	2 Participants	3 Participants	1 Participants	2 Participants	7 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Race Asian	8 Participants	5 Participants	7 Participants	12 Participants	7 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Race Black	67 Participants	51 Participants	68 Participants	61 Participants	49 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Race Native Hawaiian or Other Pacific Islander	4 Participants	1 Participants	1 Participants	6 Participants	3 Participants
Flu Vaccination at 2 Weeks After Final Outreach Date by Race Unknown	1 Participants	5 Participants	4 Participants	5 Participants	1 Participants

OTHER_PRE_SPECIFIED outcome

Timeframe: Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Population: Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.

Received flu vaccination