Abstract
In this paper we investigate, through a randomized controlled trial design, the impact of a personalized text messaging intervention designed to encourage college freshmen to refile their Free Application for Federal Student Aid (FAFSA) and maintain their financial aid for sophomore year. The intervention produced large and positive effects among freshmen at community colleges where text recipients were almost 14 percentage points more likely to remain continuously enrolled through the Spring of sophomore year. By contrast, the intervention did not improve sophomore year persistence among freshmen at four-year institutions among whom the rate of persistence was already high.
I. Introduction
Improving college affordability for students and their families has emerged as a top policy priority at both the state and federal levels particularly in light of heightened public concern about rapidly escalating tuition and the economic consequences of student debt (Lowrey 2013). Policymakers have responded with a variety of strategies to make college more affordable for students and their families. One approach has been to directly reduce the cost of college to students and families through efforts such as expanded Pell Grant funding at the federal level and the creation of merit-based aid programs at the state and local levels.
Another set of policy initiatives has focused on improving students’ and families’ awareness of available financial aid programs and on providing families with assistance to complete the Free Application for Federal Student Aid (FAFSA). The FAFSA is widely recognized as a crucial gateway to college access particularly for students from low- to moderate-income households (King 2004, Kofoed 2013). Yet, the complexity of the application, which requires students and families to provide an array of information about their income, assets, and family composition, may deter college-ready, low-income students from successfully matriculating (ACFSA 2005, Bettinger et al. 2012, Dynarski and Scott-Clayton 2006). This recognition has catalyzed a variety of efforts to provide students with additional information about, and assistance with, the financial aid process. These include the United States Department of Education (USDOE) FAFSA Completion Pilot, which provides school districts with real-time information about individual students’ FAFSA completion status; the privately funded College Goal Sunday, which offers students across the country individualized assistance with FAFSA completion; and interventions to integrate the FAFSA application into the income tax return process (Bettinger et al. 2012).1
While supporting students and their families with initial FAFSA filing during senior year of high school has received substantial attention, there has been comparatively less attention paid to helping students refile their FAFSA each year they are in college, despite the fact that students need to renew the FAFSA annually to maintain their financial aid. Yet, recent descriptive work documents that nationally 15–20 percent of freshman Pell Grant recipients in good academic standing do not successfully refile their FAFSA. Refiling rates are particularly low among community college students and students enrolled in certificate programs (Bird and Castleman 2014). Even among students who return for sophomore year, 9 percent of freshman-year Pell Grant recipients in good standing do not refile the FAFSA. Failure to renew the FAFSA is negatively associated with persisting in college or eventually earning a degree (Bird and Castleman 2014). Therefore, just as initial FAFSA filing is a critical step toward college access for many students, FAFSA refiling may be an important gateway to persistence and success in college.
In this paper we investigate, through a randomized controlled trial design, the impact of a low-touch intervention to improve FAFSA refiling rates among college freshman. In partnership with uAspire, a Boston-headquartered nonprofit organization that provides college and financial aid advising, we sent college freshmen who had worked with uAspire during high school a series of 12 personalized text message reminders about important steps in the FAFSA refiling process. The messages (1) provided students with information about resources at their own college or university where they could get help with financial aid; (2) reminded students about important aid-related deadlines and requirements, like maintaining satisfactory academic progress; and (3) offered students assistance from uAspire on key steps like FAFSA renewal. The intervention cost approximately $5 per student served.
This type of intervention has the potential to be particularly impactful for community college students. First, even among freshmen with strong GPAs, community college students are nearly three times more likely to fail to refile their FAFSA than freshmen at four-year institutions (Bird and Castleman 2014). Because community colleges are nonresidential, their students spend less time on campus compared to their counterparts at residential four-year institutions. Similarly, community college students typically have less access to individualized advising and may struggle to navigate bureaucratic processes around financial aid or course registration (Scott-Clayton 2015). Among Pell Grant recipients at community colleges (individuals who stand to benefit the most in terms of securing ongoing grant assistance from renewing their financial aid), students are more likely to be the first in their family to go to college and to work extensive hours while enrolled (Bird and Castleman 2014). For all of these reasons, the text messages may have provided valuable information and prompts about FAFSA refiling that students were not otherwise receiving.
To preview our results, we find large and positive effects of the financial aid text message campaign on the continued college persistence of first-year students at community colleges. Students who were initially enrolled in a community college and who received the text messages were nearly 12 percentage points more likely to persist into the Fall of their sophomore year of college compared to community college freshmen who did not receive the texts. This represented a 19 percent increase in persistence over the mean persistence rate in the control group. Moreover, these same students were 13.8 percentage points more likely to remain continuously enrolled through the Spring of their sophomore year in college as a result of the treatment. Impacts were particularly strong for community college freshmen with moderate academic performance in high school (high school GPAs of 3.0 or below). In contrast, among freshmen at four-year institutions, where persistence to sophomore year among the control group was 87 percent, there was no overall effect of the messages on persistence into sophomore year. We find suggestive evidence that the intervention operated differently for male and female freshmen at four-year institutions: The text messages decreased the probability that male students transferred out of four-year institutions in favor of two-year institutions while increasing the probability that female students did so.
We organize the remainder of our paper as follows. In Section II, we review the education policy, economics, and psychology literature relevant to interventions aimed at improving postsecondary access and success. In Section III, we describe our research design including the site, data, and sample; the design of the intervention; and the process of and timeline for randomization. In Section IV, we present our results. In Section V, we conclude with a discussion of these findings and their implications for policy, practice, and further research.
II. Literature Review
Students, particularly those from low-income backgrounds, commonly lack an accurate understanding of both the net costs of attending college and the availability of financial aid. For example, when surveyed, students from low-income families often provide estimates that substantially overstate actual tuition expenses (Avery and Kane 2004; Horn, Chen, and Chapman 2003; Grodsky and Jones 2007), and popular media coverage regarding the rising cost of attending a highly selective postsecondary institution fuels common misperceptions about the affordability of college (ACSFA 2005; Horn, Chen, and Chapman 2003). Further, large shares of parents and young adults—particularly those from low-income backgrounds—are unaware that federal, state, and institutional grant funds are available to reduce the net cost of college that families face (Avery and Turner 2010; Harris Poll 2002 and Sallie Mae Fund 2003, as cited by Bettinger et al. 2012).
Even among students who are aware of and understand the benefits of applying for financial aid, the complexity of the federal financial aid application process may deter college-ready students from successfully matriculating in college (ACSFA 2005, Bettinger et al. 2012, Dynarski and Scott-Clayton 2006). The near-term time and effort costs required to complete the FAFSA may be prohibitively high for students and their families and may lead them to procrastinate about or put off indefinitely completing the application, even if they recognize the long-term benefits associated with college (Madrian and Shea 2001, Thaler and Bernatzi 2004). For first generation college students who lack parental assistance with college or financial aid application processes, the FAFSA is likely to be particularly daunting. During the time period that students are engaged with college and aid applications, neurological systems that respond to immediate stimulation are at their peak activity, while brain systems required for self-regulation are still in development. As a result, adolescents are more impulsive and more likely to put off onerous tasks in favor of more pleasurable pursuits (Casey, Jones, and Somerville 2011; Steinberg 2008; Steinberg et al. 2009). High school seniors are particularly likely to procrastinate on tasks like the FAFSA, especially when their attention is spread thin across more pressing school, family, friendship, and work commitments (Castleman 2015; Castleman and Page 2014b; Castleman, Schwartz, and Baum 2015; Ross et al. 2013). This is especially true for students from lower-income communities who often have to devote substantial portions of their cognitive capacity and attention to more immediate stressors like financially supporting their families, taking care of siblings, or dealing with neighborhood violence (Mullainathan and Shafir 2013).
Encouragingly, various empirical studies demonstrate that providing students with assistance completing the FAFSA can generate substantial improvements in the rate at which students enter and persist in college. Students whose families were randomly assigned to receive assistance with the Free Application for Federal Student Aid (FAFSA) as part of their tax preparation at H&R Block offices were eight percentage points more likely to have completed two years of college than students whose families were not offered similar help (Bettinger et al. 2012). In another study, students who received school-based support to file the FAFSA were nearly 12 percentage points more likely to enroll in college than peers in previous graduating cohorts who were not offered additional support (Owen 2012).
Despite broad acknowledgement that FAFSA completion serves as a key milestone on the path to college for many high school seniors from low-income families, there has been little focus on FAFSA refiling, notwithstanding the fact that students have to renew their FAFSA on an annual basis to maintain their financial aid. Yet, substantial shares of freshman year financial aid recipients in good academic standing do not successfully refile their FAFSA. This is particularly the case among students enrolled in community colleges and certificate programs. Nationwide, 13.4 percent of community college freshman Pell recipients with a GPA of 3.0 or higher fail to refile their FAFSA compared with only 4.7 percent of their peers at four-year institutions (Bird and Castleman 2014).
Currently, standard efforts to support FAFSA renewal primarily consist of email and postal mail reminders from the USDOE as well as from students’ colleges and universities. However, communication channels such as these are less effective at reaching students. While the majority of students report texting with each other on a daily basis, only 6 percent report exchanging emails each day (Lenhardt 2012). The attention of college freshmen is often divided among a range of academic, extracurricular, and social commitments, so they may have a hard time remembering to refile their FAFSA (Castleman 2015; Castleman and Page 2014b; Castleman, Schwartz, and Baum 2015; Karlan et al. 2010; Ross et al. 2013). And while student-to-counselor ratios are high in secondary school, they are often even higher at the postsecondary level (Scott-Clayton 2015). Therefore, students may have trouble accessing professional assistance with refiling their FAFSA.
Lack of access to support with FAFSA refiling is likely to be particularly salient for students at community colleges where advising resources are often particularly limited and where students typically have to navigate complex bureaucracies in order to access individualized assistance (Scott-Clayton 2015). Furthermore, students at community colleges spend less time on campus, are more likely to be the first in their family to go to college, and are more likely to be balancing extensive work and family commitments (Bird and Castleman 2014). Each of these characteristics is likely to be negatively associated with knowing about the need to, or remembering to, refile the FAFSA.
While we are unaware of evidence from interventions designed to support students with FAFSA refiling specifically, a growing literature demonstrates that providing adolescents with additional information and prompts about college and financial aid can be quite impactful. For example, recent research illustrates how difficulty completing college-related procedural tasks can hinder college-intending high school graduates from realizing their postsecondary aspirations (Arnold et al. 2009, 2012; Castleman and Page 2014a, 2014b; Castleman, Page, and Schooley 2014). Encouragingly, low-cost, technological solutions—and text-based outreach in particular—show promise for supporting students to overcome these barriers. Text messaging has been used in a variety of policy arenas to support individuals to follow through on their intentions such as contributing to financial savings accounts or getting flu vaccinations (Karlan et al. 2010, Stockwell et al. 2012).
Educational researchers have begun to experiment with text messaging to improve student outcomes. For instance, personalized texts from teachers or school personnel can contribute to increased parental involvement and improved educational outcomes (Bergman 2013, Kraft and Dougherty 2013, Kraft and Rogers 2014). In the context of the transition to postsecondary education, college-intending high school graduates who were randomly assigned to receive text message reminders about important college and financial aid tasks required for successful matriculation were substantially more likely to enroll in college than students who did not receive the text messages (Castleman and Page 2014c). Taken together, both the theoretical and empirical literature suggest that providing students with personalized encouragement about, and the offer of support with, refiling the FAFSA has the potential to generate improvements in college persistence and success.
In this paper, we investigate whether providing college freshmen with personalized information and prompts about FAFSA refiling delivered via text message increases the rate at which they persist in college. Our paper is organized around the following questions:
Do college freshman who receive text message reminders about FAFSA renewal and the offer of help with refiling persist at higher rates than students who do not receive this outreach?
Does the impact of the text outreach vary by the type of institution in which a student is enrolled?
Answers to these questions are particularly timely and relevant given renewed focus by the Obama administration on improving college access and success for low-income students.
III. Research Design
During the 2012–13 academic year, we collaborated with uAspire to conduct the text-based campaign to promote financial aid renewal and sophomore year persistence among college freshmen. uAspire is a Boston-headquartered national nonprofit organization focused on issues of college affordability and financial literacy. Currently, their services include providing direct college financial aid advising to more than 14,000 high school students and their families in several metropolitan area school districts in Massachusetts as well as in Miami, Florida and the California’s Bay Area.
A. Data and Sample
The text campaign focused on college freshmen with whom uAspire had worked in either their Springfield, Massachusetts or Boston, Massachusetts site while the students were still in high school. Our analytic sample includes students who had received either text-based or peer mentor outreach as part of a Summer 2012 experiment to examine the impact of these outreach strategies on whether students matriculated on time in college (see Castleman and Page 2014c for details).2 We additionally restricted the sample to students for whom we had a cell phone number on file and who enrolled in college on time in the Fall of 2012.3 Across the two sites, the experimental sample includes 808 first-time college freshmen, 639 of whom had worked with uAspire in Boston and 169 of whom had worked with uAspire in Springfield.4
We utilize several different sources of data to investigate the impact of the freshman year text outreach on whether students persisted in college. First, we utilized uAspire administrative records to provide students’ demographic and prior academic achievement information. Specifically, we are able to observe students’ gender and race/ethnicity as well as a categorical, self-reported measure of high school GPA. In addition, because uAspire worked with many of these students on financial aid-related tasks during their senior year of high school, we are able to observe for a large share of students in the sample their expected family contribution (EFC) to the cost of the first year of college, as calculated by the U.S. Department of Education upon completion of the FAFSA. Within our focal sample of students, the continuous EFC variable is highly skewed owing to the fact that uAspire provides support within school districts that serve large shares of students from low-income households. Among students for whom EFC is observed, approximately 60 percent of students had an expected family contribution of zero. Therefore, we created a categorical EFC variable with the following categories: EFC of zero, nonzero EFC within the Pell-eligible range, and EFC outside of the Pell-eligible range.
Second, the text-messaging platform that we utilized to provide outreach to students captured the timing and content of all text-based communication. This included automated messages that were distributed to students as well as their responses and any student-advisor conversations that ensued. Third, uAspire advisors maintained logs of their interactions with students that occurred outside the text messaging platform, noting the content and tasks accomplished during those meetings. Finally, we matched all of these student-level data sources to data from the National Student Clearinghouse in order to examine the impact of the intervention on persistence into the sophomore year of college.5 As noted, the primary focus of the messages was FAFSA renewal. Therefore, it would have been optimal to have information on students’ FAFSA renewal status. Unfortunately, uAspire did not have comprehensive information on students’ renewal status, although we do observe, based on the advisor logs, whether an advisor provided support to a student to complete renewal.
In Table 1, we provide descriptive statistics for the students included in our experimental sample. Sixty-two percent of the students are female, and nearly all (90 percent) are nonwhite. Of those students for whom we observe expected family contribution, only 17 percent have an EFC outside of the Pell-eligible range. Therefore, the majority of students come from low-income households. Finally, in Fall 2012, approximately 28 percent of students were enrolled in a two-year community college, while 72 percent were enrolled as freshmen in a four-year institution. Students at community colleges were somewhat less likely to be female, more likely to be Hispanic, more likely to have an EFC of zero, and more likely to have lower high school GPAs than students who began college at four-year institutions.
Descriptive Statistics for Fall 2012 College Enrollers
One question is how well our sample of students generalizes to a nationally representative sample of students at four- and two-year institutions, and more specifically how similar our sample is to students enrolled at Massachusetts colleges and universities. It is possible, for instance, that our students differ by virtue of having worked with uAspire during high school and that other students may not respond in the same way to text message outreach about financial aid renewal. In Columns 3–6, we report analogous descriptive statistics for a nationally representative sample of students drawn from the National Postsecondary Student Aid Study of 2012 (NPSAS:12), and in Columns 7–9, we include descriptive statistics for the subset of the NPSAS:12 sample that was enrolled at Massachusetts colleges and universities. Compared to a nationally representative sample of college freshmen, students in our sample are more likely to be female and substantially more likely to be students of color and to have lower EFCs. These patterns tend to hold when comparing students in our sample to the national sample within the institutional sector (four- or two-year institution). Students in our sample are also more likely to have initially enrolled at a four-year institution than the national sample of first-time college students. Students who started at four-year institutions had higher GPAs on average, while students at community colleges had lower GPAs compared with the national sample, though this should be interpreted with caution as the uAspire high school GPAs are self reported. Similar patterns hold when comparing our analytic sample to the sample of freshmen at Massachusetts colleges and universities, though the differences in EFC are even more pronounced. One notable difference is that students in our analytic sample are more likely to attend two-year institutions than other college freshmen in Massachusetts. Taken collectively, these descriptive comparisons suggest that our sample tends to be substantially more underrepresented, both in terms of race/ethnicity and socioeconomic status, than college freshmen more generally.
B. Measures
We examine several indicators of students’ responsiveness to the text-based outreach. These include whether a student responded to the outreach via text; whether the student engaged with uAspire in a text-based interaction; whether the student met with a uAspire advisor during the course of the intervention; and whether the student received assistance from uAspire in renewing the FAFSA. We considered a student to have had a text-based interaction if he engaged in a two-way interaction via text that involved asking a question and participating in a back-and-forth communication. Interaction topics included how to renew the FAFSA, where to find scholarships, how to secure a high school diploma from one’s district, how to waive health insurance, how to obtain federal loans, and when to file taxes in relation to deadlines for FAFSA renewal.
To assess the impact of the freshman year outreach on students’ persistence to their sophomore year of college, we focus on several primary outcomes of interest: binary indicators of enrollment in a postsecondary institution in what would be the Fall and Spring of each student’s second year in college, as well as an indicator for whether students were continuously enrolled throughout the Spring of their second year in college; and indicators for sophomore enrollment in a two-year or four-year institution. Across outcomes, the explanatory variable of primary interest is the indicator for the experimental group to which each student was assigned. To increase precision, we include the academic, demographic, socioeconomic, and initial college enrollment covariates described in Table 1. We included indicator variables for missingness for any covariate with missing values. In addition, we include fixed effects for site-by-wave, the level of randomization, so that experimental contrasts are limited to within-site-by-wave comparisons, as described below.6
C. Intervention Design
Students selected for outreach received automated, text-based outreach from uAspire during the Spring semester of their freshman year of college. Because students included in the intervention had received financial aid-related support from uAspire during high school, freshman-year outreach from uAspire would not have been unusual for the students. Students in the control group did not receive this text-based outreach, but they received comprehensive support from a uAspire advisor if they initiated contact independently from the text messages. Message outreach began to students at the end of January 2013 and continued through the end of August of the same calendar year.7 After an introductory message, messages were sent to students approximately every two weeks. Messages primarily focused on financial aid and FAFSA renewal but also reminded students about related topics such as the importance of maintaining satisfactory academic progress in order to continue to qualify for financial aid. Please see the appendix for complete message content and the date on which messages were sent.
In our summer text-based intervention, messages explicitly prompted students to respond via text with questions or to seek additional one-on-one support. In contrast, in this school year intervention, messages more typically directed students to resources at their own college or university or to register for a FAFSA renewal event hosted by uAspire. Nevertheless, the messaging platform that we utilized allowed for two-way texting. Students could therefore respond directly to the automated text outreach to connect with a uAspire advisor who staffed the intervention.8 Signal Vine, a company focused on improving educational outcomes through the application of mobile technologies, delivered the messages and hosted the interactive messaging platform that allowed uAspire staff to monitor and respond to incoming text messages from students.
D. Randomization and Baseline Equivalence
From among the 808 students selected for the intervention, we randomly selected 413, approximately half, for outreach. Randomization was conducted separately by site and wave (see Footnote 7 for additional detail). In Table 2, we assess the equivalence of the treatment and control group students on all pretreatment covariates. In no instance do we detect significant differences in average values of baseline covariates between the two groups.9 Overall, we conclude that we have achieved baseline equivalence.
Assessing Balance in Covariates across Treatment Groups
E. Empirical Strategy
We rely on linear probability models to assess the impact of the text outreach on students’ responses to the outreach and well as on the college persistence outcomes. To assess overall impacts, we utilize an intent-to-treat model of the following general form:
(1)
where for student i in site j, ENROLL13ij is an indicator for enrollment in college in the Fall of 2013, αj is a set of site-by-wave fixed effects, and X is a vector of student-level covariates. TXij is an indicator for assignment to the treatment condition. Therefore, β1 represents the causal effect of being assigned to the text messaging intervention on sophomore year persistence.
For the reasons outlined above, FAFSA refiling may be particularly challenging but important for community college students. Therefore, in addition to this main specification, we also specify intent-to-treat (ITT) models that allow for differential effects of the text outreach by institutional sector. These models take the following general form:
(2)
In this model, θ2 represents the causal effect of being assigned to the text messaging intervention among students who initially enrolled in a two-year institution and θ3 represents the causal effect of assignment to the intervention for students who initially enrolled in a four-year institution. In our results, in addition to presenting estimates for parameters θ2 and θ3, we report the results of an F-test investigating whether the treatment effect among the two-year students is significantly different from that of four-year students (for example, testing θ2 – θ3 = 0).
We also investigate whether the financial aid texting campaign had differential effects for important subgroups of students by interacting the treatment indicator(s) with appropriate covariates. We focus in particular on whether the effects of the intervention varied based on students’ high school GPA and based on students’ gender. The former analysis is motivated by results from our Summer melt texting campaign, in which we found pronounced effects of the texting for students in the middle of the academic distribution, while the latter focus is motivated by the finding of differential effects by gender in prior college-going interventions (for example, Carrell and Sacerdote 2013). While it is possible that the effect of the intervention would also vary by selectivity or quality of institution that students attended, in practice there is relatively little variation in these measures within our sample. Our exploratory analyses did not reveal any differences in the efficacy of the texting intervention by institutional quality or selectivity.
IV. Results
We begin by examining students’ responsiveness to the freshman year text outreach. We present results associated with these outcomes in Table 3. Overall (top panel of Table 3), approximately 20 percent of students who received text outreach responded to at least one message (Column 1). This overall reply rate is lower than in our Summer text outreach campaign (Castleman and Page 2014c), but this difference is expected given that, as noted above, the messages did not include explicit prompts for students to message back. Rather, the messages more typically prompted students to follow up with their college or university or to sign up via another mechanism for a group event hosted by uAspire. Just over 11 percent of treatment group students engaged in a more substantial text-based advising conversation with a uAspire advisor as a result of the outreach (Column 2). Yet, the outreach did not improve the rate with which students received in-person FAFSA renewal assistance from a uAspire advisor. Nearly 9 percent of students in the control group obtained support with FAFSA refiling. These rates were virtually identical for students in the treatment group, overall (Table 3, Column 3). While not shown in the table, fewer than 5 percent of students opted out of receiving text messages throughout the campaign.10,11 Together with the response rates above, we interpret this to mean that students are generally positive or at least neutral about receiving this type of outreach.
Impact of Text Outreach on Student Interaction with uAspire Staff, Overall and by Type of Freshman Year Postsecondary Institution
In the bottom panel of Table 3, we examine response rates separately by whether students started at a four- or two-year institution. Similar shares of students at four- and two-year institutions responded to the text outreach. A significantly higher proportion of four-year freshmen in the treatment group interacted with an advisor via text than did treatment students at two-year institutions (13.3 percent versus 6.9 percent), but students at two-year institutions were significantly more likely to work with a uAspire advisor on renewing their FAFSA (12.3 percent versus 7.5 percent).
In Table 4, we present estimates of the impact of the intervention on persistence into the sophomore year of college, both overall (top panel) and by sector of freshman year institution (bottom panel). Overall, nearly 81 percent of control group students persisted to their second year of college (Column 1) and 74 percent remained continuously enrolled through the Spring of their sophomore year in college (Column 3). The treatment had no overall effect on the probability that students enrolled in Fall of sophomore year but, interestingly, led to a 4.6 percentage point increase in the probability that students were enrolled in the Spring of their second year in college.
Impact of Text Outreach on Sophomore Enrollment, Overall and by Type of Freshman Year Post-Secondary Institution
These results mask considerable heterogeneity in the impact of the text-based financial aid nudges by whether students were initially enrolled at a four- or a two-year institution. Absent the intervention, 87 percent of students who were freshmen at a four-year institution and 64 percent of students who were freshmen at a two-year institution persisted to the Fall of their second year of college. Eighty-one percent of four-year starters in the control group remained continuously enrolled throughout sophomore year compared with only 54 percent of students at community colleges. While the intervention had no significant overall impact on persistence among the four-year enrollers (for whom persistence was otherwise relatively high), it led to a positive and significant increase of nearly 12 percentage points in persistence to the Fall of the second year among community college students. Moreover, community college students in the treatment group were almost 14 percentage points more likely to remain continuously enrolled through the Spring of their sophomore year than students in the control group suggesting that the text messages added considerable stability to community college students’ postsecondary enrollments. Though not shown in the table, these results were primarily driven by increasing the share of students that persisted at two-year institutions.12
In Table 5, we estimate effects of the text messaging campaign on students’ sophomore enrollment for students with lower (below 3.0) and higher (3.0 or above) high school GPAs. As in the bottom panel of Table 4, we conduct these analyses within sector of institution that students attended freshman year. We find no differences in the efficacy of the intervention by GPA within freshmen who started at four-year institutions. But among community college students, the effects are particularly large for students with lower high school GPAs. Students with high school GPAs below 3.0 in the treatment group were over 23 percentage points more likely to enroll in the Fall of sophomore year and almost 20 percentage points more likely to remain continuously enrolled through their second year. As above, these results are primarily driven by increasing the share of students with lower GPAs that persist at two-year institutions.13
Impact of Text Outreach on Sophomore Enrollment, by GPA and Type of Freshman Year Post-Secondary Institution
Finally, in Table 6 we investigate whether the text intervention had differential effects by gender. We conduct these analyses within sector, as in Table 5. At both four-year and two-year institutions, we do not find significant differences in the efficacy of the intervention between males and females. However, we do find evidence to suggest that the texts operated through different pathways for males and females within each sector. Among freshmen at four-year institutions, males who received the text messages were more likely to stay at four-year institutions and less likely to transfer to two-year institutions than students in the control group. The opposite was true for females, who were less likely to stay in the four-year sector and appeared more likely to switch to two-year institutions. Among students at community colleges, males in the treatment group were substantially more likely to transfer to four-year institutions than their counterparts in the control. Females, by contrast, were much more likely to remain enrolled at two-year institutions than students in the control group.
Impact of Text Outreach on Sophomore Enrollment, by Gender and Type of Freshman Year Post-Secondary Institution
V. Discussion
At the modest cost of approximately $5 per student served, the freshman year text outreach had a large and positive impact on sophomore year persistence for students enrolled in a community college. In contrast, the additional outreach and prompting had no discernable effect on persistence among four-year enrollers. This pattern of results is consistent both with the institutional differences we discuss above and with the compositional differences of community college versus four-year enrollees. For instance, over 71 percent of the community college freshmen in the sample had an expected family contribution of zero compared with 56 percent of four-year starters. For these students, their ability to persist in college may be particularly dependent on whether they are able to maintain need-based financial aid.
Absent interventions such as these, community college students, even those who are in good academic standing, are less likely to refile the FAFSA (Bird and Castleman 2014). Further, they are substantially less likely to persist in college than their four-year counterparts.14 There are several factors that are likely to contribute to lower rates of FAFSA refiling among community college students. Because advising resources are quite limited and students spend limited time on campus, information about FAFSA refiling may simply not reach community college students. For example, more passive forms of communication about FAFSA renewal, such as posters and message board announcements, are more likely to go unseen or unnoticed by community college students. In addition, a lack of social capital—less interaction with peers and staff in college and less college guidance at home—may mean that community college students are unaware that they even have to renew their FAFSA in order to maintain their financial aid (Bird and Castleman 2014). By virtue of being spread thin among academic, work, family, and social commitments, community college students may be less likely to devote time to FAFSA refiling. In addition, as we have observed in the data presented here, absent encouragement to do so, community college students may be less likely to seek help with challenging processes like FAFSA renewal. Therefore, outreach with information and the offer of support may be particularly beneficial for these students. Taken collectively, these factors suggest that (1) there is greater room for improvement on FAFSA refiling and sophomore year persistence within the community college population, and (2) community college students may be particularly responsive to the informational and behavioral content of the text messages we sent.
The large positive effects for community college students with lower high school GPAs (below 3.0) is consistent with Bird and Castleman’s (2014) finding that these students are substantially less likely to refile their FAFSAs. By encouraging FAFSA filing among this population, the intervention may have increased the share of these students who renewed their financial aid and therefore felt they could afford to continue into the sophomore year of college. This pattern of results is also consistent with Castleman and Page’s (2014c) Summer melt texting intervention in which effects were most pronounced for students in the middle of the high school GPA distribution.
We offer two potential explanations for the differential transfer patterns we observe between males and females. One possibility, consistent with lab-based research on differential risk responses by gender (for example, Eckel and Grossman 2008), is that males and females have different levels of risk preference for debt accumulation once they are in college. Females may be more risk averse than males to accumulate debt required to continue their education, which could explain why they are less likely to remain at four-year institutions after freshman year and substantially more likely to persist at two-year institutions. Another possibility, noted by Carrell and Sacerdote (2013), is that females and males have different employment preferences, with some occupations more popular among females (for example, in healthcare) only requiring an associate’s degree. We are admittedly limited in our data to test these hypotheses but view them as important areas for further inquiry into gender differences in response to financial aid-related information and nudges.
While the positive impacts we observe were concentrated among community college students in this study, we nevertheless believe that this type of low-cost outreach still has promise for supporting students at four-year institutions as well. This relates to an important limitation of this study. Compared to the sophomore year retention rates of 64 and 87 percent for two-year and four-year college freshmen, respectively, in our control group, national analogs of these figures are far lower, at 50 and 75 percent. Therefore, the results from our experiment, which was conducted primarily with freshmen at Massachusetts colleges and universities, may lack generalizability and therefore underestimate the potential impact that this type of intervention could yield in other settings where college persistence rates are lower, particularly among disadvantaged populations of students.
A related issue we discuss above is how well our experimental sample generalizes to the population of college freshmen in Massachusetts or in other locations given that they had a prior relationship with uAspire and moreover had in many cases engaged in interactive text messaging as a means of communication with uAspire staff. Our sample has substantially more students from lower-income backgrounds and students of color than either the national sample or the Massachusetts sample of college freshmen and as a result may have had less access to family-based or professional college and financial aid guidance and information. Therefore, they may have been particularly receptive to the personalized information and offer of assistance provided by our intervention.
One question that emerges is whether college freshmen who did not have a previous relationship with an organization like uAspire would have been receptive to receiving text-based messaging about financial aid from their college. Our sense is that students are increasingly receptive to this type of outreach. In a project to provide incoming community college students with personalized information about the loan origination process, over 80 percent of students opted to receive messages (Castleman in progress). Signal Vine, the texting platform with which we worked on this intervention, reports that across its clients in the secondary and higher education sector the majority of eligible students opt to receive messages when offered.
An additional limitation of this investigation is that we were not able to observe actual FAFSA refiling behavior. Future work building on this investigation would ideally include data collection on actual FAFSA refiling, the mechanism that we hypothesize drove the improvements in sophomore year retention among community college students. Therefore, we hope to investigate the benefit of this text-based FAFSA renewal intervention in a larger and more representative setting and with implementation partners (for example, colleges and universities) that are able to observe not only sophomore year college enrollment but also FAFSA refiling at the end of freshman year. That being said, encouraging students to renew their FAFSA remains the most likely mechanism through which our intervention led to increased sophomore year persistence for community college freshmen. As we report above, response rates to the text messages were modest relative to our prior summer melt texting intervention, as was the share of students that interacted with a uAspire advisor as a result of the intervention (Castleman and Page 2014c). This suggests that the messages may have operated instead by providing students with information they did not have (for example, that they had to renew the FAFSA each year to maintain their financial aid); by prompting them to follow through on FAFSA refiling rather than put it off; or by encouraging them to seek out support from the financial aid counselor at their institution. These channels would be consistent with student responses in a survey we conducted following our summer melt texting intervention, in which we asked students how, if at all, the messages affected their decision-making and behaviors in the college matriculation process (Arnold et al. 2013).
It is also worth noting that our estimated costs of the intervention ($5 per student) do not account for several costs absorbed by uAspire and, in all likelihood, by the colleges and universities where students were enrolled. The project required both the time of uAspire advisors to respond to students’ text messages when they wrote back with questions or requesting assistance, as well as investment of a project manager’s time to oversee the project implementation. We do not directly observe whether students sought out assistance from their financial aid office, but it is reasonable to assume that financial aid officers had to time to meet with students as a result of our intervention.
Were a college or university—or a statewide system of institutions—to implement this intervention at scale, the per-student costs would likely be higher if institutions were required to increase their staff capacity to respond to messages. To provide a benchmark for these costs, in Castleman’s (in progress) loan origination intervention the participating community college hired a financial aid counselor to staff the texting intervention at an annual cost of approximately $70,000 (inclusive of benefits) with approximately 2,500 students receiving messages during the year. Assuming a similar cost per student served to implement our financial aid renewal texting campaign, this suggests a per student cost of closer to $30–$40. If the effects of our intervention persist to degree attainment, however, the likely benefits that would accrue to individuals and society would merit even this larger per student investment. Nor does our benefit-cost estimate factor in financial benefits or cost savings that might accrue to the institution from increased freshman-to-sophomore year retention and more stable enrollment among its students.
In conclusion, the evidence presented here contributes to a growing body of research that suggests that low-cost strategies such as utilizing text messaging to provide students with information and to connect them to support hold promise for helping students to successfully navigate complex educational processes like FAFSA refiling and completing the many steps required in the Summer transition to college (for example, Castleman and Page 2014c). Similar strategies may prove useful in supporting students with processes such as considering whether and where to apply to college, filing the FAFSA for the first time, and selecting courses once in college. Such low-cost strategies can help an increasingly diverse population of students to access higher education and to persist and succeed once they have matriculated.
Appendix Text Message Content
MESSAGE PURPOSE: Introductory Message
DELIVERY DATE: 1/28/13
MESSAGE TO STUDENT: Hi [STUDENT NAME] it’s ACCESS, now known as uAspire! We’ll send u texts w/info about fin aid for next yr. Save the # so u know it’s us. Call [UASPIRE_SITE_PHONE].
MESSAGE PURPOSE: Contact Your Financial Aid Office/Liaison Reminder
DELIVERY DATE: 1/31/13
MESSAGE TO STUDENT: Hi [STUDENT NAME]. Need financial aid help as you return to [COLLEGE_ABBREV] this semester? Call ur fin aid office for support [SB_LIAISON_PHONE].
MESSAGE PURPOSE: FAFSA Renewal Event #1 at uAspire
DELIVERY DATE: 2/19/13
MESSAGE TO STUDENT: Want free FAFSA help? Come to uAspire FAFSA party [FAFSA_Renewal_Date_Time]at [FAFSA_Renewal_Location]! Sign up here [FAFSA_Renewal_RSVP_URL]
MESSAGE PURPOSE: SAP Intro
DELIVERY DATE: 2/25/13
MESSAGE TO STUDENT: Hi [STUDENT NAME]! Did u get aid this year? U must have Satisfactory Academic Progress (SAP), a minimum GPA, to keep aid. Info: [SAP_POLICY_URL]
MESSAGE PURPOSE: FAFSA Reminder #1
DELIVERY DATE: 3/5/13
MESSAGE TO STUDENT: Remember to do ur FAFSA at fafsa.gov ASAP! Call us at [UASPIRE_SITE_PHONE] or ur aid office [FIN_AID_OFFICE_PHONE]. File taxes for free @ [SITE_EITC_URL]
MESSAGE PURPOSE: FAFSA Renewal Event #2 at uAspire
DELIVERY DATE: 3/14/13
MESSAGE TO STUDENT: Want free FAFSA help? Come to uAspire FAFSA party [FAFSA_Renewal_Date_Time] at [FAFSA_Renewal_Location]! Sign up here [FAFSA_Renewal_RSVP_URL]
MESSAGE PURPOSE: SAP Academic Resources
DELIVERY DATE: 3/21/13
MESSAGE TO STUDENT: Hi, its uAspire! Students can lose fin aid if GPA is not high enough. Want to bring up GPA? Use support on campus: [ACADEMIC_SUPPORT_URL].
MESSAGE PURPOSE: FAFSA Reminder #2
DELIVERY DATE: 4/1/13
MESSAGE TO STUDENT: Last reminder to fill out FAFSA! Once you/your parents file 2012 taxes, use IRS Data Retrieval Tool to fill in FAFSA at fafsa.gov. For help, call 800-433-3243
MESSAGE PURPOSE: Award Letter
DELIVERY DATE: 4/17/13
MESSAGE TO STUDENT: Hi [STUDENT NAME]it’s Award Letter time! Make sure u get urs from ur fin aid office and ask them for help if u have questions. Or call us at [UASPIRE_SITE_PHONE].
MESSAGE PURPOSE: SAP/Academic Supports Reminder
DELIVERY DATE: 4/29/13
MESSAGE TO STUDENT: Almost time for finals! Want help bringing up ur GPA or studying for finals? Use these free academic supports: [ACADEMIC_SUPPORT_URL].
MESSAGE PURPOSE: Summer Jobs
DELIVERY DATE: 5/6/13
MESSAGE TO STUDENT: Summer jobs are taking apps and they fill up fast! Apply now so u can make money for summer and college. Visit [STUDENT_EMPLOYMENT_URL] for help.
MESSAGE PURPOSE: Summer Support
DELIVERY DATE: 5/29/13
MESSAGE TO STUDENT: Hi again [STUDENT NAME]! Congrats on finals! If u have financial aid questions for fall, call uAspire at [UASPIRE_SITE_PHONE] for help or to set up a meeting.
MESSAGE PURPOSE: Tuition Bill
DELIVERY DATE: Week of 06/24/13
GENERIC MESSAGE TO STUDENT: [insert inst. name]’s fall tuition bill will be due [insert rough date]. Watch ur student acct/email for the bill. Need help? Contact fin aid [insert fin aid phone #] or uAspire [XXX-XXX-XXXX].
MESSAGE PURPOSE: Tuition Bill / Payment Plan
DELIVERY DATE: Week of 07/01/13
GENERIC MESSAGE TO STUDENT: [insert inst. name] fall tuition bill is due [X/XX - insert rough date]. One way to pay is w/a payment plan. Call fin aid [insert fin aid phone #] or uAspire [XXX-XXX-XXXX] for help.
MESSAGE PURPOSE: Health Insurance / Waiver Process
DELIVERY DATE: Week of 07/15/13
GENERIC MESSAGE TO STUDENT: Hi, this is uAspire. [insert inst. name] will charge u for health insurance. If u have ur own, try to waive the school insurance at [insert inst. tiny url waiver process].
MESSAGE PURPOSE: Work Study
DELIVERY DATE: Week of 07/29/13
GENERIC MESSAGE TO STUDENT: Work study $ on your award letter? Find [insert inst. name] work study jobs at [insert tiny url for inst. work study]. Apply early – they go fast!
MESSAGE PURPOSE: Health Insurance / Waiver Process
DELIVERY DATE: Week of 08/05/13
GENERIC MESSAGE TO STUDENT: REMINDER. Don’t forget to waive the school health insurance ASAP if you have your own insur. Visit [insert inst. waiver-specific tiny url] for help.
MESSAGE PURPOSE: Visit Your Financial Aid Office/Liaison Reminder
DELIVERY DATE: Week of 08/12/13
GENERIC MESSAGE TO STUDENT: Need fin aid help during the school yr? uAspire and ur fin aid office are here for u. uAspire: [XXX-XXX-XXXX], Fin Aid Office [XXX-XXX-XXXX; IF SUCCESS BOSTON, insert FIN AID LIAISON CONTACT]
MESSAGE PURPOSE: Conclusion
DELIVERY DATE: Week of 8/19
GENERIC MESSAGE TO STUDENT: Hi [STUDENT NAME]. Last uAspire text for now! We hope you have a great year! Need fin aid help as u return to school? Contact ur fin aid office [XXX-XXX-XXXX].
Footnotes
Benjamin L. Castleman is an assistant professor of education and public policy at the University of Virginia. Lindsay C. Page is an assistant professor of research methodology at the University of Pittsburgh. The data used in this article can be obtained beginning July 2016 through June 2019 from Benjamin L. Castleman, PO Box 400265, Charlottesville VA 22904, castleman{at}virginia.edu or Lindsay C. Page, 5918 WWPH, 230 S. Bouquet Street, Pittsburgh PA 15260, lpage{at}pitt.edu. The authors are grateful for the collaboration of uAspire, and especially Alexandra Chewning, in the design and implementation of this intervention. The authors thank seminar participants at Columbia University, the University of Pittsburgh, Stanford University, the University of Virginia, the SREE 2014 Spring conference, and APPAM 2014 Fall conference for comments and suggestions on earlier versions of this paper. The authors are grateful for generous financial support from the Heckscher Foundation for Children and to Kelli Bird for excellent research assistance. All opinions expressed in this paper and any errors or omissions are the authors’. Authorship order is determined alphabetically.
↵1. To learn more about the FAFSA Completion Pilot, see http://www.luminafoundation.org/tag/fafsa/. To learn more about College Goal Sunday, see http://www.collegegoalsundayusa.org/Pages/default.aspx.
↵2. Approximately 75 percent of eligible high school graduates provided working cell numbers for the Summer 2012 experiment. We did not have a second opt-in procedure for students to participate in the freshman year financial aid texting intervention.
↵3. We relied on data from the National Student Clearinghouse to observe whether students were enrolled in Fall 2012.
↵4. We exclude from our analytic sample an additional 21 students who were enrolled in transitional college programs in the year after high school graduation.
↵5. An important point about the NSC data is that coverage rates vary considerably by state. For instance, in West Virginia the NSC only covers 68 percent of higher education institutions. Fortunately, the coverage rates are fairly high in Massachusetts (94 percent) where the majority of students in our experimental sample attended college (Dynarski, Hemelt, and Hyman 2013). Specifically, 88 percent of students in our sample were enrolled as freshman in a Massachusetts-based college or university. Among those enrolled as freshmen in a two-year institution, 99 percent attended a Massachusetts institution.
↵6. The impact estimates on which we focus are based on models fit with the full set of baseline covariate controls. Our results are unchanged, both in terms of impact magnitude and statistical significance, by inclusion of baseline covariates. Results from models that do not include covariate control are available upon request.
↵7. Eligible recipients were divided into two separate waves. Working with uAspire, we initially identified 526 students who met the eligibility criteria described above, 270 of whom we selected to begin receiving messages on January 28, 2013. In early February, uAspire identified an additional 282 students who met the eligibility criteria. One hundred forty-three from this second sample began receiving messages on February 18, 2013. Following the second message (on February 19th), both groups remained on the same messaging schedule for the remainder of the intervention. We include a fixed effect for the message wave in our analyses to account for the possibility of varying impacts between waves on students’ persistence.
↵8. uAspire assigned one primary advisor to regularly review the message portal and assigned other advisors to follow up with students who required more in-depth attention and support. These assignments were made based on whichever advisor happened to be available (an advisor who was not meeting with high school students as part of uAspire’s core programming).
↵9. In the results that follow, we additionally present impacts disaggregated by sector of freshman year institution (two-year versus four-year). We do not detect covariate imbalance in the subgroups examined. These additional baseline checks are available upon request.
↵10. The requests to stop messages were either cursory (for example “STOP”) or polite (“please stop texting me”). We did not find evidence in terms of angry responses to the messages that students were negatively impacted by the texts.
↵11. This is not necessarily indicative of what the stop out rate would be in a population of college freshmen who had not previously worked with or received text messages from an organization like uAspire.
↵12. Results available upon request.
↵13. We largely fail to reject the hypothesis that the impacts for two-year freshmen differ by GPA. This may be primarily an issue of precision given that a relatively small number of students with high GPAs enrolled in two-year institutions.
↵14. NCHEMS Information Center for Higher Education Policymaking and Analysis. http://www.higheredinfo.org/dbrowser/?level=nationandmode=mapandstate=0andsubmeasure=223.
- Received June 2014.
- Accepted November 2014.
In this issue
Jump to section
Related Articles
Cited By...
- Nudging Demand for Academic Support Services: Experimental and Structural Evidence from Higher Education
- Low-Touch Attempts to Improve Time Management among Traditional and Online College Students
- The Impact of Prior Learning Assessments on College Completion and Financial Outcomes
- More than Dollars for Scholars: The Impact of the Dell Scholars Program on College Access, Persistence, and Degree Attainment