Abstract
Recent studies show that texting-based interventions can produce educational benefits in children across a range of ages. We study effects of a text-based program for parents of kindergarten children, distinguishing a general program from one adding differentiation and personalization based on each child’s developmental level. Children in the differentiated and personalized program were 63 percent more likely to read at a higher level (p < 0.001) compared to the general group, and their parents reported engaging more in literacy activities. Effects were driven by children further from average levels of baseline development, indicating that the effects likely stemmed from text content.
I. Introduction
Educational interventions based on behavioral economics principles have shown promise for combatting some of the persistent disparities in education outcomes. Some of these interventions focus on helping participants hold their attention to tasks that need to be completed repeatedly over long periods of time (Bergman 2016). Others provide small bits of information regularly with easily operationalized tasks in order to overcome both information asymmetries and the cognitive load required for behavior change (York, Loeb, and Doss 2019). This information and support encourages parents and students to behave in ways that are more consistent with positive educational outcomes. Researchers have fielded successful interventions at all levels of education, ranging from prekindergarten (York, Loeb, and Doss 2019), to K–12 (Kraft and Rogers 2015; Bergman 2016), to the transition to college (Hoxby and Turner 2013; Castleman and Page 2015). Such programs, due to their low cost and ease of implementation, provide researchers with opportunities not only to directly support students and parents but also to test the mechanisms underlying the effects of these programs.
This study aims to identify the importance of personalization and differentiation within a text-messaging program for parents of young children. Personalization conveys a combination of child-specific information and a potential increased sense of familiarity. It may provide parents with better information about their child and encourage a sense of connection that could lead to greater incentives for behavior change. Differentiation provides activities for parents that are targeted to their child’s level of development and thus is potentially more effective for generating learning gains than a generic program. Differentiation may, in turn, encourage parents to engage more with the program if their children more successfully complete the developmentally appropriate activities. On the other hand, if program-inspired behavior change comes solely or primarily from holding attention through regular reminders (“nudges”), we would not expect either differentiation or personalization to affect program effectiveness.
We field a randomized control trial to explicitly test the additional benefits of differentiating and personalizing information in a program for kindergarten parents modeled after the original READY4K! program for prekindergarten parents. The READY4K! program has been shown to increase the number of reported academic activities done at home and in turn to increase preliteracy skills of children (York, Loeb, and Doss 2019). This study follows the first cohort of participants from the original experiment into their kindergarten year, recruits additional kindergarten families, and randomizes families to receive a small number of control text messages unrelated to literacy, general literacy texts, or literacy texts that are differentiated and personalized.
We employ a “light touch” differentiation and personalization that leverages extant data to adjust text messages. We personalize the texts by informing parents how well their child knew a particular skill based on the child’s performance on formative assessments. We then differentiate the texts by aligning the activity more closely to the child’s skill level. Through this experimental design we are able to test whether the differentiated and personalized information provision generates a greater parental response and greater academic gains in reading when compared to a general provision of information. Thus, we are able to identify the causal effect of differentiation and personalization separate from the effect of information provision alone.
We find that differentiation and personalization increases parental takeup of the program as measured by parental survey responses and increases the reading ability of students as measured by district assessments. Specifically, differentiation and personalization caused students to be 63 percent more likely to move up a reading level than their peers in the general program (p < 0.001), with the academic effects particularly pronounced for students in the bottom and top quartiles of the baseline skill distribution. The differentiated and personalized texts also positively affected parents’ reports of the ease of building reading skills by 32 percent of a standard deviation (p < 0.05) when compared to the general texting program, while increasing parental engagement in literacy activities with their child by 26 percent of a standard deviation (p < 0.05) when compared to the control group. There is some suggestive evidence that the differentiation caused parents to use the texts more, indicating that a closer match of the text to the children’s skill level led parents to engage in the activities to a greater extent. The greater amount of information in the texts, however, may have caused parents to visit their children’s school less often.
II. Background
Recent experiments in education have demonstrated that parent texting interventions based on behavioral economics principals are effective in improving students’ educational outcomes. The precursor to this study, the READY4K! experiment conducted in the prekindergarten context, applied several of these principals. Families in the treatment group received three literacy texts per week for eight months. The program provided a parenting curriculum that was designed to remedy information asymmetries and limited attention, break down the cognitively complex task of engaging in academic tasks with small easy-to-achieve activities, provide encouragement, and reframe distal rewards to be more proximal. Though the evidence of information asymmetries is mixed (Avery and Kane 2004; Grodsky and Jones 2007; Hastings and Weinstein 2008; Valant and Loeb 2014), suboptimal behavior due to limited attention (Karlan et al. 2016), the cognitive complexity of tasks (Mullainathan and Thaler 2000), and time inconsistent preferences (DellaVigna 2009) is well established. Addressing these behavioral barriers with the READY4K! program yielded substantial literacy benefits. The program, implemented in the San Francisco Unified School District (SFUSD), increased takeup of home literacy activities and parental involvement in schools by approximately 20 percent of a standard deviation and increased some preliteracy test scores by approximately 10 percent of a standard deviation (York, Loeb, and Doss 2019).
Addressing behavioral barriers through texting has been applied to a range of levels of education. These interventions often include child-specific information to address information asymmetries and limited attention. On the K–12 level, Bergman (2016) used email, text messages, and phone calls to inform parents of their child’s missing assignments. The information given was student-specific and detailed, often containing specific class assignments and page numbers, and clearly personalized for specific parents and students. The intervention led to a 21 percent of a standard deviation increase in student GPA, a 25 percent increase in assignment completion, and a 28 percent decrease in classes missed. Kraft and Rogers (2015) used the same three mediums to establish weekly teacher–parent communication in the summer school context. In one treatment arm, teachers conveyed positive messages regarding their child’s behavior and academic performance. In another treatment arm, teachers highlighted areas where the child could improve. The authors found that this intervention increased the probability of passing the summer school class by 6.5 percentage points, a 41 percent reduction in failing the class. The results were driven mostly by the child-specific suggestions parents received on where the child could improve. Receiving positive information regarding child-specific successes produced positive, though imprecise, point estimates.
At the post-secondary level, Castleman and Page (2015) fielded an intervention to help ease the transition to college for new high school graduates. A text-messaging arm of the treatment sent differentiated and personalized reminders during the summer regarding deadlines for filling out the required paperwork to matriculate into college. The information in the messages was specific to the requirements of the college in which the student was accepted and planned to matriculate. Students received reminders to access important paperwork, register for orientation, register for placement tests, complete housing forms, and complete health insurance forms. A second treatment arm used in-person peer mentors that reached out to students directly to offer help in completing the required tasks. Both treatments increased college enrollment among students who had less access to college counseling during the academic year.
Though this line of literature is compelling, it is unclear which elements of the programs are driving the results. The current study seeks to test whether these texting programs are effective because they address limited attention through reminders (a “nudge”) or through the other behavioral barriers that require a greater interaction with the content, which is often personalized and differentiated to program participants. In this vein, we test whether differentiating and personalizing the READY4K! intervention increases (or decreases) program effectiveness. We personalize the text messages by providing information to parents about their child’s skill level, as measured by formative assessments already administered by the district. We differentiate the text messages by providing parents a literacy activity tailored roughly to their child’s skill level. A significant, differential effect of the personalized and differentiated version of the text messages will provide evidence that parents are actively engaged with the program content. The program can reduce the cognitive load inherent in parenting, provide novel information to parents, and address time-inconsistent preferences only if parents absorb the content of the messages. If they do not, then the positive effects of the texting programs are likely driven simply by nudges that hold attention.
Interaction with the content opens the door to the possibility that personalization and differentiation uniquely affects parent behavior. There are many different channels through which this effect can occur. The mere knowledge that the texts are tailored to a child may induce parents to engage with the texting program more regularly. To our knowledge, no study has tested to see whether personalization of interventions engenders more trust and fidelity to treatment from participants. However, behavioral economics has produced a robust line of literature that shows that how information is presented to people affects subsequent behavior. For example, the social norms literature shows that presenting someone with information on their peers’ behavior can lead to lower energy use (Allcott 2011) and increased savings (Kast et al. 2012), charitable giving (Frey and Meier 2004), and voter turnout (Gerber and Rogers 2009). The text messages in this experiment do not provide information about the behavior of the parents’ peers, but do provide information about the parents’ children. The close relationship between parents and children may amplify the effects seen in the larger social norms literature.
The information conveyed to parents about their children’s performance on formative assessments may also update inaccurate beliefs regarding their children’s ability. For example, parents may not realize their children are weak or strong on certain literacy skills and therefore fail to invest in their children’s development efficiently. In Malawi, Dizon-Ross (2017) illustrates that parents inefficiently invest in their child’s academic success due to inaccuracies in their perceptions of their child’s ability. The gap between perceived and actual ability is as large as one standard deviation. After receiving information specific to their children, parents began to invest in their children more efficiently. Their willingness to pay for remedial materials decreased as a function of their children’s revealed performance, and they more accurately picked textbooks that matched to their children’s ability. One year later, the higher performing children were more likely to transfer to better schools and were less likely to drop out, though lower performing siblings were hurt academically in the process.
Finally, the closer alignment of the difficulty of the task to the child’s skill may lead to a greater probability of success in carrying out the activities. This success may in turn produce a recursive feedback mechanism that encourages parents to continue with the program. Lower performing children who previously received tips that were too advanced for them may have failed at the activities, causing parents to disengage from the program. Advanced children who, in the counterfactual, receive activities that were too easy, may have gained little from the experience, causing parents to disengage from the program. Again, no study has specifically probed this mechanism.
Apart from inducing behavioral changes in parents, aligning the difficulty of the activity to the ability of the child may also produce differential academic gains. Education theorists posit that students advance in knowledge when taught concepts that are slightly beyond, but still close to, the student’s ability, a concept called the “zone of proximal development” (ZPD) (Vygotsky 1978a, 1978b). Traditionally, practitioners have used formative assessments to provide information on where a child’s ZPD lies and have grouped students by ability to tailor instruction and activities to students whose ZPD lie in approximately the same place. Many studies have shown that the use of formative assessments and data can improve the educational outcomes of children. In a meta-analysis of studies conducted between 1988 and 1998, Black and Wiliam (1998a, 1998b) find that the use of formative assessments can increase student performance by 40–70 percent of a standard deviation, with effects prominent for low-performing children. In kindergarten, benefits of using formative assessments have been seen in reading, math, and science outcomes (Bergen, Sladeczek, and Schwarz 1991). Ability grouping in the classroom context has also generally led to positive academic results for children by as much as a quarter of a standard deviation (Kulik and Kulik 1984, 1992; Robinson 2009). The effects, however, are not uniformly positive, with some evidence that lower performing children can be hurt through ability grouping (Lou et al. 1996).
With the rise of artificial intelligence, technology has been used to identify a child’s ZPD more efficiently so that instruction can be tailored to the child. As the software gathers information on the child’s ability, it tailors the program and activities to be more aligned to the child’s skill. As the child’s skill changes and grows, the software adapts the educational outputs accordingly. This process also provides teachers information that can be used to differentiate instruction. The evidence on these types of software is mixed. Van Klaveren, Vonk, and Cornelisz (2017) compare test score outcomes of Dutch children randomly assigned to either a static or adaptive practice technology and find no overall benefits to the adaptive technology and negative effects of about 8 percent of a standard deviation on higher performing children. Pane et al. (2014) analyze the efficacy of the Cognitive Tutor Algebra I program that provides a curriculum designed around an adaptive technology software. They found mixed results when comparing children in schools randomly assigned to the curriculum to children in traditional Algebra I classrooms. Positive results appear for high-schoolers in the second year of the experiment. These two studies were included in a recent review of educational technologies by Escueta et al. (2017). Across 29 studies, she finds a similar mix of results, with some interventions providing large benefits to children, and others producing null results.
To the extent that the differentiation and personalization of text messages produce differences in outcomes, we will be unable to disentangle the effect of differentiation from personalization. We will also be unable specifically test for the behavioral mechanisms through which differentiation and personalization can act. However, indirect evidence on this latter point can be culled by analyzing responses on parent survey items. We also assess whether the intervention was more successful for students of average baseline ability or for students who started at the tails of the baseline ability where differentiation was greatest, potentially distinguishing the effects of personalization engendering a feeling of familiarity from the other potential mechanisms.
III. The Intervention
This study is an extension of the READY4K! intervention run in conjunction with the San Francisco Unified School District (SFUSD) starting in the 2013–2014 school year. In the original program, treatment families received three texts per week. The “FACT” text was sent on Mondays and informed families of the skill of the week and the importance of that skill for the academic growth of their child. On Wednesdays families received a “TIP” message that suggested a home literacy activity based on that skill. These literacy activities were meant to fit as seamlessly as possible into the parents’ day and to capitalize on items and materials found in their home and neighborhood. These “TIP” texts aimed to provide an easy choice to parents and thus reduce the cognitive load inherent in parenting that stems from making multiple and ambiguous choices. Finally, on Fridays, families received a “GROWTH” text that contained a more advanced activity that was meant to extend the learning opportunity presented earlier in the week, as well as encouragement aimed to provide some immediate gratification. Control families received one text every two weeks that contained general district information and did not promote parent-child interactions. The eight-month long program touched on a variety of preliteracy skills, such as letter recognition, letter sounds, rhyming, and early literacy behaviors. Participants could choose to receive the texts in English, Spanish, or Chinese (York, Loeb, and Doss 2019). For this study, we built on the original READY4K! format of FACT/TIP/GROWTH but created new texts to match the skills covered in kindergarten. We created both a generic version and a differentiated and personalized version targeted to students’ developmental level.
To field the study, we followed the first cohort from the original experiment into their kindergarten year and recruited more of their kindergarten peers. The original participants in the control condition remained in the control condition in the second year. The original participants in the treatment condition were rerandomized to either continue receiving general literacy texts or to receive differentiated and personalized literacy texts. Newly recruited participants were randomized to receive either general texts, differentiated and personalized texts, or control texts. To keep the proportion of families treated the same in each cohort, half the new participants were randomized to receive control texts, and half were randomized into the two treatment arms.
To recruit new participants in the study, we worked with parent liaisons in each elementary school. In August 2014, we provided a brief training to liaisons to explain the study, its purpose, and provide materials with which to recruit families. Through their regular course of business, liaisons recruited families to participate in the study.1 Families that consented to participate completed a baseline survey to elucidate their home literacy habits and the skill level of their children on a variety of literacy skills. We used some of the same questions from the baseline survey in the original year so that we could pool answers between cohorts and use the responses as covariates in an effort to increase the precision of our estimates. As an incentive, liaisons were paid $10 for every family they recruited into the study. Participants in both the treatment and control conditions were paid $10 a month as long as they remained in the program, with the aim of covering texting costs for parents without unlimited texting plans.
We began texting at the end of October 2014 and continued for ten months. We used fall first grade literacy assessments as the primary outcome of interest. Details regarding the three randomized conditions are as follows:
1. Differentiated and personalized text treatment
Treatment followed the same general design as the first year of the experiment. Families received three texts a week: a “FACT” text on Mondays, a “TIP” text on Wednesdays, and a “GROWTH” text on Fridays. Only the TIP and GROWTH texts were differentiated and personalized using child-level formative assessment data on skills that corresponded to the week’s topic. The literacy texts reviewed skills from prekindergarten, such as letters, letter sounds, and rhyming. Then they eased parents into asking their child to read and helped parents teach their children to read with greater accuracy and comprehension. Figure 1 presents the differentiated and personalized versions of the texts (see Figure A1 in the Online Appendix for additional examples, http://jhr.uwpress.org/).
Text Examples
We insert two pieces of information in the TIP texts. First, we personalize the texts by giving parents an indication as to where their child falls in the distribution of skills. As seen on Figure 1 we indicate that the child is “beginning” to learn the skill, “growing” in their knowledge of the skill, has a “solid” understanding, or has a “strong” knowledge of the skill. We positively framed each text so that parents of children on the lower end of the distribution would not become frustrated. This framing is akin to an “injuctive norm” in the behavioral economics literature. Additionally, the texts are differentiated such that parents receive one of four different activities based on their child’s prior academic information. At first we used parental responses from the baseline surveys, and once available, we switched to data from the fall, winter, and spring administrations of the Fountas and Pinnell Benchmark Assessment System (BAS) administered by the child’s kindergarten teacher. We identified the relevant skill for each week as measured by the BAS and divided the skill’s scale into four equal intervals. Students scoring in each interval received different TIP and GROWTH texts. Those on the lower end of the distribution received easier versions of the TIP and GROWTH, while those at the upper end of the distribution received more advanced versions. A child was not necessarily in the same category each week because a child may be weaker on one skill, but stronger in another.2 The information in the TIP text often would not fit into one text. In those cases, families received two texts on Wednesday, one right after the other. As a result, families in this condition received one extra text message per week, though the timing and spacing of texts was very similar across treatment groups.3
2. General text treatment
The families randomized into the general text treatment also received FACT, TIP, and GROWTH texts each week. The FACT texts were identical to those received in the differentiated and personalized text condition. The TIP and GROWTH texts, however, did not include the strength of their children on the particular skill, and every family in this condition received the same activity. The activity was most often similar to, if not identical to, the activity given to families in one of the middle two groups in the differentiated and personalized text treatment. This treatment condition is directly analogous to the original texting experiment. Figures 1 and A1 (in the Online Appendix) give examples of the general texts.
3. Control text condition
Families in the control condition received one text, every other week, with information about the school district. The two examples presented in Figure 1 provide information on emergency preparedness and on how the food in SFUSD is prepared.
For all conditions, parents could choose to receive the texts in English, Spanish, or Chinese.
IV. Data and Empirical Strategy
A. Data and Sample
The initial sample included 504 children and families from the original experiment and 290 newly recruited children and families. These 794 students were randomized into one of the three conditions and received texts from October 2014 through August 2015. We collected three primary sources of outcome data on these children. In May 2015, we surveyed the kindergarten teachers of all the children in the study. We asked questions regarding how well the teacher knew the parents of the children, how often parents talked to the teacher, how often parents asked questions regarding specific academic skills, and how well the child performed on specific academic skills. Teachers were not informed of the treatment status of individual children so as not to bias the results. Each teacher received $50 for completing the survey.
In September 2015, after texting was complete, we sent parents enrolled in the program a post-survey. We asked questions regarding their attitudes towards building literacy skills in their children, how often they engaged in specific learning activities with their children, how often they interacted with their children’s teacher, and how they viewed the texts they received. We also compensated parents $50 for completing the survey.
Finally, we use the fall first grade administration of the Fountas and Pinnell Benmark Assessment System (BAS) as a measure of children’s academic skills. The BAS is a formative assessment tool that has been shown to be a valid assessment of literacy development in children (Fountas and Pinnell 2012). Teachers first assess the ability of children to recognize upper-case and lower-case letters, letter sounds, initial word sounds, 25 high frequency words, as well as to rhyme, blend sounds into words, and demonstrate early literacy behaviors. After mastering six of the eight foundational skills, children are asked to read books of increasing difficulty. The teacher begins with the easiest books, Level A. After the child reads with sufficient accuracy and comprehension, they move on to harder books (Levels B–Z). A teacher stops after reaching a book that the child cannot read with sufficient accuracy and comprehension. In kindergarten, most children are still mastering foundational skills, while in first grade, the vast majority of children are reading books of varying difficulty. The texts are therefore primarily differentiated based on a child’s performance on the eight foundational skills listed above. The outcome of interest is whether children are reading more complex books in the fall of first grade and whether they reach development benchmarks set forth by Fountas and Pinnell.
Of the 794 participating families, teachers provided information on 442 (56% response rate) students, 519 families responded to the survey (65% response rate), and 641 students completed the fall first grade BAS (81% assessment rate). The 153 students who do not have assessment data left the district. This level of mobility in the early grades is not uncommon. Only 28 children that we recruited in the beginning of the year left during the year or in transition to first grade. The remaining 125 children are from the original cohort recruited during SFUSD’s prekindergarten enrollment process. These students left the district between enrolling for prekindergarten and transitioning to first grade. To obtain the final analytical sample, we restrict the sample to those parents who answered enough baseline survey questions to construct three measures of their pretreatment characteristics. The baseline survey was designed to measure three constructs: baseline child skills, baseline frequency of literacy activities in the home, and background characteristics of the parent. To get a measure of each construct, we estimated a graded response model separately on each subsection of the survey. Graded response models (GRMs) are used frequently in survey analysis with Likert-type items, and provide an estimate for all respondents of where they fall along the construct of interest, termed their “ability” estimate. We selected GRMs over factor analysis due to their ability to produce estimates in the presence of missing data, avoiding imputation (Samejima 1997). We use these ability estimates, rather than the individual questions, as control variables in our analyses.
In the end, we have three analytic samples. The final parent survey sample consists of 475 families, the teacher survey sample consists of 409 children, and the BAS sample consists of 578 children. We check to ensure that attrition and pretreatment covariates remain balanced in all samples. Finally, we merge these data with district administrative data on student background characteristics such gender, ethnicity, and date of birth.
Table 1 presents the descriptive statistics on each analytic sample. Though there are slight differences among samples, the demographics are largely similar. Panel A presents the characteristics of children in the sample. Looking at the academic outcome sample, 51% of the children are male. The two largest ethnicities are Hispanic (35%) and Asian (33%), with fewer white children (7%) and children from other ethnicities (19%). The average age in the sample is 5.4 years old. At baseline parents rated their children 3 out of 4 in letter knowledge and a 3 out of 5 in letter sounds and rhyming, on average. In comparison, the broader SFUSD kindergarten cohort has more white students (14 percent) and fewer Hispanic and Asian students (27 percent and 23 percent, respectively). Both samples, however, have approximately the same proportion of males, students from other ethnicities, and students of approximately the same age.
Descriptive Statistics
Panel B presents descriptive statistics on the parents. Most have less than a bachelor’s degree (65%) and are on average 34 years old. More than half (53%) chose to receive the text messages in English, with fewer choosing to receive them in Spanish (26%) and Chinese (22%). A little less than half the sample (42%) are new to the program this year. On average, parents rated themselves between 2.8 and 3 out of 4 when asked how frequently they engage in literacy activities with the child. The texting program primarily served non-white and lower-income families.
B. Empirical Strategy
We use the following model to estimate the effect of the texting program on student and family outcomes:
(1)
In Equation 1 we regress an outcome Yis for student, i, in school, s, on an indicator for receiving the general literacy texts GeneralTextis, an indicator for receiving differentiated and personalized texts PersonalizedTextis, a vector of baseline characteristics Xis, and a school fixed effect αs. εis is a stochastic error term. Xis contains an indicator for receiving the texts in English, Spanish, or Chinese, as well as the child’s gender, ethnicity, age in years, and factors of baseline survey questions on literacy skills and rates of home literacy activities. Randomization occurred within school site, and the school fixed effect, αs, is the school site where randomization took place. For children in their second year of the experiment, this is their prekindergarten school site, and for children in their first year of the experiment, this is their kindergarten school site. First or second year status does not vary within randomization school sites. We therefore do not include an indicator for being new to the experiment in Xis. The coefficients of interest are β1 and β2, which provide estimates of the effect of receiving general and personalized and differentiated texts, respectively, on the outcome of interest. The omitted group in this case is the control group. We cluster all standard errors at the randomization site level.
In supplementary analyses, we replace the indicator for receiving general texts with an indicator for receiving any literacy text, AnyLiteracyTextis, which is equal to one for students in either the general text treatment or the differentiated and personalized text treatment. All other elements of the equation remain the same. In this specification, β2 provides an estimate of the effect of the personalization and differentiation, relative to the effect of receiving general text messages. In an effort to be parsimonious, we do not present the results of this model. Its main advantage is that it indicates whether the difference in effects between the general text messages and the personalized and differentiated text messages are statistically significant. We reference the significance in the body of the paper when relevant.4
The outcomes, Yis, are the individual teacher and parent survey questions and the reading level of the child as measured by the BAS. To reduce the number of outcomes from the surveys we use exploratory factor analysis to determine which questions measure the same underlying construct. The questions in the parent survey load onto four separate factors: (i) a parent belief factor regarding the ease of building literacy skills and the support they feel in building those skills, (ii) a literacy activity factor capturing the frequency with which the parents engage in literacy activities with their child, (iii) a teacher factor regarding the frequency with which parents interact with their child’s teacher, and (iv) a text factor regarding parental attitudes to the texting program. In creating the final factors, we used principal components analysis and rotated the loading matrix to create orthogonal factors. For the teacher survey, we took the analogous questions from the parent survey and created a teacher version of that factor, so that the two are directly comparable. Table A1 in the Online Appendix presents each question contained in each of these factors and the weighting of the elements variables for each of these factors.
C. Randomization Checks
The covariates are largely balanced between treatment and control for each analytic sample. Table A2 in the Online Appendix presents these results for 14 covariates tested in each of the three samples, for a total of 42 tests. No variable in the parent survey sample was significantly unbalanced. In the teacher survey sample, one variable was unbalanced at the 10 percent level (male), and one variable was unbalanced at the 5 percent level (white). In the academic sample, one variable was unbalanced at the 1 percent level (white). The rate of imbalance is about what one would expect to occur by chance in the parent survey and academic samples, but it is a little higher in the teacher survey sample. All our main specifications include covariates, and we present all results with and without covariates. For all outcomes, addition of the covariates does little to change the point estimates and ultimately does not change our inferences, providing an indication that imbalance is not a concern in this study.
We also test whether students differentially left the analytic samples. Table 2 shows that, overall, we do not find evidence that students differentially attritted from the parent survey sample or the academic sample. Attrition is marginally significant for the personalization and differentiation treatment arm in the teacher sample (−7.5 percentage points). We further check to see whether measurably different children left the sample. Appendix Table A3 in the Online Appendix shows no imbalance in the academic and parent survey samples, but older students are marginally less likely to attrit from the general texting group in teacher survey sample. Because the point estimates on the overall attrition is larger (and marginally significant) and the attrition by age is marginally imbalanced in the teacher sample, we implement Lee (2009) style bounds on the teacher survey sample as a robustness check.
Overall Attrition
V. Main Results
Tables 3–5 present the main results of the intervention and show that the differentiated and personalized texts had positive effects. Table 3 presents the results on the fall first grade Fountas and Pinnell BAS. Panel A shows the effects of the program on the reading level of children, with Level A being the easiest book and Level Z being hardest book. A small minority of children (8%) were not yet reading. We analyze the results in three ways. First, we capitalize on the ordinal nature of the reading scale and use ordinal logit models. To aid the interpretability of the results we create a standardized, linear scale from the reading levels and present results as effect sizes. We also present the results of linear probability models that show the effect on the probability of reading at Level A, C, E, or G and above. Level A indicates that the child is first able to read, and Levels C, E, and G represent the 25th, 50th, and 75th percentiles of the reading distribution. Panel B presents the effects of the program on the probability of meeting district benchmarks. These benchmarks track the Fountas and Pinnell recommended benchmarks. Levels C, D, and E are the cutoffs for approaches, meets, and exceeds expectations.
Effects on Fountas and Pinnell Benchmark Assessment System Academic Outcomes
Effects on Parent Beliefs and Home Activities
Effects on Parent Interactions with Teachers
The academic results indicate that differentiated and personalized text messages had a significant effect on the reading ability of children, while the general texts did not. Children whose parents received the differentiated and personalized texts messages were 63 percent more likely than the control group to move up a reading level (p < 0.001). This estimate translates to an 18 percent of standard deviation increase in reading level (p < 0.05). These children were also 8.84 percentage points (p < 0.05) more likely to read at Level E or above, were 12.05 percentage points (p < 0.001) more likely to exceed expectations, and 9.00 percentage points (p < 0.01) more likely to meet or exceed expectations.
Tables 4 and 5 present the survey results, which give clues to the mechanisms underlying the academic results. We present the results of the factors of survey questions in Panel A, as well as the results of individual questions in Panels B and C. Table A1 in the Online Appendix gives all questions that compose each factor. Table 4 shows that overall the texting program had limited effects on parent beliefs towards activities and building academic skills in their children. The program had the greatest effects on parent ratings of how easy it is to build literacy skills in their children. The general texting treatment caused a marginally significant 27 percent of a standard deviation reduction in parent ratings on the extent to which they feel building literacy skills is easy. The differentiated and personalized intervention significantly mitigated these negative effects. Parents in the personalized treatment group responded 32 percent of a standard deviation (p < 0.05) more positively than parents in the general treatment group. These results are consistent with the notion that knowledge of a child’s skill level, with an appropriately differentiated activity, can positively affect parent beliefs. If general text messages were unaligned with a child’s skill level and too hard for parents, they could cause parents to believe that building literacy skills is difficult.
The program had a stronger effect on the frequency with which parents engaged in home literacy activities. Panel C of Table 4 shows that the greatest effects are in reading words with children, taking books when leaving the house, reviewing parts of a book, reviewing the direction of reading, correcting mistakes while reading, and practicing rhyming, with effect sizes ranging from 24–37 percent of a standard deviation. Differentiation and personalization drove some of these results. When combining all activities questions into a factor, Panel A of Table 4 shows that that general texts had a positive, but insignificant point estimate of 14 percent of a standard deviation and that personalized texts had a significant 26 percent of a standard deviation (p < 0.05) effect on home activities compared to the control group. We do not have the power to separate a differentiation and personalization effect from a base texting effect for this composite variable.
Table 5 presents the results of the intervention on parent involvement at school both from the parent perspective (Panel B) and the teacher perspective (Panel C). From the parent perspective, the largest effect is seen on how well they know their child’s teacher and how often they inquire about how their child is getting along with other children. Columns 3 and 4 indicate that, compared to the control group, the general texts increased each of those two dimensions by 26 percent of a standard deviation (p < 0.05), but the personalized texts did not have a significant effect. The differentiated and personalization aspect of the treatment marginally significantly decreased the positive effect on knowing the teacher generated by the general texts by 21 percent of a standard deviation when compared to the general text message group. The remainder of the estimates shows that the general texts increased specific questions parents asked teachers by about 10–20 percent of a standard deviation. The frequency with which parents talk to teachers about their child’s interests and literacy skills reach marginal significance. The point estimates on the personalized and differentiated treatment in Table 5 are generally of equal magnitude or smaller than the general texts. Only one question, the frequency with which parents ask how well their child is doing in school reaches marginal significance, with a point estimate of 17 percent of a standard deviation. When combining these measures into one factor in Panel A, the pattern remains. General texts have a larger effect on teacher interactions of 23 percent of standard deviation (p < 0.10), while personalized texts had a smaller, insignificant effect of 11 percent of a standard deviation.
The results are fairly consistent when analyzing the same questions from the teacher perspective. Column 3 of Panel C in Table 5 shows that the general text treatment had positive effects on parents talking to their child’s teacher about their child’s interest, literacy skills, and home activities, with effect sizes of 23 percent of a standard deviation (p < 0.10), 25 percent of a standard deviation (p < 0.10) and 28 percent of a standard deviation (p < 0.05), respectively. The remainder of the point estimates are generally positive, but not significant. Column 4, however, shows that the point estimates on the personalized texts are negative, with the effect on teachers knowing the parents reaching a marginally significant −26 percent of standard deviation. The differentiation and personalization texts lead to an estimated 32–43 percent of a standard deviation less positive effect for many questions when compared to the general text messages. This negative effect is also seen in the composite of the teacher reports in Panel A. The point estimates indicate that the personalization of the texts may have induced parents to talk to teachers less when compared with the general text messages and thus mute any gains in teacher interactions generated by the general text messages. These results, as well as the results from the parent questions, are plausible if the greater amount of information regarding the child skill level, in combination with greater success in implementing the differentiated literacy activity, produced less of an incentive to talk to the teacher regarding how their child is progressing in school.5
Despite the potential decreases in interaction with the school, the greater extent to which parents engaged in activities in the personalization and differentiation group likely led to the academic advantages seen in Table 3. Unclear, however, is whether the knowledge that the texts were personalized engendered a greater fidelity to the program, whether updating parent beliefs about the child’s skill caused a more efficient allocation of resources to the child, or whether a closer match between the difficulty of the texts and the skill level of the child led to greater success in carrying out the tips and established a positive, recursive feedback mechanism that encouraged parents to continue to engage in the texts. Parents in both groups may have interacted with the activities in similar ways, and the closer alignment of the differentiated text to the child itself may have been the sole cause of the increased academic skills.
Though we cannot definitively identify which mechanisms are at play, responses to questions that elicited parental attitudes towards the text messages can provide some clues. Table 6 provides these results. We asked parents the extent to which they thought the texts were made for them and their children. Interestingly, there is little difference in response between the two groups that received the treatment texts. Both sets of parents reported the texts were made for their children to a much higher degree than the control group, with effect sizes around 40 percent of a standard deviation (p < 0.001). Personalized and differentiated text messages did not elicit a greater response on this dimension. These results indicate that parents in the personalization and differentiation group did not see the text messages as more tailored to their child. Without this realization, personalization likely did not induce parents to adhere to the program more faithfully, nor were they likely to allocate resources more efficiently to their children after receiving information on their child’s skill level.6
Effects on Parental Attitudes towards Texts
We see greater differences in point estimates when we asked parents to what extent they used the texts, thought the texts were helpful, and would recommend the texts. In the case of using and recommending the texts, the personalized and differentiated texts had a 29 and 23 percent of a standard deviation effect, respectively (p < 0.05), while the general text messages had positive, but insignificant effect. In the case of the texts being helpful, both groups reported a significant effect compared to the control group. For each of those three questions the personalized and differentiated group effects were about 15 percent of a standard deviation higher than the general texting group, though we do not have the power to determine if this difference is significant. While we must be extremely cautious in interpreting insignificant differences, these results do not eliminate the possibility that a greater success in executing the activities in the personalized and differentiated text messages caused parents to engage with the program more faithfully.
VI. Heterogeneity of Results
Prior research on social information experiments indicates that the effects of such interventions can vary significantly by baseline characteristics. Allcott (2011) demonstrates that providing families with information on their neighbor’s energy usage will, on average, decrease their own energy use. Perhaps predictably, the effects are concentrated on the highest pre-intervention energy users, with no effects seen on the lowest pre-intervention energy users. Gerber and Rogers (2009) illustrate that presenting voters with a script that frames an upcoming election as a “high turnout” election will, on average, induce people to vote more compared to a script that frames the election as a “low turnout” election. They present evidence that the intervention was more effective for participants who voted less frequently in prior elections. Beshears et al. (2015) present a more nuanced result and show that the same intervention can have opposite effects depending on where participants fell in the baseline distribution. Their intervention provided social behavior regarding 401(k) savings and found that the intervention encouraged those who were previously contributing at high rates to save more, but discouraged those who were not previously saving much from contributing to their plans.
We analyze heterogeneity by the baseline skill distribution.7,8 Specifically, we estimate the effects of the intervention separately on students who fall in the middle two quartiles of the baseline skill distribution and on students who fall in the tails of the distribution. To do this analysis we must restrict the sample to those families who are new to the program. York, Loeb, and Doss (2019) showed that texting in the first year positively affected preliteracy skills. Because we are retaining the control group in this analysis, there will be a positive correlation between fall kindergarten test scores and texting treatment status for those families in the second year of the program.
Previous results provide some evidence that the differentiation of the texts is driving the results. If this were the case, one would expect the effects to be concentrated on the tails of the distribution where differentiation is greatest. General texts most often corresponded to texts sent to either the second or third quartile. It is also possible that parents at the tails of the distribution respond to the personalization more strongly. In this case, the effects are unclear ex ante. On one hand the personalization may have greater effects on the tails of the distribution if parents are particularly motivated by signals that their child is doing relatively well or poorly. Similarly, the intervention could have smaller, or negative, effects at tails of the distribution if parents on the low end of the distribution are discouraged by the knowledge that their child is doing relatively poorly, or if parents at the top of the distribution feel less compelled to engage in the activities after learning their child is already advanced.
Table 7 shows the results of the heterogeneity analysis on the academic outcomes.9,10 Panel A presents effects on the middle two quartiles, and Panel B presents the effects on the first and fourth quartiles. Column 3 of Panel A in Table 7 shows that there is a marginally statistically significant effect of the general texts on the probability of meeting or exceeding expectations of 17.13 percentage points (p < 0.10). Differentiation and personalization, however, produce no differential effect, with a quantitatively similar coefficient in Column 4. Meanwhile neither treatment texting intervention had a detectable effect on the probability of approaching expectations or exceeding expectations. The ordinal logit model is imprecisely estimated. The results are quite different in Panel B, which presents results for families whose children are in the first and fourth quartiles of baseline academic skills. Column 3 and 4 indicate that the general texts had no effect on the academic skills of the children, but differentiation and personalization had a large effect on the probability of exceeding expectations and on the ordinal logit. Differentiation and personalization caused students to be 2.5 times more likely to move up a reading level and increased the probability of exceeding expectations by 21.5 percentage points (p < 0.05).
Heterogeneity of Academic Outcomes by Baseline Academic Performance (First Year of Experiment Only)
These results support the previously presented evidence that differentiation is a driver of the results. Any positive feedback mechanism caused by a greater probability of success with the texts is more likely to occur in the tails where differentiation is the greatest. Also possible, however, is the fact that personalization may have been differentially effective for families in the tails of the baseline skill distribution. If parents did not previously realize their child was performing relatively poorly, the new information may have spurred them to adhere to the program more faithfully. Similarly, receiving positive feedback on their child’s performance may have encouraged parents at the top of the distribution to build on that success by engaging in the texts to a greater extent.
VII. Comparisons with the First Year of the Program
Although there is ample evidence that the personalization and differentiation of the texts provided academic benefits above the general texting curriculum and the control group, the general text-messaging curriculum produced no discernable benefits. Table A4 in the Online Appendix indicates this is the case for both cohorts of participants in the program. At first, this may seem inconsistent with the results from the prekindergarten experiment, which estimated that a general texting curriculum can increase preliteracy skills by 10 percent of a standard deviation (York, Loeb, and Doss 2019). A deeper look at the results shows that the first cohort of students in the prekindergarten experiment, which subsequently participated in this follow-up experiment, saw no significant effect of the general program on preliteracy skills. The general effects on the full population are driven by the second cohort of students that received a combination program.11 However, both cohorts of the prekindergarten experiment saw large gains for students below the baseline prekindergarten skills distribution. In both years, those children experienced a 30 percent of a standard deviation increase in preliteracy skills. This result provides evidence that general texting programs may be most effective for weaker students.
To probe this question with a direct comparison, we take the students that we followed from the prekindergarten experiment and look at the fall first grade outcomes by baseline of the prekindergarten skills distribution. Sample sizes limit any firm conclusions, but the pattern of point estimates indicates that the general texting program may have been more effective for students below the median of skills. Panel A of Table 8 shows that point estimates for the general text condition are large and equal in magnitude to the personalized and differentiated condition. Panel B shows that for students above the median, the point estimates for the general texting condition are zero or even negative. Again, one must be very cautious in interpreting insignificant point estimates, but the pattern is consistent with the notion that general text-messaging curricula benefit weaker students at baseline. Table 7 shows that for students at the tails of the distribution, the effect of personalization and differentiation is largest for the probability of exceeding expectations. This finding further suggests that general texts may be too easy for advanced students, and the differentiation ameliorated this mismatch. This evidence, however, is purely suggestive, and we cannot preclude the possibility that elements of the prekindergarten program did not replicate. To that end we support replication of the results by providing the text messages used in the prekindergarten experiment and this follow-up study.
Effects Academic Skills on Second Year Participants, by Median of Prekindergarten Baseline Skills Distribution
VIII. Robustness Checks
One threat to internal validity of a randomized control trial is differential attrition between treatment and control groups. If different types of people are attritting from each condition, our results could be biased. Table 2 presents the overall probability of attrition in each of our models. The probability of attrition is not significantly different in the academic and parent survey samples and marginally significant in the teacher survey sample. We further assessed if there was differential attrition status by covariate and found some imbalance in the teacher survey sample (Table A3 in the online appendix). Because the teacher survey has the greatest amount of attrition, we engage in a Lee (2009) style bounding exercise for that sample of students. Point estimates indicate that fewer people attritted from the two treatment groups. We therefore calculate a trimming proportion, p, for each treatment arm, compared to the control group. We then trim each treatment arm at their respective pth and 1 – pth quantile. Rerunning our models on these trimmed samples will provide our upper and lower bounds, respectively.
Table 9 presents the results of this bounding exercise. Column 1 and 2 present the original estimates from Columns 3 and 4 of Panel C of Table 5. Comparing the original and upper bound estimates, little changes. The effects on the base text treatment remain positive, become slightly larger in magnitude, and become more significant. Coefficients on the differentiated and personalized text messages generally become slightly more positive (or less negative), but their lack of significance remains. In our lower bound estimates, all point estimates become predictably more negative. Point estimates from the general text-messaging arm become insignificant and near zero or slightly negative. Estimates for the differentiated and personalized arm become more negative, and in some cases, significant.
Bounds on Teacher Survey Outcomes
Recall our general conclusion was that there was evidence that the general texting treatment increased parent–teacher contact, but that the differentiation and personalization treatment arm mitigated that effect. The upper bound estimates provide more robust evidence for this inference, while the lower bound estimates indicate that, at worst, the general texts did not affect parent–teacher interaction, and the differentiated and personalized text messages may have significantly decreased interactions. Importantly, the effect of differentiation and personalization relative to the general texting program remains the same in all three estimates. Our overall conclusion therefore remains the same: relative to the general texting program, differentiation and personalization resulted in less parent–teacher contact. This substitution may be due to the greater amount of information contained in the differentiated and personalized texts. The general texts most likely had a positive effect on these interactions, though in our most extreme robustness checks they could have had null results.
IX. Discussion and Conclusions
In this study, we demonstrate that a low-cost personalized literacy texting intervention can substantially affect student academic outcomes above and beyond a general texting program. Specifically, the differentiation and personalization of the messages caused children to be 63 percent more likely to move up a reading level. Tailoring instruction based on formative assessments has previously been associated with increased student learning in K–12 classrooms (Kulik and Kulik 1984, 1992; Bergen, Sladeczek, Schwarz 1991; Black and Wiliam 1998a, 1998b), but this is the first study to show that this approach can also improve parent–child academic interactions. Further, this study provides evidence that text-messaging interventions can do more than merely maintain parents’ attention or “nudge” behaviors via reminders. The significant effects of personalization on parent and student outcomes indicate that parents interact with, and absorb the content of the messages as well. This finding supports our hypothesis that the original READY4K! program was effective because it took the complex task of parenting and broke it down into small and easy tasks that were meant to fit into daily life and capitalize on everyday objects.
There are several mechanisms through which the additional gains seen in this study could have been realized. We hypothesized that personalization aspect of the texts could have engendered more trust with the program which would lead to a greater uptake in the activities and thus greater gains in literacy outcomes. Also possible is that information on child performance embedded in the texts updated parental perception of their children’s ability and allowed them to more efficiently allocate resources. Meanwhile, the differentiation of the messages helped to better align the difficulty of the task with the child’s developmental ability, thus increasing the chance that a parent could successfully engage in the activity with their child. This success may also encourage parents to persist in the program. Finally, the behavior of the parent could have stayed constant and the greater match between text difficulty and student ability could have led to the achievement gains.
Though we cannot definitively pinpoint which mechanisms are at play, survey results indicated that parents in the two treatment text groups saw the texts as equally tailored to their children. This result may indicate that they did not overtly recognize the personalization aspect of the texts, precluding the possibility that personalization engendered more fidelity to the program or that the parents updated their perceptions of their child skill level. More likely, the differentiation itself led to better outcomes and parents may have had more success in enacting the differentiated activities, leading to a positive feedback loop.
This hypothesis is further supported by our heterogeneity analysis, which indicates that the effects of the differentiation and personalization were particularly concentrated at the tails of the baseline skills distribution where differentiation of the texts was the greatest. There is some evidence that this differentiation was particularly helpful for the higher performing children, as the strongest effects were seen in the probability of exceeding expectations. One may be concerned that the program may increase disparities if those who were higher at baseline benefit the most from this program. Children in the higher quartiles come from more advantaged families. However, the initial study provided evidence that general text messaging, with perhaps easier activities, is more effective for children with weaker skills at baseline and the differentiation ameliorated any mismatch between the difficulty of the texts and the skills of the more advanced children. York, Loeb, and Doss (2019) show that the general preliteracy text-messaging program had stronger effects for children below the median of the skills distribution. We find suggestive evidence that this occurred in this follow-up experiment as well. The personalization and differentiation may merely improve an intervention that was not previously serving the more academically advanced children in the sample.
The parent and teacher surveys provide additional clues as to how the program changed parent behavior. Parent survey results indicate that recipients of the general text messages thought it was harder to build literacy skills in their children. If the program successfully caused parents to engage in literacy based activities with their child, it is possible that parents realize how hard it is to build academic skills in their children, particularly if the activity and the child’s skill level are mismatched. The differentiated and personalized texts successfully mitigated this negative effect, indicating that the differentiating of the texts may have indeed aligned the child’s skill to the activity. The one unanticipated result, however, is that parents in the differentiated and personalized group appear to have substituted away from engaging with teachers.
These results highlight that programs that break down complex tasks, such as building skills in children, can be effective and produce positive outcomes, but that a mismatch between the difficulty of the task and the ability of the parent and child to carry out that task can attenuate any potential gains. Differentiation and personalization of these programs can extract larger gains by minimizing these mismatches, and we demonstrate that even “light-touch” differentiation and personalization based on extant data can generate these gains. The ease and ubiquity of text messaging make it a nimble medium through which educational stakeholders can deliver this differentiated and personalized interventions that minimize frictions caused by mismatch.
Scaling the intervention can be particularly cost effective. The only additional costs over a base texting program are the costs of organizing students into groups according to formative assessment results, the costs of differentiating the activities, and the cost of one sending one extra text message. With the use of current technology we can automate the assignment of students to groups, such that the per-family cost of differentiating the tip tends towards zero as more families are added to the program. The only cost that grows with membership is the texting cost, which is very small compared to other interventions. Overall, differentiating and personalizing text-message interventions based on formative assessment has the promise to produce additional education gains with relatively little additional costs.
Footnotes
* Supplementary materials are freely available online at: http://uwpress.wisc.edu/journals/journals/jhr-supplementary.html
↵1. The job of a parent liaison is to facilitate communication between families and the school. One responsibility of parent liaisons is to coordinate school services to students and parents and refer families to school resources. It is during this process that parent liaisons recruited parents for the study. If a parent liaison talked to a parent in the context of coordinating or referring services, they informed the parent of the program. If the parent consented, they then completed the consent form and baseline survey.
↵2. While we explicitly informed parents that the tip was based on their child’s formative assessment performance, we did not explicitly indicate to parents that “beginning,” “growing,” “solid,” and “advanced” were terms that indicated a child’s performance along a continuum of skill levels. Parents could have deduced the implied meaning of these words if their child fell in different groups over time, across skills. This was the case for almost all parents. Only one child was consistently in a group throughout the experiment.
↵3. There may be an effect of receiving one extra text per week, in addition to the personalization and differentiation. In other work, we are explicitly testing the effect of receiving more tips during the week. In one treatment arm recipients received a FACT/TIP/GROWTH program akin to the general program in this study. In another treatment arm recipients received a FACT/TIP/TIP/TIP/GROWTH program. Preliminary results show no differences in outcomes, but survey responses were slightly less positive for the group that received five texts. These results indicate that the extra text received in this outcome likely did not affect academic outcomes and may have slightly attenuated survey results (Cortes et al. 2017). Results available upon request.
↵4. Results are available upon request.
↵5. The more positive reports of teacher interactions from parents of the personalized and differentiated group when compared to the teacher reports on the same questions may indicate that social desirability bias is greater for personalized and differentiated text recipients. While this is a possibility, this phenomenon would further support the assertion that parents are interacting with the content of the texts. In addition, the larger effects of the personalized and differentiated texts on academic outcomes indicate that larger survey point estimates are not completely driven by social desirability bias, and parents are, in fact, engaging in activities to a greater extent compared to the general texting group.
↵6. There is no effect of the program on the rate with which the texts were read, indicating that all parents received the texts and even the control group read the texts.
↵7. We also have analyzed the results by texting language. Splitting the sample into three languages greatly reduces the power to detect effects. Generally, the children of the parents receiving texts in Chinese saw the greatest academic gains. Results available upon request.
↵8. We find little heterogeneity in academic outcomes by year of participation in the program. Both sets of families benefited to about the same extent, with differentiation and personalization driving the results. Effects are slightly higher for first-year families, but are not significantly different than effects on returning families. If the first year participants saw larger effects, this could be a result of texting fatigue in second-year participants. If this program was rolled out to a new, comparable population of recipients, we might expect to see slightly larger effects. However, because the inferences remain stable between years, we would expect personalization and differentiation to provide benefits regardless of whether participants received texts in the previous year. Academic results by length of time in the program are shown in Table A4 in the Online Appendix.
↵9. Sample sizes are too small to draw conclusions from the parent and teacher survey samples.
↵10. Table A5 in the Online Appendix presents descriptive statistics on the sample by quartile. Families of children in the lower quartiles were more likely to be Hispanic and less likely to be Asian and white. Parents rated their children lower on baseline skills and reported engaging in literacy activities less frequently in the home. They were also less educated.
↵11. We hypothesize that the combined program may impart greater benefits because switching between domains may be more effective at maintaining parents’ attention, there may be spillover benefits to literacy by working on math and socioemotional skills, and success in one domain may encourage parents to continue when they experience difficulties in another domain.
- Received March 2017.
- Accepted November 2017.
