What works to improve early grade literacy in Latin America and the Caribbean? A systematic review and meta‐analysis

Abstract

Children across the world are not acquiring basic reading and math skills despite increases in primary school enrollment and attendance. Teacher training and nutrition programs in LAC are not effective in improving EGL overall, but they may be, under certain conditions. Technology in schools can be detrimental to learning outcomes if these programs only focus on technology. Approximately 250 million children across the world are not acquiring basic reading and math skills, even though about 50% of them have spent at least 4 years in school. Educational policies on EGL in the LAC region have long suffered from a disjuncture between school practice and research. What is the aim of this review? This Campbell systematic review summarizes findings from 107 studies to inform policy for EGL in the LAC region. Quantitative interventions (23 studies) Qualitative interventions (6 studies) Quantitative noninterventions (61 studies) Qualitative noninterventions (14 studies). Overall, programs did not have statistically significant effects on EGL outcomes. But there are instances in which programs may have positive or negative effects. For example, teacher training did not show positive effects on EGL outcomes, but a study from Chile showed that teacher training can possibly positively affect EGL outcomes in high-income economies when it is well implemented and complemented by sustained coaching. Similarly, nutrition programs did not improve EGL outcomes. However, a study from Guatemala showed positive effects on EGL, possibly because Guatemala has high rates of stunting and wasting. Although there is no statistically significant effect of technology-in-education programs on EGL outcomes in the LAC region, a study from Peru showed that the distribution of laptops to children can have adverse effects, particularly when not complemented by additional programs. Other studies showed that phonemic awareness, phonics, fluency, and comprehension are associated with reading ability. Furthermore, poverty and child labor are negatively correlated with EGL outcomes. This finding supports the result that nutrition programs may be effective in settings with high rates of stunting and wasting. Finally, the quality of preschool and promoting social learning are positively associated with EGL outcomes. Teacher training, nutrition, and technology-in-education programs on average do not show positive effects on EGL outcomes in the LAC region. However, there are several factors that could potentially enable positive impacts. These include combining teacher training with coaching, targeting school feeding and other nutrition programs to low-income countries with high rates of stunting and wasting, and combining technology-in-education programs with a strong focus on pedagogical practices. The review also identifies some opportunities for improving the design and implementation of EGL programs. Studies support the need to teach phonological awareness (PA) skills early on, but caution is required considering the small evidence-base in the LAC region. The evidence also supports investing in preschool quality through well-implemented teacher training. Finally, ministries of education in low-income countries with high rates of stunting and wasting could consider investing in programs to improve the nutrition outcomes of students. Caution is needed in interpreting these findings since the evidence base on what works to improve EGL outcomes in the LAC region is weak, with indications of publication bias. The review authors searched for studies published up to February 2016. Improvements in students' learning achievement have lagged behind in low- and middle-income countries despite significant progress in school enrollment numbers. Approximately 250 million children across the world are not acquiring basic reading and math skills, even though about 50% of them have spent at least 4 years in school (United Nations Educational, Scientific and Cultural Organization (UNESCO), 2014). The World Development Report (World Bank, 2018) presents evidence showing that learning is unlikely to improve unless the quality of each factor improves. The LAC region has experienced some positive trends in educational outcomes in the last decade, including improved EGL outcomes for third-grade students in the majority of the countries. However, educational policies on EGL in the LAC region have long suffered from a disjuncture between school practice and research. As a result, policy makers, pedagogy and curriculum specialists, and other stakeholders in the region are unable to determine high-quality research and what works in improving EGL outcomes. For this reason, they are unable to shape policy, practice, and programs in an evidence-driven manner. This systematic review examines the effectiveness of various programs implemented in the LAC region that aim to improve EGL outcomes, including teacher training, school feeding, computer-aided instruction, programs with an emphasis on nutrition, and technology in education programs. In addition, we assess the fidelity of implementation of programs that aim to improve EGL outcomes as well as the factors that predict EGL outcomes. Finally, we examine the experiences and perspectives of various stakeholders about EGL in the LAC region. What is the impact of reading programs, practices, policies, and products aimed at improving the reading skills of children from birth through Grade 3 on reading outcomes in the LAC region? What factors predict the reading outcomes of children from birth through Grade 3? What factors contribute to improving the reading outcomes of children from birth through Grade 3? We searched electronic databases, gray literature, relevant journals, and institutional websites, and we performed keyword hand searches and requested recommendations from key stakeholders. The search was conducted from July to August 2015 and we finalized the search in February 2016. In addition, we used novel computational approaches (specifically Wikilabeling) to maximize the comprehensiveness of the review. This review includes studies that are relevant for the literacy of children in early grades in the LAC region. This literature included both studies with an emphasis on education and studies with a focus on enabling factors that are linked to education programs or reading outcomes. For example, we included studies with a focus on nutrition that may indirectly influence reading outcomes. We developed a theory of change to identify these enabling factors. To answer our research questions, we included four study types. The first types are experimental and multivariate nonexperimental studies that include a control or comparison group. We defined these studies as "quantitative intervention studies." We included these studies to determine the impact of specific programs on EGL outcomes. The second study type consists of qualitatively oriented studies with a focus on interventions. These studies usually emphasize the process of program implementation or the experiences of beneficiaries about the performance of the program. We defined these studies as "qualitative intervention studies." The third study type emphasizes the predictors of reading outcomes but does not focus on the effects of a specific program. We defined these studies as "quantitative nonintervention studies." We included these studies to increase our understanding of intermediate outcomes and their ability to predict reading outcomes. Fourth, we included qualitative studies that discuss literacy in the LAC region but do not include an emphasis on a specific program. We defined these studies as "qualitative nonintervention studies." We systematically coded information from the studies included in the review and critically appraised them. We conducted statistical meta-analysis and sensitivity analysis using the data extracted from quantitative experimental and quasiexperimental studies. We also used narrative synthesis techniques to synthesize the findings from qualitative studies and studies that focused on predictors of literacy outcomes. We included 107 studies with a focus on EGL in the LAC region. Initial searches resulted in 9,696 articles. Following a manual review of the abstracts, we were left with a total of 164 studies that underwent full-text review. During this phase, an additional 57 articles were removed as not relevant, resulting in 107 studies included in the final review. The 107 included articles were comprised of quantitative intervention research, quantitative nonintervention research, qualitative intervention research, and qualitative nonintervention research. We included 23 articles with studies that were experimental or quasiexperimental with a focus on the effects of specific development programs on EGL outcomes. Three of these 23 articles (Cardoso-Martins, Mesquita, & Ehri, 2011; Larraín, Strasser, & Lissi, 2012; Vivas, 1996) each covered two distinct studies bringing the number of quantitative intervention studies included to 26. We also included 61 quantitative studies that had an emphasis on EGL outcomes but did not emphasize a specific intervention, 14 qualitative studies without a focus on a specific intervention, and six qualitative studies that focused on a specific intervention. Most of the studies included in our review of evidence were published journal articles and came from either Mexico or South America; significantly fewer articles were from Central America and the Caribbean. Almost all articles were published in English or Spanish. More than 90% of the articles were focused on high- to upper-middle-income countries. We only found few quantitative intervention studies with a low risk of bias. Of the 26 included studies, seven were rated as having a low risk of selection bias, five were rated as having a medium risk of selection bias, and eight were rated as having a high risk of selection bias. Furthermore, 11 studies were rated as having a low risk of performance bias, seven studies were rated as having a medium risk of performance bias, and eight studies were rated as having a high risk of performance bias. We rated 14 studies as having a low risk of outcome and analysis reporting bias, five studies as having a medium risk of outcome and analysis reporting bias, and seven studies as having a high risk of outcome analysis reporting bias. Finally, we rated 17 studies as having a low risk of other biases, eight studies as having a medium risk of other biases, and one study as having a high risk of other biases. Meta-analyses did not show the average and statistically significant effects of development programs on EGL outcomes, but a narrative synthesis of the limited number of high-quality quantitative intervention studies did show some examples of development programs that may have positive effects on EGL outcomes in specific circumstances and contexts. For example, a meta-analysis that focused on teacher training did not show positive effects on EGL outcomes (95% confidence interval [CI] = −0.17, 0.48; evidence from two programs). However, a study from Chile showed that teacher training programs can possibly positively affect EGL outcomes in high-income economies when they are well implemented and complemented by the sustained coaching of teachers (Pallante & Kim, 2013). In addition, a meta-analysis that focused on nutrition programs did not show positive effects on EGL outcomes (95% CI = −0.08, 0.25; evidence from two programs). However, a study from Guatemala showed some evidence that nutrition programs can have positive effects on EGL outcomes in contexts where stunting and wasting are high (Hoddinott et al., 2013). On average, we also did not find statistically significant effects of technology in education programs on EGL outcomes in the LAC region (SMD = −0.01, 95% CI = −0.13, 0.10; evidence from three studies). However, a study from Peru showed that the distribution of laptops to children can have adverse effects on EGL outcomes, particularly when the distribution of laptops is not complemented by additional programs (Cristia, Ibarrarán, Cueto, Santiago, & Severín, 2012). The findings of the quantitative nonintervention studies indicate that phonemic awareness, phonics, fluency, and comprehension are associated with reading ability. The research also indicates that poverty and child labor are negatively correlated with EGL outcomes. This finding on the importance of poverty and socioeconomic factors for EGL outcomes supports the quantitative intervention result that nutrition programs may be effective in improving EGL outcomes in contexts with high rates of stunting and wasting. Finally, the quantitative nonintervention studies show that the quality of preschool is positively associated with EGL outcomes. Both qualitative and quantitative studies indicated that consideration of context is key to improving reading outcomes. The most frequently discussed topic in qualitative nonintervention articles was the need to promote social learning to improve EGL. We found some indications for publication bias in the studies that focus on the effects of teacher practices, parental involvement, and Information and Communication Technology (ICT) programs on EGL outcomes in the LAC region; that is, it is possible that some studies that focus on EGL in the LAC region were not published because they did not find statistically significant effects. Our review highlighted several important implications for practice and policy related to the rollout, design, and potential impact of education programs that aim to improve EGL outcomes in the LAC region. First, our quantitative evidence suggests that teacher training, nutrition, and technology in education programs on average do not show positive effects on EGL outcomes in the LAC region. However, the quantitative narrative synthesis shows several factors that could potentially enable positive impacts of these programs on EGL outcomes. These factors include combining teacher training with teacher coaching, targeting school feeding and other nutrition programs to low-income countries with high rates of stunting and wasting, and combining technology in education programs with a strong focus on pedagogical practices. However, the evidence-base is too small to derive strong conclusions about the ability of these components to improve EGL outcomes in the LAC region. Second, the systematic review identified some promising opportunities for improving the design and implementation of education programs that aim to improve EGL outcomes. We found evidence for a strong correlation between PA and reading ability. In addition, studies focused on the importance of PA and phonics to help students become strong decoders. These findings suggest the need to teach PA skills early on, but caution is required considering the small evidence-base in the LAC region. Third, the review suggests that more resources may potentially need to be focused on enhancing the quality of preschools through well-implemented teacher training. The findings of this review suggest that such teacher training could possibly enhance reading outcomes if the training is complemented with sustained teacher coaching. The evidence-base is, however, again too small to derive strong conclusions about the effects of teacher training in preschools. Fourth, ministries of education in low-income countries with high rates of stunting and wasting could consider potentially investing in programs to improve the nutrition outcomes of students in order to improve EGL outcomes. These efforts may be less effective in middle- or high-income countries, however, and more evidence is needed to derive strong conclusions about the effects of programs that aim to improve nutrition on EGL outcomes. In general, the evidence base on what works to improve EGL outcomes in the LAC region is weak. We only found a small number of studies that can present credible estimates on the impact of development programs on EGL outcomes. The majority of the studies suffer from either a medium or high risk of selection bias or a medium or high risk of performance bias. Furthermore, we found some indications for publication bias in the studies that focus on the effects of teacher practices and parental involvement on EGL outcomes in the LAC region. There is evidence of a global learning crisis (Berry, Barnett, & Hinton, 2015; Nakamura, de Hoop, & Holla, 2019; Pritchett & Sandefur, 2013). School enrollment has improved, but EGL and math assessment data have shown high "zero" scores in literacy assessments in many low- and middle-income countries (e.g., Annual Status of Education Report [ASER], 2013; EGRA data, n.d.). The findings of the latest World Development Report on education highlight how educational outcomes are affected directly by the quality of school inputs, school management, and teachers, as well as the education preparedness of learners. In theory, improvements in the quality of one of these factors could result in improvements in learning outcomes. However, the World Development Report (World Bank, 2018) presents evidence demonstrating that learning is unlikely to improve unless the quality of each factor improves. A systematic review of Snilstveit et al. (2012) also argues that education programs are unlikely to improve learning outcomes unless they ease more than one constraint. The LAC region is composed of more than 40 countries and territories on two continents with five different official languages (English, Spanish, French, Dutch, and Portuguese) and many more regional languages. The region has experienced some positive trends in educational outcomes in the last decade, including improvements in pupil/teacher ratios, increases in the percentage of trained teachers (UNESCO, 2014, p. 8), and improved EGL outcomes for third-grade students in the majority of the countries (see Figure 1). (1) Only changes shown in blue or black are statistically significant. (2) The mean score for the region includes all countries in this graph with equal weight Source: from Are Latin American children's reading skills improving? Highlights of the second and third regional comparative and explanatory studies (SERCE & TERCE). Washington, DC: American Institutes for Research; p. 15. Reprinted with permission However, we still find great disparities among the poor, rural, indigenous, and other disadvantaged groups in the region. In addition, one in four third graders performed so poorly that they were categorized in the lowest level of the reading test, and <5% of the third graders performed so well that they were categorized as achieving the highest levels of reading. Figure 2 depicts these challenges by demonstrating that there are still a significant number of third graders scoring at the lowest levels of reading. In fact, more than 60% of third-grade students have only achieved basic reading skills (Levels 1 and 2). (1) Lowest levels include level 1 and below. (2) The mean score for the region includes all countries except for Cuba, El Salvador, and Honduras with equal weights. (3) Cuba's and El Salvador's scores are from 2006 Source: from Are Latin American children's reading skills improving? Highlights of the second and third regional comparative and explanatory studies (SERCE & TERCE). Washington, DC: American Institutes for Research; p. 19. Reprinted with permission There are many reasons to explain the poor literacy outcomes in the region but one of the key potential reasons is the lack of evidence-based training, preparation and support for teachers. According to Bruns and Luque (2015) "the seven million teachers of LAC are the critical actors in the region's efforts to improve education quality and raise student learning levels, which lag far behind those of OECD countries and East Asian countries such as China." Some of the reasons they cite are the low standards for entry into teacher training, poor quality training programs that are removed from the realities of the classroom, few career incentives, and weak support for teachers once they are on the job. In addition, teachers are not receiving the training they need to deal with students at very different learning levels, different ages, speaking different languages, and so forth, which is the reality of most LAC classrooms. Evidence-informed EGL policy can contribute to mitigating some of the concerns associated with EGL outcomes in the LAC region. However, up until now, education policies to improve reading outcomes have only been informed by evidence to a limited extent. National governments and development agencies in the LAC region have created a range of programs to improve EGL outcomes. Some of these programs specifically aim to improve EGL outcomes while others might improve EGL through indirect mechanisms. This review aimed to include any program that had the potential to affect EGL outcomes. We found and included research on the following program types and practices: teacher training, technology in education programs, school feeding and other nutrition programs, school governance programs, preschool programs, teacher practices and general pedagogical approaches, parental and community participation, and curricula. We discuss each of these intervention types below. Teacher training programs can take several forms ranging from extensive multiyear, one-on-one coaching delivered to teachers in their classrooms to training workshops delivered outside of the classroom. Emerging evidence suggests that teacher training models that emphasize sustained in-class coaching may produce larger effects on learning outcomes than short-term training models in developing countries (Kraft, Blazar, & Hogan, 2018). For instance, a study from South Africa showed that monthly visits from specialized training coaches resulted in statistically significant effects on reading outcomes (0.25 standard deviations), while two 2-day training sessions (provided over the course of a year) did not result in statistically significant effects on reading outcomes (Cilliers & Taylor, 2017). Technology in education programs involve providing technological equipment (e.g., laptops, digital game-based technology, mobile phones) to teachers or learners and integrating these tools into existing curriculums or including technology as additional tools. The equipment may have been refurbished and donated by the private sector or produced specifically for classroom instruction (Barrera-Osorio & Linden, 2009; Cristia et al., 2012). Some programs may complement the distribution of technological equipment with training modules for teachers on the use of technology in the classroom for specific subjects. Other programs do not provide any complementary training. Studies that have examined the impact of technology in education programs on learning outcomes in low- and middle-income countries suggest mixed evidence with a pattern of null results for programs that do not focus on complementary training for teachers (Bulman & Fairlie, 2016, p. 2). However, recent evidence shows more promising results for programs that include a strong focus on teaching at the right level. For example, Muralidharan et al. (2019) showed that a technology-based afterschool instruction program with a strong emphasis on learning at the right level produced large and statistically significant effects on reading outcomes in India. School feeding and nutrition programs vary in their modes of delivery and expected outcomes. Most programs are delivered within schools and provide meals (typically breakfast or lunch) to participating children (Adrogue & Orlicki, 2013; Powell, Walker, Chang, & Grantham-McGregor, 1998). Other programs may provide children with specific nutrients that might be missing from their diets (Maluccio et al., 2009). Nutrition programs may aim to improve school attendance and boost learning outcomes, in addition to aiming to aiming to improve children's food security and nutritional status. School governance interventions address school management issues that affect the delivery, quality, and financing of education. These programs often focus on decentralizing decision-making at the school level or improve parents' and communities' involvement in school management. Some school governance interventions involve cash transfers to schools and provision of matching funds for private investment to schools along with institutional changes, which allows parents to how to funds Other models might provide support to poor schools on identified in a school & education programs often focus on preschool the of primary education. The effects of preschool can be by in the of spent in and quality of school quality of instruction and factors such as & 2012). aimed at parents in children's early literacy take approaches and have shown mixed In developed countries, several interventions focus on to improve children's literacy & have the findings from literacy training programs for parents National but the effectiveness of training on children's literacy has not been through systematic because of among the studies & that curriculum and teacher practices for literacy instruction take approaches as For instance, some interventions teachers to explain to learners reading in order to boost reading comprehension et al., 2012). Other interventions focus on providing PA training to boost et al., interventions involve more actors and may have outcomes. For instance, interventions may focus on the of an existing curriculum to across or teaching that to different levels 2014). We developed a theory of change all types of programs the across enabling and programs or that are associated with intermediate outcomes, and reading outcomes, as well as the that the theory of The theory of change how programs or can contribute to improving EGL outcomes in a manner. Figure 3 depicts the theory of The theory of change with the enabling factors and that are for any intervention or program to be to impact EGL outcomes in the LAC region. These factors to that need to be in to enable programs that are effective in improving reading outcomes. education programs are implemented along with other programs that may have or effects on EGL. implementation the achievement of intermediate EGL outcomes, such as changes in teacher and practices, which in improve EGL. Finally, we include key for implementation and enable implementation to outcomes and impacts. also on potential including at the community in the and of these among The theory of change also that influence how stakeholders with programs or practices, as well as or factors that influence implementation and the in the the in the can be by the enabling This enabling consists of the and other that need to be in to enable the implementation of programs that are effective in improving EGL outcomes. For example, teacher training programs are to be more effective in an with a number of teachers with the to school. Similarly, teaching students how to is to be more effective in an in which students are not or Finally, a strong governance the for high-quality education by that schools and teachers are and have a within which they can programs or practices. The World on Education for in the focus of the education from to quality and a in the quality of education students (World on Education for of the six at the to and support for EGL were the of early and development and in learning There is evidence that programs in low- and middle-income countries that focus on educational without other to learning are not to improve learning outcomes et al., 2012). et al. that to have an impact on student if what is