The Covid-19 pandemic that prompted a nationwide shutdown of schools and a shift to online instruction in spring 2020 also prompted a wave of articles calling this instructional change a “natural experiment” that could be used to study the effects of online education. Yet the pandemic disrupted so many aspects of children’s academic, social, emotional, and economic lives that its broad scope poses serious challenges to isolating the causal impact of any specific change, such as the switch to remote instruction.
Educators and policymakers should proceed with caution when interpreting studies that attempt to identify such specific effects. Instead, researchers should focus on helping education leaders understand the overall impact of the pandemic on students, putting particular emphasis on discovering which groups have suffered the worst effects. The best evidence to date indicates that Covid-19 has had a substantial negative impact overall and has disproportionately harmed the learning of disadvantaged students. Continued research in this direction could provide a sharper picture of which students have faced the most severe challenges under Covid-19, pointing the way toward how best to allocate resources to address learning losses.
Econometric Challenges
In econometrics, an “instrument” is a variable that has a direct impact on the probability that an individual is treated by a policy of interest. For example, some people have suggested that the pandemic could serve as an econometric instrument to study the effects of online learning, since the pandemic dramatically increased the number of students learning virtually in 2020. Major concerns arise, though, in using the pandemic as an instrument to study the impacts of such a policy change. These concerns relate to the fact that a rigorous causal study must have both internal and external validity. Internal validity requires that econometric analyses capture the true causal impact of only the policy change of interest—in this case, the shift from in-person to online instruction—rather than the possible effects of other contemporaneous changes. External validity requires that the estimated effects of a policy change induced by the pandemic would accurately predict the effects of similar treatments in other contexts, such as a typical school year. The nature of the pandemic presents serious challenges to both the internal and the external validity of most research designs, including the use of the pandemic as an instrument.
Challenges to internal validity. To understand why many studies of pandemic-induced policy changes could suffer from serious threats to internal validity, first consider a common research design with notably strong internal validity: a randomized experiment. If researchers were to use a randomized experiment to estimate the impact of remote instruction on student learning, they would need to assign students at random to the treatment group (students who receive remote instruction) or the control group (students who receive in-person instruction). Assigning the treatment at random ensures there are no systematic differences between the two groups other than the treatment itself. A randomized experiment has high internal validity because it clearly isolates the impact of a particular treatment.
Quasi-experimental research designs, or “natural experiments,” take advantage of variations in treatment status that occur as a result of policy changes or other “natural” phenomena outside of a researcher’s control. While natural experiments eliminate the need to actively assign individuals to treatment and control groups, they typically face greater threats to internal validity than do randomized experiments. One common type of natural experiment uses an econometric instrument to estimate the effects of a particular policy change. The onset of the pandemic holds obvious appeal as an econometric instrument for studying the effects of online instruction, since the crisis caused a sudden shift to remote teaching. A key assumption of this research design, however, is that the policy change of interest (the shift to online learning) does not coincide with other relevant changes. In econometrics, this is known as the “exclusion restriction,” which requires that the econometric instrument (the pandemic) affect the outcome of interest (student learning) only through the policy change of interest (the shift to online learning) and not through other channels.
While the Covid crisis did spur the shift to online instruction, it fails the exclusion restriction because of the many contemporaneous changes that likely also affected student learning. Students shifted to remote learning as their parents lost jobs, as their family members suffered Covid’s health effects, and as they lost the ability to leave the house and see friends, among other significant changes to their lives. If Covid-19 did affect student learning, it would be difficult to attribute the changes in outcomes to remote instruction rather than any of these other contemporaneous factors. We illustrate this in Figure 1, which shows the potential use of Covid-19 as an econometric instrument for remote instruction. The exclusion restriction requires that there be no “causal arrow” between the other channels affected by Covid-19 and student learning, an assumption that is certainly violated based on both common sense and prior empirical evidence. It may be possible to estimate the overall effect of Covid-19 on student outcomes, but attributing that effect to any one channel is likely impossible.
We can illustrate the violation of the exclusion restriction with an example from our own recent research into Covid’s impacts on household Internet-search behavior. In that work, we show that Covid-induced school closings caused parents to seek out online learning resources that might compensate for lost in-school instructional time. An example of this can be seen in the top panel of Figure 2, which shows a large increase in Google searches for “online learning” that corresponded precisely with the timing of the pandemic outbreak in the United States. This provides evidence that the Covid-19 crisis indeed represents a sudden shock to the demand for online learning resources.
At the same time, however, there were many other changes in students’ lives that are reflected in Internet search behavior. Data show, for example, that there were sudden and contemporaneous increases in Google searches for terms relating to the economic condition of households, such as “unemployment insurance” and “food stamps.” The pandemic changed students’ educational experiences but also generated a considerable economic shock to many households. These large, simultaneous changes make it difficult to separate the effects of one shock from another.
Challenges to external validity. The unprecedented circumstances surrounding the Covid-19 crisis also present serious challenges to external validity. Researchers typically examine whether the context and implementation of a policy shift are representative of potential future enactments of the same policy. Neither the context nor the implementation is likely to be representative in this case.
First, the learning environment during the pandemic is unlikely to generalize to typical school years because of the many changes in students’ lives that likely put a strain on their capacity to learn. For instance, students and young adults have reported substantial increases in anxiety and depression during the pandemic. Second, pandemic-induced policy changes were implemented in a way that is unlikely to resemble a more carefully planned implementation of the same policy in a typical year. For example, the pandemic-induced shift to online learning required teachers, with no advance warning, to quickly redesign lessons originally intended for in-person instruction. Under normal circumstances, teachers would have been afforded time to prepare lessons specifically for online instruction. Even in the fall of 2020, there was still substantial uncertainty around schooling logistics and instructional modality, making it difficult for educators to plan instruction effectively in advance. The unprecedented circumstances of the pandemic and the corresponding ad hoc policy shifts are therefore unlikely to generalize to well-planned policy changes in a typical school year.
The Overall Impacts of Covid-19 on Students
Although it is nearly impossible to disentangle the effect of any specific policy, the overall effect of the pandemic—including economic, health, social, and educational changes—is something we can attempt to assess. Moreover, as the pandemic continues to disrupt daily life more than a year after schools first closed, it is increasingly important to understand the impact of the pandemic itself. In addition to the short-run impacts on learning, a range of prior evidence suggests that the effects of health, social, and economic experiences in early childhood can persist into adulthood.
Emerging evidence on the short-run impacts shows that Covid-19 has caused substantial disruptions to students’ learning, particularly for disadvantaged students. Raj Chetty and colleagues find that student progress on Zearn, a popular online math platform, decreased by roughly 30 percent over spring 2020, with children in the lowest-income schools seeing progress drop by 50 percent and those in the highest-income schools quickly recovering to pre-pandemic levels. Nationwide evidence from fall 2020 MAP Growth assessments suggests that students lost ground in mathematics and that reading losses were concentrated among Black and Hispanic students in upper elementary grades. Recent work in Georgia suggests that students lost further ground as the school year progressed through the winter of 2020, with such losses larger among low-income, Black, and Hispanic students. Our research reveals one potential reason for these disparities: when the pandemic first struck, demand for online learning resources increased substantially less in low-income areas than in high-income areas of the United States (see “What Google Search Data Reveal about Learning During the Pandemic,” web only).
Education researchers predict that the pandemic will substantially increase achievement gaps between students from low- and high-income households, even beyond the 2020–21 school year. The best evidence to date shows that Covid-19 not only reduced the learning of the average student compared to typical school years, but that it also increased achievement gaps by disproportionately harming disadvantaged students.
What Comes Next?
Instead of framing the pandemic as a “natural experiment” for studying specific educational interventions, we propose that researchers and policymakers focus on measuring the overall effects of the pandemic itself. We believe it is possible to generate econometrically sound estimates of the overall social, emotional, and academic costs of the pandemic. The pandemic is, however, too large and unprecedented a shock to give us precise insights into individual aspects of children’s educational experiences that have changed. Too many things changed all at once.
After more than a year of pandemic-induced restrictions and shutdowns, there is reason for guarded optimism. Vaccinations have become widely available in the United States, Covid cases, hospitalizations, and deaths here have dropped rapidly, and the era of widespread school closures and fully remote instruction is ending. The educational effects of the pandemic are, however, likely to linger unless we identify the students who have been most adversely affected and provide additional resources to reverse these impacts. The best evidence to date shows that Covid has not only impeded the learning of the average student, but also widened achievement gaps by disproportionately harming the learning of low-performing students. Educators now face the challenge of not only making up for lost instructional time but also closing gaps that are even wider than usual. Though choices of how best to remedy these losses may be best left to individual states, districts, or schools, substantial resources should be devoted to these efforts. Without such investment, particularly among students who have experienced the greatest setbacks, we will likely enter an era of increased educational inequality persisting beyond the return of fully reopened schools.
Andrew Bacher-Hicks is assistant professor at Boston University Wheelock College of Education and Human Development, where Joshua Goodman is associate professor of education.
This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:
Bacher-Hicks, A. and Goodman, J. (2021). The Covid-19 Pandemic Is a Lousy Natural Experiment for Studying the Effects of Online Learning: Focus, instead, on measuring the overall effects of the pandemic itself. Education Next, 21(4), 38-42.
