After years of steady growth and a pandemic-related explosion, online learning has become a common format for college courses. A decade ago, just 28 percent of all U.S. college students took at least one of their classes online. By the 2021–22 school year—after widespread pandemic-related lockdowns had ended—that had doubled to 54 percent, or 10.1 million students nationwide.

This shift has helped institutions by keeping the virtual door open during emergencies, broadening their pool of potential students, and decreasing brick-and-mortar operating costs. It also has benefitted students by expanding access to advanced coursework for degree-seekers who live in remote areas or juggle coursework with other responsibilities.

We know less, however, about how online learning affects student learning. While prior research has found negative or mixed effects, these are based on self-selected groups of students. College students typically choose their institution and course of study, including whether they take some or all of their classes online.

Our study is the exception. We are, variously, officers in the United States Army and graduates of and former civilian faculty at the United States Military Academy West Point, a unique institution established to educate future Army officers. To apply, aspiring students face a demanding battery of admissions requirements, including outstanding academic and athletic records, evidence of leadership potential, and a nominating letter, typically written by a member of Congress. To graduate, they must complete academic, military, and athletic training, pass several physical fitness tests, and earn 120 college credits. Unlike other institutions, students have little control over their class schedules. More than 80 percent of the academic program is standard across all majors, and coursework is identical across instructors, who follow a unified curriculum with the same materials and graded events.

These unique attributes allowed us to conduct a randomized controlled experiment during the fall 2020 semester, when West Point brought students back to campus but also used online instruction to limit class sizes, maintain social distancing, and slow the spread of Covid-19. We compare coursework, grades, and results of a post-course survey among 551 students who were randomly assigned to online and in-person sections of Principles of Economics, a required class that most students take their sophomore year.

We find that student learning, focus, and engagement suffer when instruction moves online. Online instruction reduces a student’s final grade by 22 percent of a standard deviation, or about 1.7 percentage points—equivalent to declining to a B+ from an A–. The impact is larger for males, at 27 percent of a standard deviation, compared to 9 percent of a standard deviation for females. There is virtually no impact for Black students, while grades for white students are 28 percent of a standard deviation lower when they take the class online.

We see negative effects from online learning across the entire grade distribution but find the biggest impacts on the least academically prepared students. Compared to their in-person counterparts, grades are lower by 45 percent of a standard deviation for students with prior military service, 38 percent of a standard deviation for students who attended a stepping-stone preparation school prior to being admitted, and 27 percent of a standard deviation for students with the lowest scores on an admissions exam. We also find that online students are less likely to report feeling connected and focused compared to in-person students.

These findings do not indicate that all online learning is detrimental. The classes in our study were identical except for their setting, so students did not experience some unique attributes of some online courses, such as self-paced study, that could positively affect outcomes. In addition, because West Point classes are smaller and feature more individualized attention than the large lecture halls used in previous experiments comparing in-person and online courses, the negative impacts on instructor-student relationships may be especially large.

However, when we consider other ways that West Point differs from traditional colleges and universities, our findings are concerning. West Point students are among the nation’s most disciplined young people. They have met stringent, multidimensional requirements to enroll in a structured military leadership program that accepts about 12 percent of applicants. They attend class in uniform, maintain peak physical fitness, and commit to at least five years of active-duty military service after graduation. If online learning has negative effects at West Point, what does that mean for the typical student?

A Pandemic Experiment

Like many colleges and universities during the Covid-19 pandemic, West Point pivoted to online learning to finish the 2019–20 school year. But while many institutions remained entirely online in fall 2020, West Point students—who are known as “cadets”—were required to return to campus to continue their physical and military training as well as their academic studies. To limit classroom capacity and incorporate social-distancing measures, the academy offered both online and in-person classes. As in years past, students were expected to follow schedules set by the academy and did not have the opportunity to request changes. Attendance was mandatory, regardless of modality.

Our experiment builds on the academy’s regimented coursework and existing practice of random assignment. That semester, 551 students were slated to take Principles of Economics, a core course similar to introductory courses at other universities. We received permission to randomize students across 12 instructors in 36 class sections, half of which were online. Each class section included between 12 and 18 students. In all, 61 percent of students were in online sections and 39 percent attended classes in person.

The syllabus, graded events, homework assignments, and midterms were the same in all classes, but instructors used different final exam versions on different days. Both online and in-person class sections were taught in each class hour the course was available, and we randomly assigned students within the hour into each modality. The only exception was for student-athletes, who are prioritized for morning classes to accommodate team schedules. Within those morning class hours, we randomized student-athletes into either an in-person or online class section.

Each instructor agreed to teach their four-section teaching load half online and half in person. West Point classes meet roughly every other day, so instructors taught no more than two class sections per day. We chose which class sections were online, and the registrar’s office randomly assigned students to class sections. This environment created counterfactual classrooms where one could see a given instructor teach both in person and online, which allows us to control for instructor talent, experience, quality, and familiarity with the course material.

Data and Method

Our analysis is based on three sources of data: student demographics and pre-West Point academic achievement data; grades for daily homework, problem sets, midterms, and a final exam; and a voluntary post-course survey, which 73 percent of students completed for extra credit.

Student data show that our sample is representative of the West Point student body: 23 percent female, 14 percent Black, 3 percent Hispanic, and about 6 percent Asian. In keeping with the academy’s focus on physical fitness, 29 percent of students are NCAA Division 1 athletes. About 17 percent are enlisted soldiers who enrolled to earn a college degree and commission as officers, and another 14 percent are students who previously attended an onsite prep school, which offers a yearlong academic development program for high school graduates to remediate weaknesses and prepare to apply to West Point. We also look at these characteristics by course section enrollment and find no statistically significant differences in students being assigned to online versus in-person classes.

During the semester, online software published and executed all graded events. Daily homework assignments were graded for completion, not accuracy. Midterms and final exams contained multiple-choice questions and software-generated graphing exercises. The software graded all assignments and exams. Because instructors had no discretion over grades, we eliminate any potential differences in grading between modalities.

After the class, a post-course survey asked students to rate their ability to concentrate and feelings of connection to their classmates and instructor on a five-point scale. The students who took the survey are representative of the whole group in terms of demographics and whether they were online or in-person. Researchers had sole access to their responses.

We use these data to conduct a straightforward comparison of grades for students enrolled in online and in-person sections, controlling for instructor, time of day, and student characteristics. Because instructors taught the same class both online and in person, this simulates a hypothetical experiment comparing a student who learned from an instructor in person to an identical student who only saw the instructor online.

Impacts on Student Performance

Overall, West Point students did reasonably well in Principles of Economics in fall 2020—the average grade is an 83, or a B. But in comparing outcomes for students assigned to online versus in-person course sections, we see that grades for online students are 22 percent of a standard deviation lower overall (see Figure 1).

The largest negative effects are for the least-prepared students. Students in the bottom quartile based on their academic records from high school, including grades and scores on the SAT and ACT, experience declines that nearly twice as large as those for students with above-average high school performance: 27 percent of a standard deviation compared to 15 percent. Grades for students who came to West Point from the military are lowered by 45 percent of a standard deviation, nearly triple the effect for students without prior military experience, whose grades are 16 percent of a standard deviation lower. The negative impact for students who attended a preparatory program out of high school is twice as large as students who did not attend the prep school, at 38 percent of a standard deviation compared to 18 percent.

Males are more negatively affected than females, with declines of 27 percent of a standard deviation compared to 9 percent. And while there is virtually no effect on Black students, grades for white students are 28 percent of a standard deviation lower when instruction moves online.

Online education adversely affected students across nearly the full range of performance in the course. While students at the extremes—those who earned either As or Fs—performed similarly in online and in-person sections, we find that at every other point on the distribution, student grades in the in-person sections dominate the online sections.

We also look at how online learning affects different graded events and again find lower student performance across the board. Daily homework assignments, worth five points each, are usually viewed as a measure of student engagement and persistence as students earn points solely for completing the assignment. The learning software grades homework automatically, so there is no instructor discretion. Online instruction lowered a student’s homework grade by 21 percent of a standard deviation, a result similar to the overall course grade.

Instructors in the course distributed exams via online learning software regardless of teaching modality. There were two versions of the final exam that were randomly assigned to students without regard to instructor or teaching modality. All students took the final online in their dorm rooms using the instructional software provided by the textbook publisher. We control for the version of the exam students saw when comparing their scores and find online learning lowers a final exam grade by 13 percent of a standard deviation.

Impacts on Student Experiences

In a post-course survey, online students reported lower levels of concentration and feelings of connectedness compared to in-person students (see Figure 2). On a five-point scale, the average student rated their ability to concentrate in class at 3.5 compared to 3.0 for online students, a decline of about three-fourths of a standard deviation. Online students also reported spending 2.3 additional minutes per day studying for the class, a finding that is not statistically significant but interesting nonetheless, since their grades were lower.

When asked how connected they felt to their instructor, with 5 being the strongest connection, the average student response was 3.8 compared to 3.5 for online students, a decline of two-thirds of a standard deviation. When asked if they felt that their instructor cared about them, the average response was 4.2 compared to 4.0 for online students, a difference of about a quarter of a standard deviation. Finally, we asked students to rank their connection to their peers. The average student response was 2.7 compared to 2.2 for online students, a decrease of roughly a half a standard deviation.

These estimates are based on comparisons of students taught by the same instructor in different teaching modalities and therefore control for instructor attributes, teaching styles, and personalities. It seems that the many costs of online education during the pandemic included student satisfaction, concentration, and many of the intangible benefits a professor provides in an in-person class.

Taken together, our quantitative and qualitative findings illustrate some of the limitations of online learning. In this experiment—and as experienced at colleges and universities across the United States during Covid-19—instructors had little time to prepare and adjust teaching styles and pedagogy and thus ported a traditional classroom environment online.

This approach negatively affected student outcomes. But our experiment and analysis informed an immediate response. While it is West Point policy to randomly assign students to classes, the academy prioritized students with weaker high school academic records for in-person classes during the 2021 spring semester. In addition, final course grades for students who had been assigned to online sections of Principles of Economics were adjusted upward by about 1.7 percentage points, in line with our findings.

A Cautionary Tale

In reflecting on the limitations of our findings, we return to the unique attributes of West Point that made this experiment possible. The rigor and structure of a military leadership institution certainly mean that a West Point cadet’s experience differs from that of the average college student, and the extraordinary requirements for admission mean these students are more disciplined than the typical young adult.

We believe that our findings represent the lower bound of the negative impact of online instruction—that is, in the best possible scenario, where high-performing students lived on a closed campus, did not have financial responsibilities or jobs, and were somewhat less exposed to a disruptive virus, online learning negatively affected student outcomes and experiences. Many other college students, particularly those from disadvantaged backgrounds, may have fared far worse, and we find that even at West Point, less-academically-prepared students experienced greater negative effects.

While instructors can develop new pedagogy and thoughtfully adapt coursework and instruction to an online environment, the potential for widespread learning loss as we observe at West Point should give policymakers and college administrators pause. Online learning may be popular, but it’s not clear that increasing online instruction is in students’ best interests.

Michael S. Kofoed is an assistant professor at the University of Tennessee, Knoxville and was formerly a faculty member at the United States Military Academy. Lucas Gebhart, Dallas Gilmore, and Ryan Moschitto are officers in the United States Army. The views expressed herein are those of the authors and do not reflect the position of the United States Military Academy, the Department of the Army, and the Department of Defense.