Kassidy Fann clears off the table in the tidy kitchen of the wooden triple-decker north of Boston that she shares with her husband and a pair of inquisitive cats.
Fann takes a seat and opens her laptop. There’s no sound but the creaking of the pipes and the rain against the windows. Then a disembodied voice with a slight southern accent says hello.
“I will be your proctor today,” says the voice, which belongs to a woman named Leigh Ann Majerik, who’s in an office in Hoover, Alabama. Majerik will supervise as Fann takes a test—called, in this case, an assessment—to gauge her progress toward the master’s degree she’s pursuing in science education and physics from the online Western Governors University.
Fann holds up her driver’s license and looks into a fisheye webcam so the proctor can confirm her identity and see that there are no unauthorized parties in the room other than one of the cats. After a few more clicks, the test pops up on Fann’s laptop screen, along with a countdown clock in the upper-right-hand corner.
“Start when you’re ready,” says the proctor.
It’s not just being tested in her kitchen toward a graduate degree that makes Fann something of a pioneer. It’s also how she’s being tested, and for what: her grasp of everything she’s learned so far, and not just some of it.
Unlike conventional colleges and universities, Western Governors doesn’t require students to spend a set number of hours in a classroom, average out their performance on assignments and tests, then hand out letter grades and credits. Using a complex system of assessments developed over the two decades the university has been operating, WGU’s method is competency-based, requiring that students prove they’ve mastered all the skills and knowledge offered in a given subject area. Until they do, they don’t advance.
“You have to pass. You have to meet the criteria. There’s no [grading] curve,” says Fann, a high-school science teacher who earned her undergraduate degree from Barnard College. “It does actually test whether I know this stuff or not.”
Fann not only likes this approach, she also thinks it could be applied to her 9th-grade physics students.
Although the model was originally designed for older-than-traditional-age college and graduate students, Fann believes “there’s nothing stopping” the principles of competency-based education and assessment from working in primary and secondary schools.
“I don’t know that all of my students would have the self-starter capabilities to succeed at this,” she says. But competency-based education, by definition, would let them progress at their own pace. Those who were ahead could push forward by working online; those who were behind could get more help from the teacher. At the end of the process, they would have to prove, just as Fann does now, that they understand the subject—not just some of it, but all of it. And they would have to demonstrate that mastery not to their classroom instructors but to separate and impartial assessment faculty.
“Every day my students ask me, ‘Is this going to be on the test?’” Fann says, shaking her head. “I don’t want to tell them. Here [at WGU], you actually have to know all the material.”
Like Fann, a growing number of people are thinking about how competency-based education and assessment might work in high schools and even lower grades. The Every Student Succeeds Act of 2015 opened the way for primary and secondary schools to experiment with this and other new kinds of testing, and states that include New Hampshire, Kansas, Maine, Arizona, Colorado, and Vermont are already giving it a try.
New Hampshire is the furthest along, with its Performance Assessment of Competency Education initiative, or PACE. Rather than counting how much time students spend in seats, this system tests whether they’ve met “learning targets,” requiring them to pass incremental assessments—demonstrating the skills they’ve acquired—in order to keep moving forward. Students in middle-school English, for example, write research papers showing they can analyze and present information from different sources. Fourth graders in math design a new park, estimate its construction cost, and produce a presentation arguing in favor of building it.
Competency-based education and assessment is relatively new to K–12 education, and at this point, there are not sufficient data or large-enough sample sizes to discern how well it works there. But in higher education the model has been used for more than 20 years, affording an opportunity to examine its track record and potential.
Competency-Based
A bipartisan collaboration from an era when such cooperation still happened, Western Governors University was jointly proposed in 1995 by the Republican governor of Utah and the Democratic governor of Colorado. It launched two years later with $100,000 from each of its 19 founding states, then struck out as an independent, nonprofit operation.
In spite of early skepticism toward online education—which persists, to some degree—Western Governors grew to 38,000 students by 2012. It is now in its 20th-anniversary year, and enrollment exceeds 80,000 undergraduate and graduate students in schools of education, business, health professions, and information technology (see Figure 1). It ranks fourth in the nation in the number of bachelor’s and master’s degrees awarded to nonwhite students in nursing, and awards 11 percent of bachelor’s and master’s degrees for teachers of technology, math, engineering, and science. Graduates hold top jobs at the likes of Aetna, American Express, Coca-Cola, Delta and United airlines, JPMorgan Chase, Pfizer, and Toyota. This year, the university debuted new degree programs in the hot field of data analytics.
But what Western Governors is best known for is its embrace of competency-based education.
Each of its degree programs includes a list of topics to be mastered, called “domains.” For a bachelor’s degree in accounting, for example, required domains include organizational behavior, business law and ethics, quantitative analysis, information technology, marketing and communications, systems administration, and the liberal arts. Accounting students must demonstrate that they understand such things as the nature and purpose of information systems and that they have mastered skills that include information systems auditing. These competencies are determined not by the faculty alone but in consultation with employers.
The point of involving employers in the process, says Scott Pulsipher, president of Western Governors, is “to develop a credential that has value in the jobs that [students] are pursuing.”
Students first take no-stakes “pre-assessments” to track their progress whenever they want (including at the very start of a course). If they fall behind, a mentor intervenes to catch them up on the material they don’t know. Advocates liken these pre-assessments to the PSAT many high-school sophomores and juniors take to practice for the SAT and discover where they still need work.
A dramatic departure from the conventional “seat-time” standard, competency-based assessment allows students to proceed at their own pace, a feature that’s attractive to both those who want to accelerate their degree program and for older-than-traditional-age students juggling schoolwork with families and jobs. (In the K–12 realm, more-advanced students can speed through, while others can take extra time if needed.)
Pre-assessments help students know when they’re ready to move on to “objective” assessments that determine mastery of the required domains. These assessments combine easy-to-score multiple-choice exams, such as the one Fann took in her kitchen under the remote eye of a proctor, with real-world “performance” assessments, including presentations and case studies graded by people whose only role in the process is providing this assessment. At the time of her exam, Fann had also just wrapped up an essay on thermodynamics, enhancing it with illustrations she created on her iPad.
If they don’t pass an assessment on the first try, students are usually asked to retake the pre-assessment before seeking permission from their faculty mentors to try again. (There are different forms of each assessment, so the questions are always different.) If they don’t pass on the second attempt, the process starts again, but with an additional $60 fee per test. More than 90 percent of students pass on the first or second attempt, WGU says.
The results of these evaluations are easier for third parties to interpret than a letter grade from a conventional class, say advocates of competency-based assessment. For instance, if a student gets a B- at a traditional university, it can be tough for another teacher, a college admissions officer, or an employer to know what he or she didn’t understand, or how that B- compares to the same grade given at a different institution.
“The feedback is durable,” says Kim Kostka, a chemistry professor at the University of Wisconsin, Rock County, who also teaches students in that university’s competency-based UW Colleges Flexible Option, or UW Flex. “In a traditional curriculum, let’s say a student passes the class but they have failed Unit Two. That just gets averaged into their final grades. What can I say about that student’s understanding of Unit Two? They didn’t demonstrate it. In competency-based education, you can’t average out a poor performance.”
Measuring Effectiveness
Western Governors preaches what it practices. It’s home to the online quarterly Journal of Competency-Based Education, hosts seminars on the topic for educators and policymakers, and, using data collected on its more than 80,000 graduates (see Figure 2), teams up with researchers from Harvard, Stanford, Carnegie Mellon, the University of Chicago, and other institutions who study competency-based education, or CBE.
Measuring the real-world effectiveness of CBE and its assessments is as complicated and contentious as it is essential; after all, the success of the approach depends on the acceptance of CBE credentials by licensing agencies, graduate schools, and employers. Competency-based education needs to earn external validation if it is to endure, pronounced a landmark paper from the Center on Higher Education Reform at the conservative-leaning American Enterprise Institute.
For Western Governors, its officials say, this largely means checking in on graduates and what they got from their education, to provide proof that—as the paper described it—a CBE credential “stands for a level of rigor and preparation equivalent to a traditional postsecondary degree.” The paper called a competency-based model “workable only insofar as its measures of learning yield trustworthy data about students’ prospects for future success.”
To test how well its methods work, Western Governors resorts to an unusual form of evaluation: a Gallup poll. It surveyed its alumni last year—2,676 of them, randomly selected, who graduated between 2000 and 2016—to ask if what they learned prepared them for life. Thirty-one percent responded that it did, which might seem low, except that only a surprising 26 percent of graduates nationally said this, 24 percent from public universities and 20 percent from for-profit universities.
In a separate Harris poll of 1,207 WGU graduates and 1,403 graduates of other colleges nationwide, also conducted last year, 78 percent of Western Governors grads said what they learned was directly related to their work, compared to 68 percent of the other graduates. Ninety-nine percent of employers said WGU graduates met or exceeded expectations, and 100 percent reported that the grads were prepared for their jobs. A Gallup poll conducted for the Lumina Foundation, which promotes increased access to higher education, found that just 11 percent of business leaders said they were getting the skills they needed from the college graduates they hire overall.
Another way of measuring WGU’s effectiveness is through student pass rates on professional licensing exams. Western Governors says its students pass teacher certification tests administered by Pearson Education at a rate of 96.2 percent, and that they pass the Educational Testing Service’s Praxis teacher tests at a rate that’s 3.8 percentage points above the national average. (The average national pass rate on the Praxis is difficult to calculate, because ETS provides such information largely on an institution-by-institution basis, and passing scores vary by state; some education colleges do better than Western Governors, some not as well.) Nearly 90 percent of WGU nursing graduates passed licensing exams last year, which is slightly higher than the national rate of 87.8 percent.
The university uses this kind of data to occasionally tweak its approach. “It’s really a continuous-improvement culture,” says Jason Levin, WGU’s vice president of institutional research. For example, although its model relies heavily on student self-direction, the university has added mentors from whom students can seek help during their course of study, and who check in with students if they fall off track. “We want to be a self-paced institution and we don’t want to get in the way of students who come with competencies and learn quickly,” Levin says, “but the reality is we have a wide range of ability and experience in our students.”
Western Governors also toughened its pre-assessments and has added an online coaching report, giving its students real-time looks at how they’re faring. The latter feature resulted in a nearly 4 percent improvement in the number of students who pass the final assessments, which is gradually helping the university raise the proportion of students who graduate within six years above the current 41 percent. (The six-year graduation rate from colleges and universities nationwide is 52 percent, though Western Governors compares itself to open-admission institutions, for which the rate is 36 percent.)
But some of the most noteworthy innovations have been built into the process since the outset, such as the practice of designing competencies in collaboration with employers. Western Governors convenes “program councils” composed of industry experts who advise on each of its colleges, on general education, and on assessment. These councils meet at least twice a year at WGU’s Salt Lake City headquarters, and members are paid stipends and compensated for expenses.
“We had a lot of concerns around the curriculum and the matching of the curriculum itself to what employers were looking for in the graduates,” says WGU’s Pulsipher, who previously worked at Amazon and chaired a technology industry advisory group at the school of management at his alma mater, Brigham Young University. There, he says, “it was basically academics competing with employers saying, ‘I get what you want to teach, but we’re trying to tell you that what you’re teaching doesn’t map with what we need.’”
Surveys bear out that perspective. In the Gallup poll of business leaders done for Lumina, only a third agreed that “higher education institutions in this country are graduating students with the skills and competencies my business needs,” compared to 96 percent of chief academic officers who thought so.
Other programs using CBE also enlist the advice of industry leaders. Srirajasekhar Bobby Koritala, CEO of a Naperville, Illinois, software company, was one of the businesspeople invited to help plan out the competencies for UW Flex. “I frankly never had any college ask me that before,” he says. “I found it refreshing. In some ways, employers are somewhat disconnected from colleges.”
He’ll get no argument about that from the Reverend Dennis Holtschneider, president of DePaul University from 2004 through June of this year. DePaul’s School for New Learning uses competency-based education and a mix of online and in-person courses in its degree programs for older-than-traditional-age students. The university has contracts with local businesses and public agencies, from the Chicago Police Department to Fifth Third Bank, to enroll their workers, and can ask these employers “exactly what they need” from their employees, Holtschneider says.
Atypical Assessment
In another significant departure from traditional education practice, Western Governors and institutions like it deploy separate groups of faculty for different areas of responsibility: establishing competencies, teaching, and testing. That means the teaching faculty are on trial in assessments, too, a complete upending of a long-held classroom culture in which instructors evaluate the knowledge of their own students.
“You don’t want the fox guarding the hen house,” Pulsipher says. “The design of [the assessment component] was important, because part of the integrity of a competency-based model says there are standards for proficiency … If you take assessment out of the hands of the teacher, I would argue there’s a higher level of integrity there.”
Separating teaching from assessment raises the stakes for instructional faculty, Pulsipher says. “They want to know that their students are ready to take the assessment.” That’s because a faculty member who has a cohort of students not doing very well will draw scrutiny from managers to see what he or she is doing differently, he notes. (A years-long audit is underway by the U.S. Department of Education’s inspector general over whether the Western Governors teaching faculty spend enough time with their students, though the question is principally a technical one over whether WGU is a distance-education program or a correspondence school for purposes of eligibility for federal financial aid.)
Having some faculty teach and others test “helps our faculty think really clearly about outcomes of their work,” says Holtschneider. “It introduces accountability when someone else is going to evaluate your students and what they’ve learned.”
Cynthia Suopis likens the approach to serving as a doctoral-student adviser. “You’re advising that student through the dissertation, but then it has to get through the committee,” says Suopis, who serves on the faculty of the University Without Walls at the University of Massachusetts Amherst, which follows the same model. UWW alumni include basketball great Julius “Dr. J” Erving and Jeff Taylor, the founder of Monster.com.
That “committee,” at WGU and similar programs, is a completely separate team of assessors. This firewall between instruction and assessment theoretically eliminates the bias that can occur when classroom teachers grade their own students.
With its enrollment swelling, Western Governors now has 900 faculty evaluators, up from 250 as recently as 2010, says Debbie Fowler, the university’s associate provost. A quarter of them are full time and the rest are part time and paid by the hour. Most work remotely. All have graduate degrees in the subject matters they assess—either master’s or doctorates, depending on the level of the course. (Just under a third have doctoral degrees, Fowler says.) Together, they score 98,000 performance evaluations a month, with individual evaluators returning them in an average of just under 40 hours.
The evaluators are on constant trial, too. Using pass-rate data, WGU looks at evaluators who might be outliers among the rest, because “we don’t want the student’s chance for passing to be determined by which evaluator is picked for the submission,” Fowler says. While some evaluators may be legitimately more accepting of a test response that the designers didn’t foresee, “we instill a real culture of collaboration and calibration, and that can sometimes be uncomfortable, because it’s not every faculty member doing their own thing. We’re all in this together and it’s a team determination.” As for the students, there’s no gray area: they need to correctly master all of the competencies to advance.
WGU won’t disclose how much all of this costs, contending that the answer to that question is proprietary. As a nongovernmental nonprofit, it doesn’t have to report such level of detail. But publicly available documents from 2015 show that those 900 full- and part-time evaluators make up more than a quarter of the university’s 3,517 employees, and employee salaries and benefits accounted for more than half of the organization’s $355 million in annual expenses. Whatever the total price of the elaborate assessment process, the average cost to students of a WGU bachelor’s degree is a comparatively low $15,000. In the academic year that just ended, the average annual resident tuition at public four-year universities was $9,650; for nonresidents, $24,930; and at private, nonprofit universities, $33,480, according to the College Board.
Despite the scale of this assessment model—in fact, because of it—Western Governors captures efficiencies, Fowler asserts. For example, if classroom faculty had to conduct and grade assessments in addition to teaching, she says, they’d need to spend more time and be paid more. Separating these roles means that faculty become highly skilled at their specific jobs and “can perform more effectively and more efficiently,” according to Fowler.
But while this assessment model and the use of online education seem to add up to potential savings for students, developing a first-rate CBE program is not cheap, higher-education leaders say.
“Recognize that you do need to invest a great amount of money in doing this right,” says Cathy Sandeen, chancellor of the University of Wisconsin Colleges and University of Wisconsin-Extension, which includes UW Flex.
Sandeen estimates that the university system has spent close to $1 million on developing each of its direct-assessment competency-based programs. “It’s a lot of work to identify those competencies . . . and then how to design the assessments for [them]. It’s not something to go into just because it’s the next shiny object. You have to put some thought into it. Remember MOOCs?” she says, referring to the massive open online courses into which many universities hurdled at full speed, only to find that their moneymaking potential had been overstated.
“This is not easy. It is not cheap. There are high startup costs as well as high operating costs,” echoes Holtschneider.
One of those costs could be for specialized professional development. Few teachers have been trained to work in settings tailored to individualized learning, says Neal Kingston, a professor of educational psychology and director of the Achievement and Assessment Institute at the University of Kansas, who is in the midst of a three-year project studying these issues. “If what you’re used to doing is preparing one lesson for today’s class, but kids are all over the place, that doesn’t work,” he says.
Pulsipher says Western Governors looks at adapting its model for use in lower grades “on a pretty consistent basis.” Secondary-school districts and others, he says, have already asked the university for help adopting it. But so far, he’s said no.
“The short answer is, the timing is: not yet,” Pulsipher says. “We would only want to partner with people who truly have the commitment to stick with this over the long term.”
He uses Western Governors’ own beginnings as an example of this.
“For the first five to ten years, there were a lot of naysayers, a lot of skepticism, a lot of challenges to be overcome. You have to have a high, high, high level of commitment … enough confidence over the long term that you’re willing to be misunderstood in the short term.”
Jon Marcus is higher-education editor at the Hechinger Report, a foundation-supported nonprofit based at Columbia University, and North America correspondent for the (London) Times Higher Education magazine.
This article appeared in the Fall 2017 issue of Education Next. Suggested citation format:
Marcus, J. (2017). Competency-Based Education, Put to the Test: An inside look at learning and assessment at Western Governors University. Education Next, 17(4), 26-33.
