Executive Summary
From the 1990s until the end of the federal No Child Left Behind Act (NCLB) in 2015, state and federal education reform policies had a virtually exclusive focus on holding public schools accountable for student test scores in reading and mathematics. The new Every Student Succeeds Act, the successor to NCLB, provides an opening for states to broaden their accountability regimes by including a non-traditional measure along with academic test scores. One possibility that has been embraced by many advocates is some type of measurement of student soft skills, which include social skills, self-management abilities, academic soft skills such as listening carefully to instructions, and approaches to learning such as willingness to take on challenging tasks.
Attention to soft skills among education reformers is presently skewed towards attempts to enhance and measure broad student dispositions that are abstract, context-free, not directly observable, assessed through self-report questionnaires, and dominated by genetic influences. A much more productive approach would emphasize soft skills that are specific, contextual, socially observable, easily malleable within the environment of classrooms and schools, and widely accepted as a responsibility of schools to support. Grit is an example of an abstract soft skill. An example of a non-abstract soft skill is a particular student working hard on challenging math problems during the first quarter reporting period in Mrs. Thomas’ fifth grade class.
Whereas personality inventories in which students report on their personal dispositions are the preferred measures of abstract soft skills, a student report card completed by a teacher is the embodiment of measurement of specific, non-abstract soft skills. This report introduces a worked example of how to measure specific soft skills, The Brookings Soft Skills Report Card, and uses it to illustrate important functions such a low-abstraction approach provides in contrast to the high-abstraction alternative. These functions include the ease with which teachers and other adults who are regularly around individual students can directly observe the soft skills they are expected to support, the clear implications for intervention suggested by low scores on a particular skill by a particular student or group of students, the signals sent to administrators about teachers and groups of students who may need additional help, and the usefulness in communicating with parents.
The thrust of this report is to demonstrate the value of having measures of soft skills that are simple and close to the classroom. Doing so is not incompatible with system-wide measures of soft skills that can be used for monitoring and accountability—this report illustrates how student characteristics captured in a report card frequently produce artifacts in available administrative records that can be used for system-wide accountability. These two efforts—classroom tools to be used by teachers and administrative records to be used by administrators—can proceed in parallel. Both differ from and are superior in education settings to testing children with personality inventories.
Background
This is the third of a series of Evidence Speaks reports on soft skills in K-12 education. The first two provided research reviews and conclusions that set the stage for a consideration of how to measure soft skills in schools.[i] The central takeaways are that:
• The domain of student soft skills as conceptualized by most education reformers is dispersive.
Soft skills that are the targets of present education reforms range broadly both in the type and in level of abstraction. The circumplex of soft skills represented in Figure 1 captures four categories or domains of behavior: social skills, self-management, academic soft skills, and approaches to learning. The vertical dimension, i.e., the height of the column or circumplex, represents abstraction: the degree to which any particular soft skill or soft skill category is specific, contextual, and socially observable (low abstraction) vs. broad, context-free, and available only as a student report of a self-reflection (high abstraction). An example of a low-abstraction soft skill is whether a student is observed by her teacher to finish math homework assignments on time. An example of a high-abstraction soft skill is whether a student reports on a questionnaire that he or she is a reliable worker. This is illustrated in Figure 1 with an example of high and low abstraction within the category of academic soft skills.
Figure 1: The Soft Skills Circumplex
The mashup of multiple categories of soft skills and various levels of abstraction into unitary school reform approaches is problematic.[ii] It leads to program descriptions and mission statements that are all over the waterfront, and to efforts at implementation that lack granularity and pose severe challenges in aligning goals, program content, desired outcomes, and measurement.
We have a critical need for more specificity, i.e., less abstraction, with respect to what soft skills students are to learn in school and for what purposes; when, how, and to whom those skills will be taught; and how the success of those efforts will be defined, measured, and evaluated.
• The broader and more abstract the soft skill that is the focus of a school reform effort, the more likely is the skill to have a dominant genetic basis.
This doesn’t mean that a student’s school environment is irrelevant to abstract soft skills. For instance, it is possible to teach individual students particular ways of behaving, e.g., submitting class assignments on time, that might look to a disinterested observer like a disposition or trait, e.g., conscientiousness. Further, the particular forms that are viewed as socially desirable vary from culture to culture and setting to setting and thus have to be learned by students (e.g., interrupting teachers to ask questions or to express opinions is standard practice in American classrooms whereas Japanese students are expected to be very quiet during class).[iii] And, finally, there are several research studies that demonstrate that what happens in schools and classrooms can impact measures of abstract soft skills.
Nevertheless, when behaviors that denote abstract constructs such as conscientiousness, grit, and growth mindset are observed for individual students in a large number of settings or captured through surveys and questionnaires that focus on generalities (“Do you complete tasks successfully?”), the rank ordering of individuals from high to low will have a substantial genetic component whereas the influence of the shared family or school environment will be weak.
This is demonstrated, for instance, in longitudinal studies that compare identical twins with same-sex fraternal twins reared in the same families or separated through adoption. Siblings who share the same genes (identical twins) end up being substantially more similar on abstract social and emotional traits such as conscientiousness and grit than siblings who share only half their genes (fraternal twins), regardless of whether they are raised in the same families and attend the same schools.[v] Thus, if we know only the genetic relationship between two students, we can make strong predictions about the degree to which they will be similar in an abstract soft skill such as conscientiousness. But the same effort to predict similarity between students will be weak if it is based only on knowledge of whether the students grew up in the same family or attended the same schools. The same is true for IQ.
This means that schools that intend to teach soft skills are rowing against a strong current when they focus on abstract dispositions such conscientiousness, grit, empathy, and the like. The school’s task will be much more easily accomplished if it is focused on soft skills at the lower end of the vertical dimension of abstraction.
• Neither the dispersiveness of soft skill reform efforts nor the high genetic loading of abstract soft skills argues against the importance of incorporating soft skills into the intentional mission of schools and classrooms.
As observation, intuition, empirical research, and a quick examination of the Department of Labor’s occupational employment statistics[vi] will demonstrate, success in life depends on hard skills: the individual’s capabilities in subject matter and tasks that are valued in society and are passed on through formal and informal instruction, e.g., being able to write computer code, or service heating and air conditioning equipment, or cook gourmet meals, or understand market derivatives.
But soft skills are also important, as evident intuitively, through surveys of businesses, and through systematic research reviews: social skills, self-management abilities, emotional and attitudinal approaches, and a host of situation-specific soft skills and knowledge that are ancillary to hard skills are important factors in success in school and in life.
The challenges for schools and those involved in efforts to improve the teaching and learning of soft skills are significant given the nascent nature of the enterprise and the significant gaps in knowledge. Meaningful progress depends on informed modesty about the likely returns on current efforts; greater specificity and more emphasis on context in the curricula and school-level approaches to teaching soft skills; and the development and use of practical assessments that are closely aligned with a specific framework for teaching and learning.
Defining and measuring soft skills
Should schools focus on high-abstraction dispositions or low-abstraction skills?
A disposition is a customary way of behaving that distinguishes one person from others, e.g., does the person stand out as cooperative, assertive, responsible, empathic, conscientious, persistent, agreeable, anxious, etc. across a large variety of settings and tasks? A skill, in contrast, refers to a person’s ability to carry out a particular activity successfully, e.g., giving effective forms of feedback to others, staying on task in the classroom, self-monitoring whether one’s behavior is having the intended effect, engaging in timely and expected social routines, and engaging in anticipatory thinking about automatic behaviors and biased beliefs that lead to trouble.[vii] Skills can be specific to situations, e.g., a student may be very good at staying on task in a computer game and deficient in doing so with math homework. Dispositions, in contrast, are behavioral tendencies that occur across disparate situations.
Dispositions are difficult to teach, not only because they have a strong genetic component, but also because they are, by definition, not tied to specific situations. Skills, in contrast, are typically acquired through specific instructional practices and observation learning so they readily lend themselves to generating relevant instructional approaches. Skills vary on a dimension of complexity/difficulty of acquisition from something so simple that it can be learned by one observation of someone else doing it, e.g., raising a hand in class to ask the teacher a question, to something that takes a lot of time, instruction, and effort to acquire, e.g., self-monitoring and correcting biased beliefs about other people’s motives.
Implicit in the prior discussion and Figure 1 are strong reasons for schools to focus on skills rather than dispositions: Skills can be taught, are typically publicly observable and specific, lend themselves readily to selection based on what the school or teacher intends students to learn, and aren’t heavily constrained by genetics.
What soft skills should be taught?
There is no single correct answer to this question because what is to be taught is a reflection of values and goals. A military school will almost surely have a different set of priorities for the soft skills it tries to inculcate in its students than a school for the performing arts. The answer will also depend on the age of the student and the student’s particular areas of strength and weakness—adolescents have different needs than younger students and students who already are competent in a requisite category of soft skills have different needs than those who are not. Thus, what students in a particular school system or classroom should know and be able to do with respect to soft skills requires conscious and thoughtful decisions by teachers and school leaders. Those decisions are foundational to everything else, including curriculum, measurement, and evaluation.
That said, there is substantial similarity across different types of schools and educational missions with regard to basic soft skills that benefit all students. The basic soft skills discussed below should have a comfortable fit within the explicit or implicit mission of a large proportion of schools and classrooms. The remainder of this report draws lessons on how to measure soft skills from a worked example, The Brookings Soft Skills Report Card (Report Card).
The Report Card, which is presented below, covers four categories of soft skill that most school leaders, teachers, and parents would agree are within the responsibility of schools to monitor and, when necessary, develop: social skills, self-management, academic soft skills, and approaches to learning.
The first of these categories, social skills, includes how a student interacts with other students as observed by teachers and other adults. The second category, self-management, refers to observable manifestations of what has been referred to as executive functions or self-regulation, i.e., the student’s ability to take control over what would otherwise be automatic reactions by planning, focusing attention, reframing experiences, and using mental tools. These cognitive processes are frequently not publicly observable. However, the absence of them is, as, for example, when a student blurts out responses that because of their content and short latency suggest a lack of thoughtfulness. They can also be accessed by teachers through direct questions to students, for example: “What were you thinking when you did that?” The third category, academic soft skills, are both social and cognitive. Their defining feature is their ancillary role in carrying out traditional academic tasks, e.g., the ability to work independently. Finally, the category of approaches to learning includes such things as the student’s engagement in school, pleasure in learning, and anxiety about performance.
The vast majority of the individual item descriptions within each category in the Report Card are articulated in terms of observable behaviors, e.g., bullying, being respectful of teachers. The few that are not involve straightforward attributions about internal states of mind, e.g., a child who acts worried and anxious probably is. The items themselves are of my construction, inspired both by items used on an older generation of social behavior checklists designed by psychologists[viii] and by the categories of soft skill that frequently appear in the literature.[ix] Note that the items on the Report Card are for the purpose of creating a worked example. Schools/districts could very reasonably substitute or add items to fit their particular needs.
The low level of abstraction and high level of observability of the student behaviors to which the Report Card refers have two important practical advantages. The first is that it is easy for teachers and other adults who are regularly around individual students to experience directly what the Report Card asks them to score, e.g., does the child have friends, without the requirement for an investment in training. The second is that low scores on a particular item for a particular student or group of students have obvious implications for intervention. For example, students who have low scores on confidence in abilities and willingness to work hard may benefit from growth mindset training.[x] Students who are frequently late to class or absent from school may need counseling and an intervention with parents. Students who are aggressive with peers and quick to anger may benefit from training on how to think about and reframe the actions of others before reacting automatically.[xi] And so on.
Administrators and teachers can also take advantage of the Report Card items to identify both individual students and classrooms that need additional help, e.g., a classroom in which a lot of students are receiving low scores on self-management skills is a classroom in which the teacher needs help in classroom management.
A Report Card that summarizes scores over a reporting period in school also serves as a tool to inform parents.
What about accountability?
One of the consequences of the high-stakes state assessments that were mandated in NCLB and the requirement for a fifth indicator of school success in the present-day successor of NCLB (The Every Student Succeeds Act) is a preeminent concern among school and district leaders with how to measure student soft skills in a way that lends itself to grading teachers and schools.
The Report Card and anything built on a similar template is not intended for or designed to be a high-stakes assessment. That is why it is called a report card rather than an assessment. It is designed to support individual teachers in the task of thoughtful observation of students to identify their strengths and weaknesses in terms of soft skills, and thereby to aid in efforts to help students.
It is also designed to be useful for parents, e.g., sent home at the end of each quarter with the average score for the student for each item indicated, and with notes from the teacher where relevant.
Schools already send home report cards with elements of the Brookings Report Card on them, so this is an expected form of communication. Further, districts/schools often provide information to parents on how to build skills at home, e.g. resources for literacy. They could provide information for parents on a website with direct mapping to the items on the Report Card. This opens the door to inexpensive, low-tech ways parents and schools can help build soft skills.
Of course, there are legitimate needs at the school building, district, and state levels for information on soft skills that can be used for monitoring and accountability. The Report Card is not designed for that, but it provides a framework for thinking about how to go about creating summative measures that can be used for accountability.
The task with respect to accountability is to look at each item or related set of items on the Report Card and ask whether there are administrative data that could serve as an indicator of what the Report Card item describes. In several instances, there will be. With respect to self-management, for example, administrative reports of disciplinary infractions, referrals to the principals’ office, and the like could serve.
There are several possibilities for using administrative data for accountability with respect to academic soft skills. For instance, districts or states could produce a measure applicable to the school or grade level of the proportion of students who are underperforming on state or district academic tests in a particular year with respect to the scores that are predicted for those students from a regression formula that includes demographic information and past performance. Administrative records on late arrivals and absences could capture information at a system level that is similar to what teachers are asked to observe for their own students on the Report Card.[xii] The number of items completed on state tests can be a powerful and unobtrusive measure of what teachers address on the Report Card through the question about whether the student completes assigned tasks. [xiii]
Outcomes related to the Report Card category of approaches to learning can captured in administrative data on student participation in extracurricular activities such as clubs, sports, and music.
Critical for measures of soft skills used for accountability is that they not be easily gamed and that they reflect outcomes that are important. The examples given here with respect to administrative data have these characteristics. Scores on the Report Card can be gamed, which is why the tool is not suitable for high-stakes accountability.
What about psychometrics?
Statistical analysis of the scores generated by the Report Card should be descriptive along dimensions that are directly related to practical action. For the classroom teacher, this might take the form, for example, of a roster of students who are flagged because they are consistently receiving low scores on a particular Report Card item or group of items, e.g., having friends. The teacher could then take steps to address this problem. At the level of the principal’s office, the focus might be on percentages of students with problematic scores by category by grade and classroom. This information could inform decisions on the need for extra efforts in some categories. For instance, if a significant proportion of students in the school are reported as not enjoying school, that would be a call to action. Or if a lot of students were reported as having anger control problems it would suggest the need to provide assistance both to teachers with respect to classroom management and to students with respect to self-management.
It would be straightforward for anyone with the relevant skill set to turn the Report Card into an assessment scale. This would involve, for example, factor analyzing a corpus of completed Report Cards to identify the dimensions that account for the most variance and the items that clump together in terms of providing similar information. This could lead to subscales and to the improvement and substitution of items to generate better psychometric values. So scaled, it would be easy to develop teacher and school scores, norms across a school district, and goals for statistically significant improvements over time.
That said, to anyone interested in turning the Report Card into an assessment scale: Please don’t (unless your intent is to use the resulting instrument only for research purposes). As soon as the Report Card is turned into a test in which a teacher learns not that a student is having trouble making friends but rather that the student is at the 18th percentile for the district in terms of sociability; or not that four particular students in her class are frequently late or absent but rather that the classroom is at the 40th percentile on the dimension of student timeliness, the function of the Report Card is lost.
The principal psychometric tasks with respect to the Report Card items are face validity and test-retest reliability. Test-retest reliability over short periods of time is the preeminent psychometric question for report card items because the data are not useful if scores that teachers generate for individual students on individual items are unstable during a period of time in which it is unlikely that the student has changed. For example, we would not expect a student who receives a low score from a teacher on October 14 for the ability to listen to and follow teacher directions to receive a high score from the same teacher for the same item on October 28. For items reflecting skills that take time to develop, changes on the Report Card should be gradual rather than sudden.
With regard to face validity, good report card items should capture things that are by general consensus in the school and community important and foundational in their own right. So, for example, if parents and teachers agree that students should have friends or meet deadlines or dog paddle for five minutes and a teacher observes using a report card whether or not they are able to do these things, then the report card has high face validity. In this context, a traditional psychometric concern with predictive validity, e.g., whether answers to assessment items predict other behaviors in other situations, is not primary—having friends is the end goal assessed by the report card, not having friends as a predictor of something else.
Conclusions
We are at the very beginning of understanding what educators should be doing in schools to advance students’ soft skills, how the outcomes of those efforts can be measured, and who should be held responsible for what, and how. The present report focuses on measurement. The recommendation, exemplified through the worked example of The Brookings Soft Skills Report Card, is to use measures of soft skills that are naturally occurring, easily observed, at low levels of abstraction, relevant to the expressed mission and instructional goals of a teacher or school, and useful as feedback at the classroom and parental levels. This recommendation pushes in a very different direction from the current embrace of survey instruments such as the Grit scale that are intended to capture individual differences in abstract student dispositions. Districts are being pitched a lot of metrics by vendors for the ESSA 5th indicator. These metrics are not only of questionable utility for reasons discussed here, but they are expensive. What is proposed here is organic and easier and nearly free.
There are clear implications for schools of the choice of high- vs. low-abstraction measuring tools for soft skills. The thrust of this report is to keep it simple and close to the classroom. Doing so is not incompatible with system-wide measures of soft skills that can be used for monitoring and accountability—this report illustrates how the student outcomes captured in a report card format frequently have parallels in available administrative records that can be used for accountability. These two efforts—classroom tools to be used by teachers and administrative records to be used by administrators—can proceed in parallel. Both differ from and are superior for use in education settings to testing children with psychological instruments that are forms of personality inventories.
—Grover J. “Russ” Whitehurst
Russ Whitehurst is a Senior Fellow in the Center on Children and Families in the Economic Studies program at the Brookings Institution.
This post originally appeared as part of Evidence Speaks, a weekly series of reports and notes by a standing panel of researchers under the editorship of Russ Whitehurst.
Notes:
[i] https://www.brookings.edu/wp-content/uploads/2016/07/Download-the-paper2.pdf ; https://www.brookings.edu/research/more-on-soft-skills-time-to-flit-the-grit/
[ii] As an example, the California Core Districts include within their Social-Emotional/Culture-Climate Domain soft skills as disparate in abstraction and focus as suspension/expulsion rates and “the ability to take the perspective of an empathize with others from diverse backgrounds and cultures”: http://coredistricts.org/core-index/
[iii] http://leo.stcloudstate.edu/kaleidoscope/volume3/cultureshock.html
[iv] http://scholar.harvard.edu/files/mkraft/files/teaching_for_tomorrows_economy_-_final_public.pdf
[v] http://psycnet.apa.org/psycarticles/2016-06824-001.pdf
[vii] Daniel Kahneman. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.
[viii] http://pbismissouri.org/wp-content/uploads/2016/08/CaldarellaMerrell1997.pdf
[ix] https://consortium.uchicago.edu/sites/default/files/publications/Noncognitive%20Report.pdf
[x] https://www.mindsetkit.org/
[xi] http://www.hamiltonproject.org/assets/legacy/files/downloads_and_links/v10_THP_LudwigDiscPaper.pdf
[xii] http://www.hamiltonproject.org/papers/lessons_for_broadening_school_accountability_under_the_every_student_succee
