Mathematics Content and Process Scoring System:

A Multidimensional Approach to the Scoring
of Balanced Assessment Tasks

 

 

Principles and Premises

 

In order to approach the design of balanced assessment packages and the scoring of student work, one must first have a clear view of the nature of mathematics. One must also have a clear idea of the kinds of understandings and skills that we wish to assess in our students. In this section we will describe our views on the structure of the subject of mathematics. In the following sections we will show how this view of the structure of the subject shapes the way we formulate tasks, how we assemble them into balanced assessment packages, and how we score student work on these tasks. We will provide concrete examples of this process, and also document the use of the MCAPS scoring tool.

 

The Way We See the Structure of the Subject

 

Like many subjects, it is possible to identify both content and process dimensions in the subject of mathematics. Unlike many subjects, where most of the process dimension refers to general reasoning, problem-formulating and problem-solving skills, the process dimension in mathematics refers to many skills that are mathematics specific. As a result, many people tend to lump content and process together when speaking about mathematics, calling it all mathematics content.

 

We believe it is important to maintain the distinction between content and process. In part we say this because we believe that this distinction reflects something very deep about the way humans approach mental activity of all sorts. All human languages have grammatical structures that distinguish between noun phrases and verb phrases. They use these structures to express the distinction between objects, and the actions carried out by or on these objects.

 

We believe that the content-process distinction in mathematics is best described by the words object and action. What are the mathematical objects we wish to deal with? What are the mathematical actions that we carry out with these objects? We will try to answer these questions in a way that makes clear the continuity of the subject from the earliest grades through post-secondary mathematics. Seen in the proper light, there are really very few kinds of mathematical objects and actions.

 

The Objects of Mathematics

 

The first set of mathematical objects we need to consider is number and quantity. Indeed, elementary mathematics is largely about these objects, and the actions we carry out with and on them.

 

number/quantity

·       integers [positive and negative whole numbers and zero]

·       rationals [fractions, decimals and all the integers]

·       measures [length, area, volume, time, weight]

·       reals [, e, etc. and all the rationals]

·       complex numbers

·       vectors and matrices

 

Along with number and quantity we introduce very early a concern for another kind of mathematical object, namely shape and space.

 

shape/space

·       topological spaces [concepts of connected and enclosure]

·       metric spaces [with such shapes as lines/segments, polygons, circles, conic sections, etc.]

 

From the beginning we try to make students aware of pattern in the worlds of number and shape. Pattern as a mathematical object matures into function, which is the central mathematical object of the subjects we call algebra and calculus.

 

function/pattern

·       functions on real numbers [linear, quadratic, power, rational, periodic, transcendental]

·       functions on shapes

 

There are several other kinds of mathematical objects that have less prominent roles in the mathematics we expect our youngsters to study. Most important among these are

 

chance/data

·       relative frequency and probability

·       discrete and continuous data  (Some aspects of data collection, organization and presentation can be done in the earliest grades but little, if any, data analysis; notions of probability are not realistically addressable until late middle school. )

 

and

 

arrangement

permutations, combinations, graphs, networks, trees, counting schemes (which at the youngest grades tends to blend with the study of patterns of numbers and shapes.)

 

The Actions of Mathematics

 

The process dimension of mathematics has many actions that are mathematics-specific. It also involves actions that are properly regarded as general problem-formulating, problem-solving, and reasoning skills. We divide these skills into four categories:

 

·       modeling/formulating

·       transforming/manipulating

·       inferring/drawing conclusions

·       communicating

 

With the exception of communication, each of these actions has aspects that are specific to mathematics, and aspects that are not specific to mathematics, but that are quite general in nature. Here is a list of some of these aspects:

modeling/formulating

            domain-general

                        -observation and evidence gathering

                        -necessary and/but not sufficient conditions

                        -analogy and contrast

            domain-specific

                        -deciding, with awareness, what is important,

                           and what can be ignored

                        -formally expressing dependencies, relationships and constraints

transforming/manipulating

            domain-general

                        -understanding “the rules of the game”

                        -understanding the nature of equivalence and identity

            domain-specific

                        -arithmetic computation

                        -symbolic manipulation in algebra and calculus

                        -formal proofs in geometry

inferring/drawing conclusions

            domain-general

                        -shifting point of view

                        -testing conjectures

            domain-specific

                        -exploitation of limiting cases

                        -exploitation of symmetry and invariance

                        -exploitation of “between-ness”

communicating

                                    making a clear argument orally and in writing (using both prose and                                     images)

 

It is evident that there is no reasonable way to separate, nor should there be any interest in separating, the domain-specific and the domain-general aspects of the process dimension of mathematics.

Procedures

 

We now describe how these principles are applied in practical procedures of scoring. This description will 'come to life' when you apply it to specific tasks, rubrics, and examples of

student work .

Weighting of Tasks

As previously mentioned, in order to approach the problem of designing balanced assessment packages in mathematics one must have a clear view of the kinds of understandings and skills that we wish to assess in our students, and the ways in which the tasks we design elicit demonstrable evidence of those skills and understandings.

 Each task is classified according to content domain, i.e. the mathematical objects that are prominent in the accomplishment of the task. Most of our tasks deal predominantly with a single sort of mathematical object, although some deal with two.

 We also consider to what extent the following four kinds of mathematical actions are demanded by the task:

Modeling/Formulating: How well does the student take the presenting statement and formulate the mathematical problem to be solved? Some tasks make minimal demands along these lines. For example, a problem that asks students to calculate the length of the hypotenuse of a right triangle given the lengths of the two legs does not make serious demands. On the other hand, the problem of how many three-inch diameter tennis balls can fit in a (rectangular parallelepiped) box that is 3" ´ 4" ´ 10", while exercising the same Pythagorean muscles in the solution, is rather different in the demands that it makes on students’ ability to formulate problems.

Transforming/Manipulating: How well does the student manipulate the mathematical formalism in which the problem is expressed? This may mean dividing one fraction by another, making a geometric construction, solving an equation or inequality, plotting graphs, or finding the derivative of a function. Most tasks will make some demands along these lines. Indeed, most traditional mathematics assessment consists of problems whose demands are primarily of this sort.

Inferring/Drawing Conclusions:  How well does the student apply the results of his or her manipulation of the formalism to the problem situation that spawned the problem? Traditional assessments often pose problems that make little demand of this sort. For example, students may well be asked to demonstrate that they can multiply the polynomials (x+1) and (x–1), but not be expected to notice (or understand) that the numbers one cell away from the main diagonal of a multiplication table always differ from perfect squares by exactly 1.

Communicating:  How well do students communicate to others what they have done in formulating the problem, manipulating the formalism, and drawing conclusions about the implications of their results?

Since we do not expect each task to make the same kinds of demands on students in each of the four skills/understandings area, we assign a single digit measure of the prominence of that skill/understanding in the problem according to the following scale of weighting codes:

Weighting codes

0    not present at all

1    present in small measure

2    present in moderate measure, and affects solution

3    a prominent presence

4    a dominant presence

Note that these numbers are not measures of student performance, but measures of the demands of the task for a given performance action.

Most tasks will involve these skills and understandings in some combination; different tasks will call differently on these actions. Therefore, in designing tasks it is necessary to pay particular attention to the nature of the demands on performance that the tasks make, and to balance these demands across any collection of materials.

 

For each task, a decision is made as to how the task’s demands are distributed among the content domains

Content domain weighting (Entries must sum to 1)

and among the various sorts of performance actions:

Process weighting  (Each entry is on a 0-4 scale)

Modeling/ Formulating	Transforming/ Manipulating	Inferring/Drawing Conclusions	Communicating

 

Writing Rubrics for Tasks

After a task has weights assigned that reflect the different demands the task makes on a student, it is possible to write scoring rubrics. The rubric is specific to the task, and analyzes performance in the areas we have described.

No one can write rubrics that exhaustively anticipate the richness and variety of student responses. Teachers who use the rubrics we write are urged to use them as a guide where they are helpful, and to use their own good judgment when they find our rubrics not shedding light on their own students’ efforts.

Assessing Student Performance

Once a task is classified according to the mathematical objects that are prominent in the accomplishment of the task and the mathematical actions that are called forth, we can approach the scoring of student work with the following guidelines held firmly in mind:

·       Balanced Assessment packages are designed to allow their users to make informed judgments about both the success of individual students, and the success of instructional programs as a whole.

·       All BA tasks are inherently multidimensional, and thus notions of uni-dimensional ranking of students are inappropriate.

·       Scoring along any single dimensional is at best ordinal, and notions of ratio or even interval scales are inappropriate.

For each task we expect the person scoring the student’s performance to use one of the following performance icons for each of the four different kinds of skill and understanding, or mathematical actions. These performance icons may be thought of as ordinal measures with the following descriptors:

Performance icons

 the student shows little evidence of skill or understanding

[internal code 0]

 the student shows a fragile skill or understanding

[internal code 1]

 the student shows an adequate level of skill or understanding

[internal code 2]

 the student shows a deep and robust level of skill or understanding

[internal code 3]

 

The score sheet for a group of students for a given task might look like this:

Task name	Content domain(s)	Process Weights
		Modeling/ Formulating	Transforming/ Manipulating	Inferring/ Drawing Concl.	Communicating

 

Student scores	M/F	T/M	I/DC	C
Student 1
Student 2
Student 3
Student 4
Student 5

 

The assessor calculates a weighted score for each of the four mathematics actions, and inserts into each of the pertinent cells one of the four performance icons (or their internal codes) described above, based on the student’s performance as delineated in the rubric. This process is described in detail in the scoring example on page 8.

No inference of competence of a student with respect to any mathematical action or mathematical content domain should ever be based on performance on a single task.

Aggregation of Student Information

Assessing the overall performance of individual students in mathematics requires us to record their performance on individual tasks, and to aggregate their performance across a large number of individual tasks while preserving, to as large an extent as possible, the richness of the information yielded by the observations on individual tasks.

One could imagine reporting the complete student record of performance on each task. This volume of information is likely to overwhelm whoever looks at it, and, except in the case of the clinician specifically interested in a particular student, to be of little use to anyone. For example, a complete student record might have this structure:

Student performance by task
	Modeling/ Formulating	Transforming/ Manipulating	Inferring/ Drawing Conclusions	Communicating
Task 1
Task 2
Task 3
Task 4
Task 5

Each cell in this table contains the weighting code of that task on the skill/understanding in question for a given kind of mathematical object. Each cell corresponding to a task the student has tried also contains a performance icon (or its internal code) denoting the quality of his or her performance on that aspect of the task.

Such a body of information might well be useful to teachers for informing instructional decisions and, in addition, can be thought of as a cumulative record of a student’s mathematics activities throughout the school years. However, it may be somewhat inundating for purposes of accountability.

Scoring Example

Here is a comparison of the work of two students on a collection of tasks. The students were asked to choose five tasks. Their choices were to be governed by the following constraints:

·       they could choose no more than two tasks from any single content domain

·       they could choose no more than three “skills” tasks[short tasks, primarily involved with transformation/manipulation.]

·       they must choose at least two “problems” [longer tasks, considerable formulation and inference demand]

Here are the results of the students’ efforts:

Task	Content Domain	M/F Wgt.	Score	T/M Wgt.		Score	I/D Wgt.		Score		C Wgt.	Score
STUDENT 1
L to Scale	ss	0	0	2	3			1		2	1	1
Egyptian Statue	ss	1	2	3	2			2		2	2	2
Bathtub Graph	f	4	3	2	2			0		0	3	3
Gligs & Crocs	f	3	2	2	3			2		3	2	2
Fermi I	n	4	3	1	0			3		3	2	2
			10		10					10		10
STUDENT 2
Egyptian Statue	ss	1	2	3	2			2		2	2	2
Don’t Fence Me In	ss	4	2	1	2			3		2	2	2
Books from Andonov	f	2	2	2	2			2		2	2	2
Melons & Melon Juice	f	3	2	2	2			2		2	2	2
Survey says...	cd	2	2	2	2			2		2	3	2
			10		10					10		10

The leftmost column indicates the name of the task. The next column indicates the predominant mathematical content domain (n = Number and Quantity, ss= Shape and Space, f = Pattern and Function, cd = Chance and Data, a= Arrangement). The bold figures are the weights assigned to each of the mathematical action categories for each of the problems. The figures in italics are the scores (0, 1 or 2 for partial level, 3 for full level) that each student received on that section of each of the problems.

Note that a column sum of the scores at this stage would record the fact that the students were equally competent at modeling, transforming, inferring and communicating, although a teacher might intuitively feel that there were distinctions to be made on the performance of these two students.

To calculate the weighted sum of the first student’s performance on modeling in the domain of Function we note that the first function problem was assigned an M/F weight of 4 and that the student was given a score of 3 (i.e. full level) on it. The second problem in the domain of function was assigned an M/F weight of 3 and the student made partial level on it, receiving a score of  2. Thus, the student received 18 M/F marks (3´4 + 2´3) out of a possible 21 (3´4 + 3´3) in the domain of function, resulting in a decimal score of 0.86.

Similar calculations are done for each cell. Performance icons are then assigned as follows:

 

Now here are the records of these two students aggregated over domains:

Student record aggregated over tasks within content domains

Student 1
action domain	Modeling/ Formulating	Transforming/ Manipulating	Inferring/Drawing  Conclusions	Communicating
Number and Quantity
Shape and Space
Pattern and Function
Chance and Data
Arrangement

 

 

 

Student record aggregated over tasks within content domains

Student 2
action domain	Modeling/ Formulating	Transforming/ Manipulating	Inferring/Drawing  Conclusions	Communicating
Number and Quantity
Shape and Space
Pattern and Function
Chance and Data
Arrangement

 

Note that this presentation clearly reflects the inherent difference in the performance of the two students, which the numerical presentation did not reveal.

It should be stressed that at this level of aggregation there are no longer any numerical measures entered in the cells of the aggregate student record, only the aggregated performance icon appropriate to that cell.

The full power of this method of recording and reporting student performance becomes clear when one has enough data to fill all the cells. Here are the student records of four fourth-grade students aggregated over eleven tasks. These tasks were distributed over the content domains as follows: four Number and Quantity tasks, one Shape and Space task, three Pattern and Function tasks, one Chance and Data and two Arrangement tasks.

Student record aggregated over tasks within content domains

Student A
action domain	Modeling/ Formulating	Transforming/ Manipulating	Inferring/ Drawing  Conclusions	Communicating
Number and Quantity
Shape and Space