Tag: assessment (page 1 of 2)

June 11, 2013 / 2 Comments

Bias in assessment

Every form of assessment of learning has bias. This bias may be hidden, or it may be quite obvious. As Cathy O’Neil points out, assessment is a proxy for what we want to measure – learning. We cannot measure the building of connections between neurons that is happening in the brain directly (or even potentially understand what that growth even means) so we use a proxy in the form of an assessment of the externally visible signs of learning.

One bias therefore is our tendency to forget that we are not actually measuring learning, we are measuring a proxy for learning.

Another bias is the language we use to do the assessment, whatever form that language takes. Language is necessarily a construct of our minds and no matter the appearance, it has deeply personal meaning based on our own experiences. As such, our use of language to communicate an assessment to a student contains an inherent bias based on our understanding of the language we have used.

The medium of the assessment also introduces bias. An assessment done on paper on pencil is limited to what can be collected in this form. As Dan Meyer and Dave Major are showing, digital assessments have different affordances (and different biases) that can change how students can interact with the assessment. Students who share their understanding verbally may have a very different explanation than if they write down their explanation.

There is further bias in the assessment based on our interpretation of the results of the assessment. If one knows the names (or race) of the people doing the assessment, is one sometimes more or less lenient? Look at this video of people assessing whether or not someone is stealing a bicycle. Do they exhibit any bias? Is it possible that teachers may experience similar bias (hopefully to a lesser degree!) when examining student work? How much of a difference to the student’s marks does it make if the work is messy or neat?

There are no doubt other biases in our assessments that I have not mentioned.

To combat bias, we must be first aware that it exists, and next that we should look at our assessments and ask ourselves, what bias is likely to exist in this form of this assessment? Can I address this bias? If not, can I use a second assessment in a different form and compare results?

March 6, 2013 / 2 Comments

Assessment for learning

The director of our senior school, Brad Smith, is doing a workshop today in our staff meeting on assessment for learning. He’s found the following quotes, which he wants to use in his presentation. I’m helping him edit his presentation (since I have time, and he does not), and I’m hoping to find some photos or other prompts which describe these statements to include in his presentation.

Many argue that “Formative Assessment” is a misleading term and is open to a variety of interpretations. “Assessment for learning” may be preferable and requires 5 elements to be in place:

The provision of effective feedback to students

The active involvement of students in their own learning

The adjustment of teaching to take into account the results of assessment

The recognition of the profound influence assessment has on the motivation and self-esteem of students, both of which are crucial influences on learning

The need for students to be able to assess themselves and understand how to improve

Dylan Wiliam

When a teacher teaches, no matter how well he or she might design a lesson, what a child learns is unpredictable. Children do not always learn what we teach. That is why the most important assessment does not happen at the end of learning – it happens during the learning, when there is still time to do something with the information.

Dylan Wiliam, 2011

The worst scenario is one in which some pupils
who get low marks this time
also got low marks last time
and come to expect to get low marks next time.

This cycle of repeated failure
becomes part of a shared belief
between such students and their teacher.

Black and Wiliam, 1998

The more you teach without finding out who understands the concepts and who doesn’t, the greater the likelihood that only already-proficient students will succeed.

Grant Wiggins, 2006

The initiate-respond-evaluate cycle:

I’ll ask the question,
a few of you will answer
for the entire class,
and we’ll all pretend
this is the same thing as learning.

Fisher and Frey, 2007

If students left the classroom before teachers have made adjustments to their teaching on the basis of what they have learned about the students’ achievement, then they are already playing catch-up. If teachers do not make adjustments before students come back the next day, it is probably too late.

Dylan Wiliam, 2007

For teachers, getting annual test scores several months after taking the test and in most cases long after the students have departed for the summer sends a message: “Here’s the data that would have helped you improve your teaching based on the needs of these students if you would have had it in time, but since it’s late and there’s nothing you can do about it, we’ll just release it to the newspapers so they can editorialize again about how bad our schools are."

Doug Reeves, 1998

If you know of any photos or visual prompts/diagrams which you think may be useful, please let me know. His workshop is this afternoon, but I’m sure the resources will be useful another time as well. If I find resources which are useful, I’ll share them here.

June 8, 2012 / 0 Comments

Imagine something different

See this piece of paper?

Piece of lined paper
(Image credit: D Sharon Pruitt)

Throw it away.

Imagine the limitations of the piece of paper shown above do not influence how you share the record of learning your students have done, with their parents, and the wider community.

Now remember the history of grading, which started with one William Farish (in Western culture – Chinese culture has been apparently giving grades to students for many centuries for the purpose of sorting their children into social classes.). William Farish (re)invented grades as a way to increase the number of students he could "teach’ for the purposes of lining his pockets (at the time, more students meant more money).

What would you do differently to share your student’s evidence of learning, if the limitations of the paper above did not exist, and if your purpose was neither to sort students into social classes or line your pockets by being able to teach more students?

June 4, 2012 / 0 Comments

How does your assessment fit into the big picture?

I would feel more motivated if I knew how my assignment fits into the big picture.
(Image credit: Dilbert comics)

I think this comic speaks for itself. How do your assessments fit into the big picture? Is this clear to your students? (I doubt many educators are giving students assessments that a monkey could be taught to do.)

November 28, 2011 / 0 Comments

Experiments in assessment

Here a few experiments in assessment I’m considering for next year.

Compare the results between an oral assessment (as in, find out what they can tell me they know verbally) and a traditional test. . Question: How much of a difference does the mode of assessment make?
Compare the results between a 10 minute quiz and a full length test. Question: Do I find out significantly more with a longer assessment?
Give my students an assessment where I only give them written feedback and no numbers or check marks. Compare this with an assessment where I only give check marks, and another where I only give numeric feedback. Question: Are the numbers and check marks necessary?

What other experiments would you suggest that I try?

October 27, 2011 / 3 Comments

Grading is a compression algorithm

The objective of traditional grading is to compress information teachers have gathered about a student down into a single score to make understanding the information easier. One of the original reasons for this compression was the limitation on how much information could be shared on a single piece of paper. One of the purposes of comments is to uncompress the grade a bit, so that parents and students have some ideas on how to improve their grades.

This process is used to change the size and quality of pictures as well. Compare the two pictures below, and ask yourself, which one conveys more information?

Is there a way we can share information parents and students can understand, while not reducing the information too much?

October 24, 2011 / 1 Comment

It’s time to redesign the report card

The typical report card looks like this (click to embiggen):

image credit: rutlo

image credit: Richard Giles

image credit: clintjcl

A problem with these reports is they do not share with parents information that can be used to help their children improve their learning. What they share is information that is helpful to rank their children with respect to the other children in their classes. They are essentially an autotopsy of learning, rather than a document which can be used to help students improve.

A child who does "poorly" is rarely given sufficient advice to help them improve via their report card. Most comments from teachers are of the "what did David do wrong" variety, rather than "David should do x to help improve learning." A child who does well on their report card is given a free pass, and rarely pushed to extend themselves. The comments you put on your report cards should be ones that help students improve. Canned report card comments are a waste of time! If teachers do not have time to give appropriate comments for each of their students, that points to a systemic problem with classroom size (and workload) and that can’t be rectified by adding useless comments to report cards that teachers can just select.

I don’t think that online grading systems are the answer either. These lead to situations where teachers are forced into an unhealthy practice (grading everything students do) just so parents can always keep on top of the "progress" of their students. It is counter to the purpose of formative assessment to include it in an overall summative grade, and it is counter to the purpose of a summative assessment to include everything students do. Students also need a bit of freedom from their parents in order to experience this learning process themselves, and having their helicopter parents whirring around all the time checking in on them is counter-productive to them developing their own sense of independence.

We’ve used student led conferences at our school, which are an opportunity for students to show to parents directly, the results of their learning, based on portfolios our students have constructed. We are hoping to eventually have these portfolios be online, and I’d like to see the specific assessments students have done linked from their report cards. These experiences are far more valuable, both for "strong" and "weaker" students. They do have issues; not every parent takes the time to come and see their student’s work, and not every student is able to adequately explain how well they did.

I wonder what a report card that was sent home that just listed student’s (apparent) strengths and weaknesses would look like? Could you send home information that helps students improve, rather than information which helps them be numbered?

Could we design an electronic report card that gave far more information than our current ones do? What would a report card with each assessment criteria for the year, and how well our students did on each look like? Would that be too over-whelming? What if we sent home a link to an eportfolio for each student, with suggestions and comments for how to improve attached to each assessment the student does? What would happen if we gave students more ownership (perhaps with some oversight to start) over how they reported their learning to their parents?

I don’t have the answer to what the report card of the future should look like, but I do know that our current report cards need improvement.

August 12, 2011 / 14 Comments

A problem with averages

This xkcd comic demonstrates a big problem with averages.

The problem with average ratings

By the way, this same problem occurs when you average grades as well.

June 24, 2011 / 1 Comment

Separate process of work from the product of work

When I started working at my current school, I realized that they do something in assessment which is very rare. Our school separates the behaviours and all of the types of things which are part of the process of work from the summative grades which come from the assessments which are the product of work. We end up with an "Approaches to Learning" rubric, which we use to provide feedback to students about the learning habits they’ve developed. Our current rubric has a quality of work column which I think we are planning on deleting for next year, so here is the rubric we will (hopefully) use.

	Quality of effort	Reflection	Remediation	Application to next situation
7	An appropriate level of effort was evident throughout the term. Assignments were done on time. Student is an example to the rest of the class.	Student thinks about the tasks and the results. Accepts responsibility for learning, and looks for ways to improve. Always makes corrections.	Student clarified expectations and did follow-up with teacher when needed. If needed, appropriate steps were taken to fix any difficulties.	Is successful in making changes to approaches to learning to ensure ongoing success. Develops a variety of strategies. Seeks to improve and is successful.
6	An appropriate level of effort was evident, although there was an occasional letup.	Student often reflects on work and quality, and takes responsibility for learning. Usually makes corrections.	Student clarified expectations and did follow-up with teacher when needed.	Makes good effort in improving approaches to learning.
5	Effort was evident, but not always at the level it needed to be at. Occasionally (but rarely), deadlines were missed.	Student is developing a sense of understanding of the ways he or she learns best, and is making effort to improve learning.	Student readily asks for assistance, in class, but does not make much out of class effort to seek help.	Cares about performance and makes effort, sometimes unsuccessful, to improve work.
4	Effort level is inconsistent. Student sometimes misses deadlines, but always gets work in eventually.	Student reflection is only evident when teacher prods. When that occurs, student demonstrates awareness of learning issues. Makes corrections only when required.	Student only seeks help when teacher requires it. However, student takes a positive approach.	Occasionally, student makes purposeful changes in approaches to learning.
3	Student needs constant prodding to get work in, often after the deadline. Most work is handed in.	Student is able, but often unwilling, to be reflective about work. Seldom takes pride in corrections.	Student only seeks help when the teacher requires it, and is not positive about the experience.	Makes limited adjustments to learning, even when faced with poor initial work.
2	Work is seldom handed in, even after much prodding by teacher.	Student does not seem to be able to reflect on learning. Corrections seldom done, and only when pushed by teacher.	Student resists invitations for extra help.	There is limited attempt to make any adjustments, even when initial work is poorly done.
1	Work is not handed in.	Student does not seem to be able to reflect on learning. Corrections never done.	Student never seeks extra help, especially when needed.	Student does not learn from mistakes.

There are some things I like less about this rubric. For example, while I think giving students feedback on their learning habits is useful, I don’t like the number associated with it. The 1 to 7 scale comes from the International Baccalaureate program, and is a mapping from descriptors like "very poor" and "good" to a number system. The temptation to find the average of the numbers from each column in the rubric above is too great, and the result of the average is a too brief summary of the learning habits of students. They often need the whole picture, not the summary.

Somethings I do like about this system a lot is that we often have students self-assess themselves on this rubric, and then we have a conversation about their self-assessment. I also really like that it means that all of those types of assignments which in the past I would have carefully graded and included as part of an average mark for students are now more formative tools. I don’t need to include a student’s marks on quizzes, homework, or small projects in their final grade. I can give them feedback on the assignment, and then keep track of their general learning habits using this rubric. The final summative grade reflects the rigorous assessments of learning I’ve given the students.

In the act of separating the learning habits from the summative grades, we learn a lot about students as well. A student who has strong work habits, but struggles to demonstrate mastery of material is now more obvious, and can receive the support they need. A student who has weak work habits, but is able to produce excellent summative work now gets feedback about their learning processes.

When I assessed students during my time in New York, at our school we were required to give students 20% of their grade for "work habits" (we weren’t allowed to call it participation), 20% for homework, and 60% for the assessments from class (included tests, quizzes, exit slips, whatever). The result was that students with "good" work habits were virtually certain to get a passing grade, even with horrible understanding of the material. The system was designed to pass students along to the next person, rather than verify that they understood what they were learning.

On the summative end of our assessment, for each course we have different criteria we assess. For example, math teacher assess using "Knowledge and Understanding", "Investigating Patterns", "Communication in Mathematics", and "Reflection in Mathematics." To be perfectly honest, it can get complicated for students to understand all of the assessment criteria, given that each is different for each course. In some respect this is similar to what happens when each teacher decides on how they want their work formated, and how it will be graded. However, the advantage of this system is that students get more feedback about what areas they need to improve, rather than just an overall grade. One aspect of this system I really like is that the final grade in each criterion is not an average of the marks the student has received, it is supposed to instead be a snapshot of what we think the kids are capable of doing on a good day, given their previous performance, and we discuss what we mean by snapshot so that there is some consistency across the school.

As much as is possible, I believe we should separate the learning habits students have from their performance on their summative assessments. Giving 0s for missed work, late penalties, etc… all dilute the meaning of a grade so that it doesn’t reflect the learning the student has done. Given that we still work in a system which expects student and teacher accountability through grades, at the very least these grades should have some meaning. A performance grade diluted by aspects of student behaviour just leads to questions like, "What can I do better?" Either it hides poor performance but good work habits, or it hides poor work habits with students who are good at summatize assessments.

January 16, 2011 / 6 Comments

Arbitrary Deadlines

I read this article by Alan Stange on assigning penalties to students who hand in work late. He makes the point at the end of his blog post, "There is in fact relatively little significance to learning to complete on time." I agree with this statement and I’m going to expand upon it.

Who set the deadline for the assignment? Why was that particular deadline set? In most cases I am sure the answer to the first question is the teacher, and the answer to the second question was that because that particular deadline was convenient for the teacher. Some schools have got better answers to the 2nd question in the form of homework and assignment calendars, and if your school isn’t already doing some sort of load-balancing of assignments on students, I recommend it as a good starting place.

When the teacher assigns the deadline for the students, they are sending a message. "I am your boss, you will do as I say," which reinforces the teacher student hierarchy. If the teacher explains the reasons for a particular deadline, the hierarchy still exists, but now the teacher has become the "supportive" boss. If a teacher is willing to extend a deadline for a student, they are now the "empathetic" boss. However, they are still the boss. Do we want to tell students that what is important in your classroom is who is in charge?

Deadlines teachers assign are largely arbirtrary. In some cases they are meant to be logically placed around holidays, end of semesters, or between other teacher’s assignments, but these are arbitrary placements. If a deadline is arbitrary, why are we so stuck as teachers on making sure students meet it? What outcome are we hoping for from students when we are in charge of the deadlines?

If you listen to some educators, they’ll tell you that meeting deadlines teaches responsibility, and that meeting deadlines gives you a sense of purpose. So we might assign deadlines for these two reasons, if either of them was true. As far as I know, there is no research to support either of these claims.

Tell me the last time you met a deadline for a major project, say a curriculum review. Did you feel like had a sense of purpose? Did you feel like it taught you responsibility? It’s not the meeting of the deadline that gave you your sense of accomplishment, it was the completing of the project. Completing things makes us feel good. We feel proud of ourselves when we produce something awesome. It doesn’t matter in most cases if it took us 2 days or 20 weeks (unless you are trying for a speed record), we feel good because we finished.

Furthermore, who can really say that a project is really done? Even a published book has errors, and ends up being republished with revisions. Nothing is really ever done, we just decide we are done working on it when we feel like it is a "finished" project. I’ll probably come back and read this blog post in a year and find things I want to change.

So here’s my challenge to you. If you really feel you must have deadlines for assignments, find reasons why the deadline matters. Make the deadline less arbitrary. For example, "we need to finish these projects by Friday because the Mayor of the city is going to come and see them then" or "We have to have the letters done soon because our friends in Kenya are expecting them."

Alternatively, take the time to discuss the deadlines with the students. Ask them what they think and what would work for them. Remember, to you the deadline is mostly arbitrary, letting the students decide on the deadline won’t make it any less arbitrary, but it will give them some choice.

Our schools are currently intended to produce worker bees, and drones, but not thinkers. Our insistence on enforcing arbitrary deadlines just reinforces the power relationship between students and teachers and prevents students from being able to make their own deadlines and become more self-directed in their learning.