I was reading an article on the Huffington post tonight that suggested communities should boycott standardized tests. Some of the comments seemed to support the current tests, with the comment “I took tests in school so why shouldn’t these kids?” and other similar arguments. It occurred to me that maybe these people don’t actually KNOW what a modern standardized test looks like? It then occurred to me that, except for parents who are sent home practice material, that many people outside of education don’t know what the tests look like.

Here’s a link to the August 2010 NY Regent’s exam in Integrated Mathematics. Let’s take a look at it so you can see why I’m opposed to standardized tests.

First the front page.

Front page of August 2010 paperThis is too much text. An entire page of instructions for a single exam? It is difficult to follow a couple of simple instructions, but an entire page is a bit ridiculous. If you’ve tried getting adults to follow some simple instructions in an important email you send them, then you’ll understand my point here.

“Print your name and the name of your school above.” I mean seriously, do they think the kids are stupid? The two lines above say quite clearly “Student name” and “School name”. What the hell else could they mean? Treating kids like they are idiots is not the best way to start off an exam. This could be an instruction for the invigidulator to read out to the students to remind them to include their names. Nothing is worse for a teacher than receiving an exam in which a student has put a lot of work, but has forgotten to include their name. However, it is not necessary to print out instructions which are completely clear from the previous two lines.

You can’t see it this small picture, but at the top and bottom of the exam paper, it says “Integrated Algebra” and at the bottom of the paper, it is written upside down. This is clearly not for the student, it is to help the invidulator of the exam keep track of what exam each student has. It’s probably an anti-cheating measure, but it is a distraction and unnecessary. It says in ENORMOUS BOLD TEXT at the top of the page which exam the student has.

In a big giant box there is a notice on the exam. It says “A graphing calculator and a straight-edge (ruler) must be available for you to use while taking this exam.” This is not really a message to the students, who would be powerless to react to this notice if this exam was in front of them, it is really a notice for the schools administering the exams that they are responsible for ensuring that their students have access to the necessary tools. Instructions like this should be sent to the school, not to the students themselves, or at least should be worded in a less aggressive tone. For example, “You may use a graphing calculator or a straight-edge (ruler) for this exam. All other aids are not permited.”

Now let’s look at the questions themselves.

Questions 1, 4, 7, 18, 20, 23, 24, 32, 34, 38, and 39 are all word problems apparently intended to be based on “real-life” examples. Every one them is an example of a classic @ddmeyer “pseudo-context” problem. They attempt to model real life, but do a very poor job of it. No one would ever try to actually solve any of those problems in real life. By including these types of questions on the exam, textbook companies now have to create similar problems for students and teachers (the majority of whom unfortunately rely on the textbook for their classroom practice exercises) have to teach those similar problems. Now you have a case of a standardized exam reinforcing bad pedagogy.

Worse, these questions all have built in cultural references students have to decipher in order to be able to do the questions properly. When I taught in NY state, I showed kids how to draw a line through the parts of the question they didn’t understand and then try to decipher the actual mathematics they were being asked to do. What is the point in having a description of a problem at all if many of the students who will take the exam won’t understand your descriptions?

The other questions on the exam are all computational style questions. None of them has any attempt made to make them a relevant question, they are apparently uninspired by reality at all. Of course now we have the same problem as the other set of questions! Textbook companies will now have to print questions just like these from the exam, and again, teachers will use them. Guess how many students in NY state will get to solve really interesting, well motivated problems using mathematics? Some will, but I bet most won’t. What a shame.

The first 30 questions of the test are all multiple choice with the other 9 questions being broken into two parts somewhat arbitrarily between the easier short-answer questions, and some harder longer-response questions.

Multiple choice questions are awful. Students choose between similar sounding answers, each of which has been chosen either to match a misconception or miscalculation the students can make. Only one of the answers is correct. These questions are multiple choice for one reason only, because it saves NY state money in marking the exams. If you really wanted to find out what students were learning in the schools, you would use a deeper assessment tool.

The exam is translated into 5 languages other than English. This is a great start, students who read and speak those other languages can now have a better chance to take the exam. However, 170 or so languages on the planet is spoken in New York City. What about the other 164 languages? Why aren’t their translated editions of each exam for those languages? Oh right, it would cost too much money. Having a few translations of the test is not sufficient to addressing the inequity of testing students in something other than their mother tongue.

There are many other flaws with this exam. I have a lot of other questions about choices that have been made in its construction. Why 39 questions, for example? Why not 38? Is this so the exam “fits” into a neat time frame? How has the curriculum been chosen? Is this the best set of things students should learn at this age? Does any of this mathematics have relevance in the real world?

Standardized exams like this NY state assessment are destructive. Poorly constructed exams are used to draw inferences about the schools in the state, provide funding for schools based on performance targets, and soon they will be used to decide how much individual teachers will be paid (if the merit pay reform goes through across the US).

You cannot fairly measure a school using a broken metric.