I was reading an article on the Huffington post tonight that suggested communities should boycott standardized tests. Some of the comments seemed to support the current tests, with the comment “I took tests in school so why shouldn’t these kids?” and other similar arguments. It occurred to me that maybe these people don’t actually KNOW what a modern standardized test looks like? It then occurred to me that, except for parents who are sent home practice material, that many people outside of education don’t know what the tests look like.
Here’s a link to the August 2010 NY Regent’s exam in Integrated Mathematics. Let’s take a look at it so you can see why I’m opposed to standardized tests.
First the front page.
This is too much text. An entire page of instructions for a single exam? It is difficult to follow a couple of simple instructions, but an entire page is a bit ridiculous. If you’ve tried getting adults to follow some simple instructions in an important email you send them, then you’ll understand my point here.
“Print your name and the name of your school above.” I mean seriously, do they think the kids are stupid? The two lines above say quite clearly “Student name” and “School name”. What the hell else could they mean? Treating kids like they are idiots is not the best way to start off an exam. This could be an instruction for the invigidulator to read out to the students to remind them to include their names. Nothing is worse for a teacher than receiving an exam in which a student has put a lot of work, but has forgotten to include their name. However, it is not necessary to print out instructions which are completely clear from the previous two lines.
You can’t see it this small picture, but at the top and bottom of the exam paper, it says “Integrated Algebra” and at the bottom of the paper, it is written upside down. This is clearly not for the student, it is to help the invidulator of the exam keep track of what exam each student has. It’s probably an anti-cheating measure, but it is a distraction and unnecessary. It says in ENORMOUS BOLD TEXT at the top of the page which exam the student has.
In a big giant box there is a notice on the exam. It says “A graphing calculator and a straight-edge (ruler) must be available for you to use while taking this exam.” This is not really a message to the students, who would be powerless to react to this notice if this exam was in front of them, it is really a notice for the schools administering the exams that they are responsible for ensuring that their students have access to the necessary tools. Instructions like this should be sent to the school, not to the students themselves, or at least should be worded in a less aggressive tone. For example, “You may use a graphing calculator or a straight-edge (ruler) for this exam. All other aids are not permited.”
Now let’s look at the questions themselves.
Questions 1, 4, 7, 18, 20, 23, 24, 32, 34, 38, and 39 are all word problems apparently intended to be based on “real-life” examples. Every one them is an example of a classic @ddmeyer “pseudo-context” problem. They attempt to model real life, but do a very poor job of it. No one would ever try to actually solve any of those problems in real life. By including these types of questions on the exam, textbook companies now have to create similar problems for students and teachers (the majority of whom unfortunately rely on the textbook for their classroom practice exercises) have to teach those similar problems. Now you have a case of a standardized exam reinforcing bad pedagogy.
Worse, these questions all have built in cultural references students have to decipher in order to be able to do the questions properly. When I taught in NY state, I showed kids how to draw a line through the parts of the question they didn’t understand and then try to decipher the actual mathematics they were being asked to do. What is the point in having a description of a problem at all if many of the students who will take the exam won’t understand your descriptions?
The other questions on the exam are all computational style questions. None of them has any attempt made to make them a relevant question, they are apparently uninspired by reality at all. Of course now we have the same problem as the other set of questions! Textbook companies will now have to print questions just like these from the exam, and again, teachers will use them. Guess how many students in NY state will get to solve really interesting, well motivated problems using mathematics? Some will, but I bet most won’t. What a shame.
The first 30 questions of the test are all multiple choice with the other 9 questions being broken into two parts somewhat arbitrarily between the easier short-answer questions, and some harder longer-response questions.
Multiple choice questions are awful. Students choose between similar sounding answers, each of which has been chosen either to match a misconception or miscalculation the students can make. Only one of the answers is correct. These questions are multiple choice for one reason only, because it saves NY state money in marking the exams. If you really wanted to find out what students were learning in the schools, you would use a deeper assessment tool.
The exam is translated into 5 languages other than English. This is a great start, students who read and speak those other languages can now have a better chance to take the exam. However, 170 or so languages on the planet is spoken in New York City. What about the other 164 languages? Why aren’t their translated editions of each exam for those languages? Oh right, it would cost too much money. Having a few translations of the test is not sufficient to addressing the inequity of testing students in something other than their mother tongue.
There are many other flaws with this exam. I have a lot of other questions about choices that have been made in its construction. Why 39 questions, for example? Why not 38? Is this so the exam “fits” into a neat time frame? How has the curriculum been chosen? Is this the best set of things students should learn at this age? Does any of this mathematics have relevance in the real world?
Standardized exams like this NY state assessment are destructive. Poorly constructed exams are used to draw inferences about the schools in the state, provide funding for schools based on performance targets, and soon they will be used to decide how much individual teachers will be paid (if the merit pay reform goes through across the US).
You cannot fairly measure a school using a broken metric.
Kate Nowak says:
Although I agree with your conclusion and am no fan of NY math Regents exams in their current form, I find your analysis dissatisfying. First you say that today’s tests are unlike tests from years ago. Then you find fault with both story problems and abstract exercises. What do you think Regents exams were like years ago? They weren’t authentic project-based portfolios…they consisted of multiple choice questions, extended response questions, with some story problems and some abstract exercises. Do you think any written math test can be a valid assessment of conceptual understanding and/or procedural fluency? If not you should just say so. If so, maybe an example of a good vs a bad question supposedly assessing the same concept would help clarify your meaning.
There are meaningful reasons today’s tests are worse than they used to be, you just didn’t mention them. For example, 1) there is much more content in the curricula of each course, so that to fit in everything that might appear on the exam, we have to teach with greater breadth and loss of depth. 2) The extended response questions on the IA exam are now each worth 2, 3, or 4 points instead of 10 points. That means if a kid rounds her answer wrong, and loses a point, she might lose 25% or 50% of the credit for the question. 3) The scores as reported to parents and on transcripts have a very cloudy meaning, as there are only 87 points available on the exam, and we use a mysterious table to convert a student’s raw score to a scaled score out of 100. 4) The conversion chart is skewed to make it very easy to pass the exam (because the IA exam is a graduation requirement) (you only needed a 35% to pass last June), but very hard to do well. 5) The IA exam is simultaneously supposed to set a minimum bar for graduation, measure program effectiveness, measure student mastery of algebra concepts, and measure teacher effectiveness. No one test can effectively serve all these purposes, so it does none of them well.
January 7, 2011 — 2:02 pm
David Wees says:
I don’t think I actually said the exams were not like those years ago, I just said that the public doesn’t know what the tests are like today. This could be because the exams didn’t have the same weight as in the past, or that the typical person has just forgotten the format of the exams. The exams themselves maybe haven’t changed much, except for the change of courses in NY in 2002. I remember that switch, it didn’t seem to me like it much of a difference either way, the exams still sucked.
Your chief complaint seems to be that I’ve not gone far enough in my analysis, which is reasonable. I agree with your description of the other problems with the exam, which have more to do with how the test is used than how it is constructed.
January 7, 2011 — 4:02 pm
Anonymous says:
The easiest column to write must be about criticizing something to do with education, specifically NY State, even more specifically, the NY State Mathematics Regents Exams. As a high school mathematics teacher, I could write lines upon lines concerning criticism of the exams, but what is the alternative? OK, let’s start with the premise that standardized tests are not indicative of a student’s ability in a particular. One alternative would be to eliminate them, and have each teacher in each school construct their own exam and send the results to some higher authority. Then is it fair to compare the student’s in one school to those of another school?
You, of course, take an easy route and provide NO solution, NO alternative, nothing constructive to add. Perhaps that is because you don’t have an easy solution. Do you honestly think that what you have written has not be thought of or discussed by Mathematics teachers in conferences or around the lunch table. Ironically, some of the questions on today’s exams, have appeared in past exams. One appeared on the January Exam that was from the “good old days” of 1971…back when I took the exam.
Yes, there are problems. Yes, 39 is a strange number of questions. But, honestly, who cares? Does it really matter if there are 39, or 38, or 40 questions on the exam? Do you really care? Have you done ANY research at all to see if there is any indication of performance by the students that can be measured by these exams.
The answer is NO, you haven’t. You just sit in your glass house, thrown stones, and then move on to another topic.
What a waste of your time!
June 29, 2011 — 11:32 am
David Wees says:
Clearly you haven’t read the rest of my blog wherein I describe a few alternatives to standardized testing.
June 29, 2011 — 1:27 pm