The Fraser Institute released their annual "report" on the Foundation Skills Assessment (FSA, a standardized exam given to 4th and 7th grade students in British Columbia) results. As usual there have been complaints about the validity of the results, and some interesting side stories. I decided to look at the Fraser institute’s results from a particular perspective. I took the FSA data and isolated the "parent income" and available FSA results (out of 10) from the Fraser Institute data. (I could have used the data from the BC government’s education site, but the Fraser Institute data was more convenient to work with.)
After writing a script to scrape the data (download the raw data here) from the PDF provided by the FSA, throwing away schools for which either FSA results or parental income was unknown, I graphed the data.
From the graph it seems clear that there is some sort of relationship between the two variables, but it is not clear how strong the relationship is between family income and FSA scores.
So I’ve calculated a couple of regressions for a linear fit and a logarithmic fit between the two variables. The value of the correlation coefficients are approximately 0.485 and 0.517 respectively (square roots of the r2 values given above), which given the large number of data points is statistically significance, showing that there is a moderate to strong relationship between the mean family income of a school, and the mean FSA scores for a particular school.
This type of analysis has been done before for other standardized tests, most notably the SAT exams in the United States. What this type of analysis shows is that standardized exams like the FSA are better measures of the wealth of a community than the strength of the schools in that community.
The Fraser Institute rankings are a flawed comparison of schools. It is not possible to fairly rank schools of different socio-economic status because of this strong relationship between scores and the family income. What would be more fair would be to re-rank the data and break it down into different sub-classes based on socio-economic status and not try to compare apples and oranges.
What would be even more fair would be to throw out the exams all together. What we really want to know is how effective our schools are at ensuring that our students are successful. Given the enormous complexity of this issue, and the wide variety of variables involved, finding a solution to that issue will be extremely difficult. A standardized exam is like a fast-food approach to collecting data, it is cheap and fast, but not very filling.