(Image source: MindShift blog)
Stephen Downes just shared this study suggesting that students see a 20% improvement in their test scores on their state exam after using an iPad loaded with HMH Fuse.
I am a supporter for using technology in mathematics education, but it’s probably worth examining these results closer. Here are some quotes from the study itself, and my unpacking of what this means for the reliability of this study.
- “Earhart has been a school eager to employ new technology in the classroom…” (p5):
This suggests a selection bias. Further, it also suggests that this program has been attempted at one school, or that all that has been shared with us are the results from one school. Were there other schools that had an opportunity to pilot this program which have not been shared in this study?
- “Coleman approached his teachers about this opportunity and two teachers, Jackie Davis and Dan Sbur, were ultimately chosen to take part in the study.” (p5) :
This suggests that the process for choosing the teachers was anything but random. The study makes careful mention that the students were carefully chosen, but underplays how teachers were selected.
- “…this meant more work and time required by the teachers…” (p5) “Like any new technology, there was a slight learning curve with adopting a tablet in the classroom. “In the beginning of the year I tried a little bit of everything, trying to find out what was best for my class and for me,” recalls Jackie Davis. Dan Sbur also found that “Over time, it became easier to use and I could use it more in my class as I became comfortable with the device and app.” (p6) :
The teachers who were chosen (or volunteered?) had to work harder to implement this program. This suggests that at least part of the effect on their test scores could be attributed to the efforts their teachers put in.
- “…This meant that students were allowed to take the devices home and “customize them,” adding their own music, videos, and additional apps. This approach also allowed students to have 24/7 access to the HMH Fuse: Algebra I program.” (p5) :
So now, are we measuring the effectiveness of the program, or the effectiveness of time spent learning math? Students who spend much more time working on math are obviously going to see an increase in their test scores.
- “As one would expect, those students who were randomly selected to be part of the HMH Fuse study were very excited – as were their parents. In fact, Coleman quickly found that one benefit of the HMH Fuse: Algebra I app was enabling parents to provide more support to their children: “Parents could watch the videos or review problems with their children to help them if they did not understand.”” (p6) :
Clearly parental involvement makes a difference in a student’s education, and if this app helps parents be more involved, that’s excellent. If this program wasn’t considered so innovative, and new, would parents be as involved? In other words, if we standardized this program, would parents get excited by it?
- “In addition, Mr. Davis found students took the initiative to use HMH Fuse: Algebra I to check their work during class, freeing him up to do more one-on-one work with struggling students in need of individual attention. In this regard, the HMH Fuse app essentially enabled a “flipped classroom” model in which students learned and worked independently at home, and then came to class ready to do problems and practice what they had learned (see Bergmann & Sams, 2011). This “flipped classroom” dynamic gave both Mr. Davis and Mr. Sbur the ability to provide personalized instruction to many students during the normal school day.” (p6) :
If the HMH Fuse app allows students to work in a more self-directed way, that’s a good thing. If their teachers are changing their pedagogical approaches to suit the affordances of the device, that’s probably a good thing too. So one wonders how much of the learning effect was due to this personalized attention. Did the two teachers in this study also find ways to personalize and give individual attention to their students in their other non-iPad classes?
One thing not at all discussed in this study is what they hope to accomplish by improving mathematics instruction. Test scores are one measure we have for mathematical ability, but they are not the only measure. Did this program give students additional time to work on improving their mathematical reasoning and their problem formulating & solving skills? Hopefully the authors of this paper will submit it for formal review so that any of the issues that I’ve addressed can be peer reviewed.