One of the ways computers are being used in math education is to provide students with online practice. There are a bunch of serious problems with most of these programs.

Here is one example from the Khan Academy (apparently at least one of the flaws outlined below no longer applies to the Khan Academy. But that same flaw still applies to IXL. And Prodigy Math. And a thousand other practice apps out there.)

**Feedback is terrible or nonexistent**

How does everyone feel about those online maths practice things that only let you progress if you get a streak of several completely correct in a row? #MTBoS

— David Butler (@DavidKButlerUoA) November 27, 2017

Many programs will, as David points out above, only allow a student to progress after they have gotten a certain number in a row correct. But if a student is struggling to complete an activity and the feedback to the student is terrible, how exactly are students meant to achieve the streaks necessary in order to advance?

Note: Watching a video of a concept isn’t feedback if the learner has already watched that video before. That’s information the learner already has.

**Impossible to see patterns**

One way that people learn math is by observing patterns in their work or solution strategies as they work on a set of problems in a row.For example, what pattern do you see if you try the following exercise:

5 x 3 = ?

4 x 3 = ?

3 x 3 = ?

2 x 3 = ?

1 x 3 = ?

0 x 3 = ?

-1 x 3 = ?

-2 x 3 = ?

-3 x 3 = ?

I’ve tried this with students and most of them notice that they are subtracting each time to find the next product, and so then they make a leap and decide that -1 x 3 must be -3.

But if an online practice program only ever shows one question at a time and the numbers for these questions are selected randomly, there will be very limited opportunity for students to notice and subsequently use any patterns that emerge.

**Blocked practice**

Except for a handful of studies, there is a lot of research that suggests that for most people, if the goal is to remember some mathematical idea, practicing topics in blocks will take longer than if different topics are interleaved together. Almost all of the programs out there focus on students practicing discrete topics. Caveat: I did read a study recently that suggested that for students entering a course with weak prior performance, while interleaved instruction was beneficial, interleaved practice was less effective for these students than blocked practice. Further caveat: I cannot find the link to this study.

**Too easy or too hard**

For some students the exercises are too easy. Sometimes this is because kids select easier problems for themselves, sometimes this is because students already know a bunch of mathematics and do not need this particular practice activity. Either way, needing to work through a streak of 5 or 10 problems just to be able to move on is ineffective for these students.

For other students the exercises are too hard. A student who really doesn’t know a particular area of mathematics doesn’t benefit from practice in that area – they need teaching or access to information.

**Inappropriate medium**

For many, many math problems, the best choice of a medium to work on the problem is a piece of paper. Or maybe the best choice for working on a particular problem is a programming language.

These online systems offer neither. This means students are often working in a possibly unfamiliar medium without the most useful tools available for them to work.

This also restricts the people who design questions for the system as they end up likely severely restricted as to what kinds of questions they can ask if they need the answer to the problem assessed by a computer.

**It obscures information from teachers**

If you are a teacher and you are using one of these online systems for your students to practice, there is usually a dashboard you can look at to see how well your students are doing with a particular exercise. But these dashboards truncate an enormous amount of information about the progress of learning and actually make it harder for you as a teacher to gather the information you need to be able to act to improve your students’ learning.

They also are likely to lead to teachers looking for students making mistakes instead of looking for student conceptions, which promotes a deficit view of students instead of treating students as sense-makers.

**It can lead to bad practice**

Virginia Tech has an online remedial math program where students go to sit at a terminal and watch videos on math and then take quizzes on what they learned, over and over again. There is a Facebook post where almost all of the students complain about how much they hate this mathematics class. If the online practice programs did not exist, neither would this course.

Teach to One uses a computer online practice program to inform teachers when small group instruction should occur. But in this middle to upper class neighbourhood, parents revolted and the program was scrapped. But what about districts where parents have less power?

Dan Meyer outlines the many problems with Rocketship Learning Labs, another personalized learning model, in this post.

**It isn’t really mathematics**

If you ask a mathematician or anyone who uses mathematics regularly what mathematics is, literally none of them will answer “it is a series of multiple choice questions or short response questions asked and answered on a computer screen.” While practicing mathematics is a decent way to get better at what you know how to do, it isn’t really the goal of teaching students mathematics.

If answering a series of problems is the only experience of math that students have, they are likely to end up with a very limited definition of what mathematics actually is.

Note, I’m not opposed to students practicing math at all. This is obviously an example of good practice and there is plenty of research to support this claim. I’m opposed to this being the *primary* experience of math that students have.

**Conclusion**

If you can possibly avoid it, don’t use these programs. Or at least try the program yourself for a couple of hours to see what the experience is really like for students. And if you are a designer of one of these online computer practice systems, for the love of God please do a better job than the industry currently is.

Feedback is terrible or nonexistentAgreed. No worse than a textbook w/ an answer key in the back to check (KA tells you if right or wrong). Better than a worksheet w/ no answer key to check.

Obviously worse than a teacher working beside you.

Impossible to see patternsAgreed

Blocked practiceI have my students use the blocked practice to practice a skill the first time. Later I encourage them to do KA’s “mastery” quizzes which interleaves concepts they’ve practiced over time to help with retention.

So some of these offer both blocked & interleaved options.

Too easy or too hardI use the progress tab on KA to look at a series of practice sets. Students that still haven’t mastered the first on the topic from back in September practice that again. Students that have shown competence with more skills will be assigned the next in the progression for their needs. I can do this student-by-student on the KA dashboard so that they each get what they need to practice next.

Inappropriate mediumNo worse than a textbook. My students solve the KA problems on paper with pencil and then input their answer to see if their right … same as they used to do w/ textbook practice.

It obscures information from teachersYes – it doesn’t show me the student’s work like when I used to pick up paper copies. But I also no longer spend hours checking/correcting homework. Instead I use that time to better prepare the in class activities we do where I am able to offer in the moment feedback while they work on problem solving.

It isn’t really mathematicsAgain, no worse than a textbook.

Obviously worse than the problem-solving activities I run in class …

…. But here’s the thing: Khan Academy [or insert other online practice medium] is not meant to replace a teacher or a math class. I use it as a tool to allow my students independent practice like they used to do w/ a textbook. I think practice is useful to my students, and sometimes they do need to practice skills one at a time in addition to the problem-solving we do in small groups all the time in my class.

I’m not a fan of these blanket statements that these online tools are totally horrible and can’t be used in helpful ways. Is Khan Academy perfect? Far from it – but nor was the textbook my school used to offer us to use. Here are the things I like about it:

The report it generates is a useful tool to communicate home about students that aren’t practicing (b/c I believe that independent practice is still important)

That I can differentiate who gets assigned which exercise set.

That students can work ahead & KA can even predict what the next skill in their progression might be

That students don’t lug home a textbook each night

That it has videos (even if I don’t always love his strategies) a kid can watch when stuck

That I no longer have to spend time marking homework for correctness or completion

Probably more reasons too that aren’t coming to mind at the moment …

KA is not a valid replacement for good Math activities and teaching. But is it a useful tool to offer independent practice to students? My argument would be yes. Does that make me a bad teacher?

[…] Since I’ve been using Khan Academy, an online math practice website, with my students for a few years now, I was intrigued & promptly read the blog post. […]

It sounds like Laura’s points fall under the heading “do no harm.” I subscribe to this mantra a lot when trying new things out with students. Do no harm is not a standard that guarantees improvement – but it’s still a useful benchmark when you have competing priorities.

So bringing this mantra to independent (I’m assuming at-home) practice, “do no harm” would be, is it better than doing nothing, and is it worse than other available alternatives.

I like some other alternatives better: like open middle, which I think does great at maximizing reasoning skills in a small space. BUT it also includes zero feedback except what students give themselves. And while some students’ reflective commentary is enough to move them forward, others get stumped and frustrated, and while they’re still getting practice, it’s also no better than textbook problems for those kids. so again, it passes the ‘do no harm’ test, and also passes another test: offers something more. But that ‘something more’ (in the case of open middle, opportunity to build new connections as students start to strategize) is not a guarantee for all students.

So I can see how online practice would lie smack in between “do no harm” and “offers something more” if there’s no useful feedback offered, if the problems are grouped into cookie cutter copies, and if there’s not explicit connections being drawn. These are all also problems with a textbook alternative – which Laura clearly argues above.

Knowing how challenging it is to get students to value math, to work on math outside of school, to get help when the teacher isn’t there, and to have homework or practice as a tool for preparing them to engage in the next day’s learning, I think my feelings about independent practice are that independent practice should do no harm, particularly when it comes to students’ mathematical identities or definition of mathematics as a field. the student that has mastered a concept should not be doing the same 40 problems that others do who are still struggling to understand. Meanwhile, the kid who is struggling to understand probably shouldn’t be doing 40 problems, because they’ll either do them all wrong without useful feedback, or they’ll get super discouraged looking at all that work to do, or both.

If the standard is “does it secure ways for students to learn in ways they wouldn’t have on ‘ordinary textbook problems,'” then I agree with all of David’s points. There’s no good replacement for well-crafted activities facilitated by a more knowledgeable other, like a great teacher. While I don’t think that’s the argument David was making, I DO think that’s a point that needs to be made louder in schools – particularly those who have been romanced by the idea of “personalized learning” through these computer generated learning programs. I’m a skeptic at best, and a hater at worst.

I also think Laura also has a great point about this other competing priority: the teacher’s time. That’s a finite, and important, resource for kids’ learning, in that time spent on planning great lessons is NOT being spent marking homework. that’s a very real trade-off.

Finally, the pragmatist in me has to speak out: ‘practice’ may not be real mathematics. But in a world where students are ranked and valued, sorted and shuffled, admitted and rejected, based on their mastery of discrete skills in time for a test at the end of the year, I can’t in good conscience dismiss the value of even crappy practice, so long as that crappy practice isn’t infringing on juicy learning time in math class. I may not agree in principle with the ‘math’ kids are tested on. But in the meantime, they are.

Okay, I was sure I replied to this.

Regarding: “do no harm,” I can tell you that a parent told me that her kiddo knows how to click around and get through Khan Academy without ever actually learning anything (basically, it’ll give you the answer after enough attmepts).

She says her kiddo is pretty ingrained with the idea that math is something you click through, not something you learn.

I’d not count that as “no harm.”

Did my other comment not pass moderation?

Susan,

It looks like my blog is eating some comments for some unknown reason unfortunately. I don’t know why. Sorry! It is not your fault, I need to figure out what is going on with my blog.

David

Susan. That just sounds like poor implementation by the teacher, though, yes? I mean, we could equally say we know students who think of math as writing down convincing numbers on a worksheet and hoping for a check mark…

Also: that isn’t how Khan Academy problem sets work. You can click through to see “hints”, which are step-by-step work throughs of the problem, which naturally provide an answer by the last step, yes. But if a student uses hints then that problem is not marked correct.

(oops, other comment… probably one I didn’t actually send tho’ I remember figuring out the captcha math 😉

W/ Khan Academy and several other online math practice, I keep tripping over errors, too. For practice I send people to math-drills for worksheets with answers.

I have the luxury of being able to refuse to say “okay, I have to go along with the horribleness of math instruction, and this online procedural drill is what they need to pass, so I’ll put them on it.” I’ve often wondered if I’d have stayed with this job if we didn’t have really good teachers who worked w/ students coming in with very limited skills to build understanding and confidence. Yes, we use online practice — but it’s ALEKS which has much better quality control and design, even if it’s still basically 100% procedural in its “explanations.”

That makes a huge difference.

In another course we use a different platform that’s sloppy and sprinkled w/ errors… and it really does impede the learning. The students who don’t need much practice can figure out the glitches; the others who get stuck… stay stuck. So… it reinforces that “some people just don’t get math” idea.

One nice feature of worksheets is that you have lots of prior worked examples you’ve just done that you can reference when trying to solve a new problem.

Also, I think my argument could be more nuanced, “Online Practice Programs With These 8 Features Are Terrible Practice” but nuance is hard to communicate over the Internet.

I’m all for practice, but I really wish the actual programs people are using were better than they currently are.

Susan, I’ve been assigning practice on Khan Academy (and completing the problems myself) for years. I’ve gone through probably half of the Math content currently on the site. I’ve run into a few technical quirks on some of the problems that involve manipulating points or lines, but I don’t recall seeing any errors.

And, I know, I know, it feels like I’m spamming with three separate comments here, but it might be worth your time to check on some of the improvements Khan Academy has made? The fact that you mention five question streaks as if they are a current feature implies pretty strongly that you haven’t been to the site this year.

I’d also argue they’ve improved their feedback system. Though it’s far from perfect, it’s significantly better than nothing.

And, I’ll stop there. Laura Wheeler argues the rest of my response better than I would have.

Justin,

I have not been to the site this year. But over the last ten years I’ve tried pretty much every online practice programs that exist in the hope that some of them would not have these flaws. They almost all do. This post really was not intended to call out the Khan Academy per se but to introduce the idea that many of online practice programs have serious flaws and most should not be used with kids.

David

There is no need for on-line programs to have limited range of multi-choice. I have seen ones with free answer, and others with multi-choice out of 20 or more. To set the test with only four possible answers is laziness on the part of the designers. (I should add that I have programmed my own electronic revision sheets, and they are never multi-choice.)

The program my current school uses also give feedback on which students are clicking around to find the answer. Again, only a lazy designer doesn’t make it obvious to teachers which students are just guessing.

The too easy/too hard distinction never bothers me, because all my set homework is easy (except for seniors). If you set hard homework you always risk them learning the wrong things. All homework should just be an opportunity to see the material one more time. If they can get it right in 5 minutes, all the better. Hard problems are done in class, or for topic revision on a sheet with fully worked answers for them to follow.

As for “really Mathematics”, do real mathematicians not learn core knowledge thoroughly? Do they need to look up terms like “term” and “expression” every time they see them? Do they learn a skill and never practice it? So I make no apology for making my students do routine things for homework.

Chester,

I’m not opposed to practice. I’m opposed to that being the primary experience of learning math that students have.

David

I might add, I’m not a huge fan of electronic homework. But then I’m not much a fan of homework anyway.

David,

~ Regarding drill practice, generally ~

When math class is nothing but drill practice, it’s drudgery. When the MTBoS is at its best, its brilliance is shown in our collective ability to meet the drill practice needs of our students — preparing them for the poorly-designed state tests most of us face — while making that drill practice a subplot of a larger narrative arc of meaning or discovery.

Wishing that our job description didn’t include covering the standards isn’t too helpful. We are, after all, at the mercy of our employers.

And neglecting drill practice is simply not an option in states where students must pass a state test for graduation. Students must be given adequate drill practice to prepare for that test. Even if only the weakest 10-20% of students are at risk of failing the test, planning should begin with their needs in mind.

What I find helpful in blog posts is when people suggest a way to layer meaning atop drill. I know you’re very good at this, because you make a point of hiding meaningful patterns among sequences of drill problems. I always learn from your posts on that theme.

~ Regarding online drill practice ~

The MTBoS is full of great teachers. I learn so much every day from all of us on here. There’s one thing, though, that I think continually gets short shrift: the learning speeds in a typical classroom vary by a factor of 50-100. The top 4 kids in a class can here something explained once, for 5 minutes, and remember it for the rest of the year. The bottom 15% may not even truly hear that fact, or be aware that it was presented, until they’ve gotten related questions wrong on repeated quiz retakes. And even then it may take a week or two of intermittent practice to encode the fact in long-term memory.

Generally, teachers march through a curriculum as if students were following behind in relative cohesion. Teachers know some students are falling behind, but they don’t do much to adjust instruction to that fact. If you ask them about their plan to address stragglers, they’ll probably say something like, “I sit those students in the front and help them at their desk frequently”. That’s great, but while the top students are learning to solve tough literal equations, the weakest students may still have significant problems with the conceptual foundation of that skill, GEMDAS. Those weakest students do need to try some literal equations, but they also need ongoing help with GEMDAS.

It’s not the teacher’s fault that these weaker students are weak right now — it’s the way life is. Some students transfer into your school in late December with little knowledge of what you’ve taught so far this year. Some students have emotional disabilities or personal problems that cause their effort to vary dramatically from month to month. Some kids have learning disabilities, language barriers, etc. I’m assuming you deal with these issues, too, David. Whatever the reason, we can’t just leave those kids behind. They actually can learn, as long as instruction is designed for their situation.

If it’s late November, we have to have a plan to give these kids the highlights of what we covered in Sept, Oct, and Nov (the power standards), followed by a plan to give them everything else they’re missing. Spiraled homework doesn’t help, because it’s super demoralizing for a kid to have to learn 20 new topics to complete a single night’s homework. You need blocked practice, followed by spiraled review, and the review for a particular kid should mostly cover the things that particular kid has already successfully learned. Meanwhile, you need to continue doing whole-class conceptual lessons, 3-Acts, etc. You need to balance all of these needs, each week. That’s why the job feels so impossible sometimes.

I really don’t see a way to do this sustainably without technology. You’d have to design and hand out different worksheets to different kids every day for homework. I say this as someone who designed and uses a paper-based differentiation system:

https://ijkijkevin.wordpress.com/2016/10/21/burn-the-computer-personalized-practice-or-assessment-on-paper/

Even with that system, I think Khan Academy is a great help with the blocked practice.

And Khan Academy still has so many things they need to do better! For example, right now if you assign a 7th grade skill to a high school algebra student, the system won’t track their retention of that skill. The memory retention quizzes (called “Mastery Checks”) only track mastery of skills in the current grade level.

Anyways, when you try to actually adjust teaching and practice to the time scales on which different students learn, I really don’t see a way to avoid some form of online practice.

What off-line solutions have you found that work for this?

PS – please don’t think that my having settled on this approach means I have it all worked out. It’s easy to present one’s classroom in writing as if it’s perfectly smooth sailing, even if the reality doesn’t match that. But I’ve had enough failures and successes to become convinced that if I want to meet students’ needs, this is the direction in which I should be growing.

Hi Kevin,

Thank you for the very thoughtful responses. I think that it would be really nice if online practice was a nice way to mitigate the issue you describe, which is clearly a very real problem of practice basically all teachers face. However, I worry that the weakest students are the ones least well served by the current online practice systems.

There are some fundamental ideas in any math course that if kids remember, they’ll do better in the course. I know a lot of teachers are successful at getting all of their kids to understand these ideas for a day but then many kids quickly forget. So it may be that most of the variation we see in student understanding is mostly kids forgetting things at different rates.

Here is an alternative practice system that I think would be relatively easy to implement and easy to see if it works. Give kids index cards. On one side they write some piece of information and explain it well enough, with your help, that the idea makes sense to them on the index card. On the other side is a question that asks students to recall this information. Students then get 3-4 minutes a day to look at the cards and self-quiz themselves on the ideas. If they want to check if they understand, they just flip the card. Kids could also partner up and quiz each other. As the year progresses, so do the stacks, but if you don’t vary the order of the cards then as the stacks get taller, kids end up spacing out how often they attempt retrieval practice with each idea.

Another thing we do in our project is to suggest teachers use a variety of pedagogical strategies that aim to give all kids access to the mathematical ideas in a lesson. These are meant to increase the odds that kids learn the things we’d like them to learn from our lessons.

Neither of these is a silver bullet but both have some decent evidence to support their use, in contrast to the online practice programs which as in the study you shared via Twitter, appear to be mixed results at best.

David

[…] (Want more? David Wees shared some additional thoughts in a recent blog post.) […]

I’ve actually considered creating a flash card system like you describe, but I think it has to be layered on top of a drill practice regimen.

I’m not sure weaker students would get anything out of verbally describing how they’d solve something like -3x+2x-5=4(x+1).

Most of their mistakes are in little things, like not distributing the 4 to the 1, or combining -3x and 2x to get -5x. So if their practice consists of narrating, like, “First I’d distribute the 4. Then combine like terms on the left. Then get all the x’s on one side. Then finish solving”, I’m not sure that review would help. They’d have to actually solve the problems.

In addition, weaker Algebra 1 students take 3-4 minutes to solve something like 2x-3+5x=4(x+1), so if they were actually solving the problems out, they’d spend all the allotted review time on a single question. You’d need much more time than 3-4 minutes.

You also need many more problems, say, 8 flash cards for each skill. Otherwise kids are just re-solving the same questions over and over. If you have 30 problem types for the year, you’re talking 240 flash cards. I’m not opposed to this, but you’re kind of recreating online practice in full, but without the reporting system or the unfortunate drawbacks of computer interfaces.

What worries me is students’ low standards of a correct answer, eg, “Well, I got it right except for the negative sign.” Occasional mistakes are ok, but if they happen most of the time, you either have a misunderstanding or an incorrect habit.

Here are my 3 tips for using online practice in class:

1) The system needs a low but reasonable standard of accuracy. The 70% standard on Khan Academy seems like a good middle ground.

2) Use only paper-pencil practice for the first 1-2 days of learning a skill. The online practice is to catch all those little mistakes and force attention to accuracy (and retention) once the main idea is clear. But for initial learning, too much immediate right/wrong feedback can be demoralizing.

3) Never allow an online system to choose autonomously what kids will work on. That should be the teacher’s decision. For example, Khan Academy has “assignments” that let you select what kids will work on. The online practice should develop skills that you have already covered in class. Kids’ first introduction to a topic should be conceptual, meaningful, and lively — not procedural or recorded. And it should connect to what the kids already know, which computers can’t seem to figure out yet. (The exception is when you have kids who have already mastered all the assigned skills. At that point, you could let them explore the system on their own. But I still prefer to direct their extra credit work rather than letting them wander about).

Finally, you point out that the best online programs have mixed results. They do! That’s quite different from saying they are “terrible” — in fact, it means their effects are more or less equivalent to the effects of paper-based practice. And if “mixed effects” is what you get from average, frazzled implementation, that means you probably get significantly positive effects when online practice is used well.

Since you seem to be in a position of offering PD, I think you’d have a lot of good wisdom to offer on good/bad ways of using online practice. The examples you tweeted about were pretty brutal.

You undoubtedly sensed this from the reaction you say your post received, but the post did come across (to me) as accusatory and dismissive of the efforts of teachers who are using these kinds of tools to solve a problem nobody has solved yet.

Thanks, David, for your analysis of the frequent failures of online math practice systems. Your conclusion speaks directly to me: I’ve been writing math practice software to use in my own Algebra classroom for the last 13 years. I call my system ‘Get More Math.’ I recently resigned from the classroom and turned GMM into a math software company, so now I’m one of those developers that you were talking to.

I believe my system addresses some (not all) concerns you and others raise. It’s utterly founded on the principle that software should empower teachers, not replace them. Machines should do what machines do well so that teachers can do more of what teachers do well.

So what can machines do well? At a basic level, present math problems to students, then check student answers for accuracy. At a higher level, pay attention to student performance and use that data to produce individualized practice to promote long-term retention. That second function is where GMM shines. It provides students with cumulative mixed review sessions based on historical strengths and needs. Every day, a student receives a new mix of practice problems, selected for her based on data collected during all her former sessions. Is she weak on graphing? There may be a graphing problem. Is she a champ at multiplying decimals? It is less likely that problems of that type will show up, although occasionally GMM will include one to make sure she still remembers.

GMM presupposes the presence of a qualified teacher — a supportive class is helpful, too! GMM has no instructional content: there is no tutorial if a student gets stuck, there are no helpful videos. If a student gets a problem wrong, that problem won’t ‘go away’ after clicks/swipes/logging off. It remains until the student fixes it and works out another similar problem. If the student needs help, she gets it old school (notes, other students, teacher).

Adaptive learning has become the gold standard for most math software. Apps typically endeavor to provide a complete, individualized math education, with each student forging ahead at her own pace. I acknowledge the utility of this for some scenarios (see Kevin’s comments, above) but I feel this is often overreach. I prefer “adaptive practice” — each student regularly pushing back over hard-won territory to ensure its retention.

A typical use of GMM:

1) Teacher teaches. So many good ways to do this, a lifetime of adventure for the committed teacher!

2) Students log in to GMM and do some focused practice on today’s lesson — no mixed review yet.

3) As soon as a student shows initial mastery on today’s lesson (teachers get to define that threshold) she is automatically shifted to ‘Mixed Review’ mode, which will include skills from prior lessons.

You mentioned you’ve tried a lot of math software systems. Sounds fatiguing! If you’re game for checking out another one, I would be delighted to show you around Get More Math. Just drop me a note:

Josh Britton

jbritton@getmoremath.com