The Reflective Educator

Fawn Nguyen recently updated her terrific Visual Patterns website and mentioned how she used AI to help her rebuild the site. She also updated the patterns on the website to all use squares instead of some of the random shapes previously used in patterns.

This got me wondering: how well can AI make sense of the patterns themselves? I decided to do some investigation.

I uploaded this pattern to Gemini.

And this is Gemini’s response when asked to analyze the pattern and determine a rule matching the pattern, and then annotate the image to make it apparent how the rule matches.

I decided that this response was a bit too detailed, and asked it to simplify the image and focus on the annotation instead of including all of the explanation on the image itself. For a simple linear pattern, Gemini’s output is more than adequate (although, it does have an error in the figure n element of the image).

Next, I tried this same task with Claude.

Interestingly enough, Gemini and Claude have different ways of decomposing the visuals to represent the linear pattern. The simple task that a 6th grade student can do was completed by both AI tools I tried.

Next, I moved onto a more complicated pattern. I fed this quadratic pattern into both tools to see what the AI models could do with it.

This is where the AI models struggled. Initially, Gemini treated the visual as a growing linear pattern, and tried to get their annotation and formulas to match linearly. Claude simply did not attempt the annotation at all, although it seemed more aware this pattern is quadratic in nature.

Here’s Gemini’s output. There are obvious errors in the output, including Gemini completing missing a column of the right part of the 3rd figure.

Here’s Claude’s initial attempt. Notice how the AI model does not follow my instructions on visually annotating the pattern and instead just counts the squares and comes up with a formula. This formula also isn’t displayed correctly.

I asked Claude next to make explicit and direct connections between the visuals and the formula. I even told Claude another form of the formula is Total squares = 2n(n+1) + 3. Here’s what it came up with.

Notice how it tries to fit the +3 to the three squares at the bottom of the image, even though these three squares are not what most people who see this image think of as the constant part of the image. On the other hand, at least Claude is able to correctly match the value of n to the formula and has shaded the right number of squares. It doesn’t make use of the structure of the shape at all.

Here are a couple of the ways my students have seen the structure of one of the terms pattern over the years. Notice that the structure is very clear, especially the symmetry inherent in each figure of the pattern. While these images do not explicitly show students making connections between a formula and the figures, my 8th grade students this past semester could nearly all visualize this relationship and come up with it on their own.

The human mind makes use of visual structure in a way that the models I tested did not. The AI models I tested, which are the general free models available to consumers, are not yet as “smart” as middle school students in seeing and making use of mathematical structure.

This amazes me in a way, since I recently read that some of these models have made headway on incredibly complex mathematics and even outputted proofs or counter-examples to theorems not yet proven by mathematicians.

There are three different ways the word learning is used, and all of them are important. Educational technology companies usually pick one of these forms of learning to focus on, but to be effective, their products really need to do all three. The feedback systems built into their products need to find ways to support all three learning scenarios.

• Learning as the introduction to ideas
  This is the most commonly way of understanding learning. A learner doesn’t know something and they are exposed to information that gives them new insight about the world.¹ This exposure may take the form of explicit instruction or it could be as part of a guided discovery. Learning is measured by student performance on tasks completed closely related to the ideas to which a student was exposed. Improving learning is focused on either improving instructional delivery or on improving student performance on follow-up tasks.

• Learning as the activity
  Sometimes learning is defined as the activity students do that results in their engagement and some measurement of performance, and this activity does not include being introduced to new information. For example, students who complete a worksheet can be said to be in the act of learning. Learning is measured by how much students participate and complete work. Improving learning centers around improving engagement with the activity.

• Learning as long-term change
  Graham Nuthall, author of the Hidden Lives of Learners², defined learning as the longterm change in what students know, believe, and can do. With this model of learning, students can be exposed to new information, complete activities, and not learn anything because the ideas didn’t stick. Learning is measured by longterm assessments measuring knowledge and skills gained over months and years. Improving learning centers around long-range strategies and finding ways for learners to remember ideas they have learned more robustly.

If one is designing a digital learning experience, all three of these ways of understanding learning need to be addressed. Ideas need to be introduced to students, students have to complete activities, and long term changes in what students know, believe, and can do should be measured.

Most educational technology solutions focus on introducing ideas to students or engaging students in activities and sometimes both. Very few products focus on ensuring students have changed in the longterm, where ironically, accountability for schools focuses nearly entirely on this aspect of learning.

Feedback while introducing ideas

When introducing ideas to students, it is an excellent practice to frequently check to see if students understand an idea. It is even better practice to check how students understand a new idea! One can then adjust instruction based on the feedback to the instructor. Ideally, this results in short cycles of introducing a small amount of information to students, engaging students in an opportunity to demonstrate their understanding, gathering evidence of that performance, and adjusting instruction.³

Here the feedback is mostly to the instructor and they make decisions about how to adjust their instruction that result in new information (e.g. feedback on learning) being given to learners.

Most educational technology products are not this deliberate in how they introduce information to students, and even the ones that include small incremental steps when introducing information do not consistently change their instruction as a result of a cycle like this.

Further, the information provided to instructors about student performance is generally unhelpful. Knowing that children have completed this activity or that activity (information typically provided in educational technology products) is less helpful than knowing what kinds of mistakes students were making and how to support students (information never provided in educational technology products). The lowest hanging fruit in reporting is completion data, but the closer the data is to the actual work students did, the more likely it is to be helpful. Imagine a product that takes all student work from activity and synthesizes it into the three to five most commonly used ways of thinking students had during the activity and then suggests activities instructors can do.

Feedback during activities

Key among the strategies educational technology companies have promised over the years is personalized feedback to students. Unfortunately, few products on the market deliver on this promise. If your product contains digital explanations, showing them an explanation of why they are wrong after they get an answer incorrect is the digital equivalent of reteaching by talking louder and slower. It doesn’t work. Feedback in those moments should depend on the thinking children have done, and few products include robust enough student testing in their development lifecycle to be able to create the feedback opportunities that actually work.

An alternative way of presenting feedback to students is to select a different task for them to work on. Perhaps a student is having trouble with 102 – 94 and you want them to think about this problem differently. A teacher would know that a related problem in the moment can be transformative and ask that child to try solving 100 – 94. If this is still a struggle for the child, they can try 100 – 95 first, then go back to 100 – 94, and then go back to the original problem of 102 – 94. To mitigate this struggle in the first place, one could reverse this sequence of problems, but then one might not know if the scaffolding led the child to the idea of changing 102 – 94 to 102 – 100 + 100 – 94 or if the child can do this independently. Using tasks to provide feedback is one of the more powerful instructional moves a teacher can make and most educational technology products do not even attempt to use it.

Feedback during long-term learning

Durable long-term learning is really about supporting students with revisiting ideas they to which they have already been exposed. Our brains are designed to ruthlessly trim ideas from active recall that we do not use frequently. This means, if there are things we want to remember, we either need to make the experience of learning them extremely memorable, or we need to spend effort thinking about those ideas again periodically.

No educational technology products on the market today deliberately and systematically offers students opportunities to revisit ideas over time. Every product assumes that once children have demonstrated proficiency with an idea that this proficiency is stable and unchanging. This is demonstrably false! We all know that learners forget things over time.

Spaced opportunities to revisit ideas learned is in the best interest of learners. Educational technology has the ability to track what students have demonstrated proficiency in, and to keep track of the hundreds of ideas that students should revisit and at what intervals they should come back. There is very solid evidence that spaced, interleaved repetition and practice lead to long term changes in memory.

Conclusion

There is a need for educational technology to be better than it is. More deliberate systems and focus on improving feedback in a product is one way nearly every product on the market can improve. They need to move beyond simplistic models of learning (the student completed an activity so therefore they “know it”) to models that include all three ways of understanding learning.

Our current “system” of education is actually hodgepodge of different competing organizations layered on top of each other. It’s not actually a system at all.

If you compared our system for professional learning to a relay race where each organization in a teacher’s career should pass the baton of learning to the next organization, it’s instead like a race with people running in different directions and there’s no baton in sight.

Our system for student learning is slightly better, if only because students tend to stay within the same school for a number of years and schools tend to be a bit better about internal consistency across year levels. Still, in most schools what a child knows about a topic from the year before is almost always lost by the time the related topics come up in the following year. In fact, despite the fact schools collect mountains of information on each child in a year, most of it is discarded before the following year. Remember, every worksheet a child does contains information about their performance at a moment in time.

Our systems for connecting educator learning to student learning are at best aspirational, with almost no schools anywhere linking the two. When was the last time you heard in a staff meeting, “We see students need support with ____ so we need to learn ____”? It’s just not done. In the best schools, teachers get autonomy to decide what to learn based on reflecting on their work with students, and in the worst, educator professional learning is neither professional nor does it result in learning.

We need a stable set of goals for students, so that we can measure student progress toward those goals and make decisions from that information. We need agreement on the relative importance of those goals, so that we stop shifting goalposts year after year. We need to create our system by working up from student learning and not down from ministries of education, so that every link in the chain of the system is built carefully based on evidence of impact.

Now that I’ve been coaching math teachers again, I have been in a lot more classrooms and doing classroom observations again.

In a past iteration as a math coach⁴ , I created this checklist of things⁵ to look for explicitly when observing teachers. The goal at the time was to be able to have our team have a clear list of observable behaviours so that we could measure whether our professional learning work was landing with teachers. With 8 instructional coaches and 30 schools in our project, we needed something that would work at scale.

We created curricular resources for teachers, ran workshops for teachers to teach them how to use the curricular resources (via modeling, rehearsals, and dialogue), and then coached the teacher leads at the schools. We would then observe teachers in their classrooms and teacher leads in their meetings, so that we could learn whether our work was landing. Over time, we iterated and improved our approach as we learned from what we did.

As an individual coach though, my goals for lesson observations are tied to the coaching work I do with teachers. The teacher I’m coaching and I decide together what I should focus on when observing them, and I spend much more time learning alongside the teacher instead of just sitting back and passively observing their classrooms.

I’ve since been heavily influenced by Elham Kazemi’s (et al.⁶) Learning Together so that instead of trying to transmit what I know to the teachers I work with, I work with teachers to try out different strategies to see what works with these students in this context, focused heavily on instructional goals informed by our conversations, data we have collected, and the goals of the school. I still do typical coaching ‘moves’ such as modeling instruction, observations, and leading team meetings, but these are in service of our shared learning.

Neither of these approaches are better than the other. They are meant to serve different types of structures and sets of resources available. But both are meant to serve the main goal — the learning of teachers in service of student learning. One’s strategy for coaching should match the available resources one has and the goals one has for that coaching.

I like Graham Nuthall’s definition of learning as ‘long-term changes in what learners know, believe, and can do.’ From this definition, we can see that one cannot directly observe student or teacher learning from a single lesson observation; all one can observe in learners are behaviours that are likely to lead to learning. Lesson observations therefore also have to be paired with collection of other kinds of data, including student performance data collected over the long term, and data about changes in teacher practice that are sustained.

Either approach therefore will require a longer term view. It’s hard to see if long term changes are making an impact as well, as one needs both a control year (how much did students change this year) and a treatment year (under this new strategy, how much did students change). You also need time to see if the strategies you and teachers learn stick, since you are learners too! My recommendation is to make sure to study the impact of your work and repeatedly iterate on both your coaching approach and the teaching strategies your team tries, until you see the results you want.

School culture trumps great teaching, or it can.

When teachers try to teach the way they think is best, they do so while navigating the culture of the school and community where they work. What is considered excellent teaching is dependent on both the school culture and the dominant culture the students come from⁷.

Our personal set of values informs how we teach, and the set of values a school community values informs what they consider good teaching. Consequently, comparing your personal values to the lived (rather than the espoused ⁸) values of a school is a good way of figuring out whether you are a good fit for a school.

If you are an educator considering transitioning to a new school, here are some questions you can ask to see if your set of values will mesh well with the school’s culture.

• What level of technology is expected in your teaching?
• How much flexibility do educators have in selecting their curricular resources?
• When one enters a classroom and everything is going smoothly, what does one see?
• How does the school support students with behaviour?
• What support exists at the school for students with special needs?
• What supports exist for teachers?

The answer to these questions will speak to what the school values, and you can use the answers to help yourself determine if you are a good fit for the school culture.

Every school I’ve worked at uses a teacher evaluation system where the primary goal is to determine if whether the teachers are doing a good job or not.

These systems typically look like goal-setting with an administrator or supervisor at the beginning of the school year, a few observations of the teacher during the year (potentially with or without feedback) and then a rubric applied to decide if the teacher needs additional support or not.

What these systems don’t normally do is connect to the professional growth of the teacher. They don’t even necessarily connect to the goals of the school. In fact, many schools don’t even have learning goals for their teachers.

In an ideal system, the learning goals of the educators are that which best support student learning, derived from a shared understanding as a community what the learning goals for students should be. Imagine a school where educators work with the community (including students) to decide on what the goals are for the school, and then decide what they will measure to see progress toward those goals, and then hold themselves and the community accountable towards reaching those goals.

Instead of a system designed to play “gotcha” and figure out who the bad teachers are, the system would be designed to benefit the learning of everyone.

Teachers could select professional development that connected directly to school improvement goals. They could then see why they are learning something because those goals would be transparent to them. They would know what to focus on during professional development and be much more likely to integrate it into their daily work. Supervisors and administrators could come observe classes and be able to know what to look for and what to focus on in terms of feedback. In fact, instead of being evaluators, these people could be learners alongside their teachers.

How might your school look different if everyone had a shared vision of where they were trying to go as a school and had the time to plan how to get there?

How can we inspire educators to do the challenging work that is required to develop and extend their practice? What role do routines play in student learning? How do we help educators focus on the right metrics for learning? What role does technology play in student learning?

I’ve created a new streamlined page for my educational consulting services on my website to help schools and educators who are looking for support answering these questions.

Check it out here: https://davidwees.com/services/

Let me know how we can work together!

Note: I am based in Hong Kong but willing to travel if needed, around my school schedule.

“I learn something and then we are tested on it, and I know I never to know it again, so I immediately forget it.”

A Grade 11 Student

It seems incredibly common to me. Teachers teach, students learn, teachers assess, and then students forget. But why does this happen, and why do we accept it?

The fundamental goals of education are to equip people with the knowledge and skills they need for the future, to open up their minds to what exists or could exist, and to help learners understand the world. If what students do is go through different cycles of learning and forgetting without anything being more permanently remembered, this makes that goal a fantasy.

There are three approaches that can be used to improve the odds that learning doesn’t fall into this trap.

1. Make the initial learning unforgettable.
2. Revisit the ideas previously learned over and over again.
3. Connect everything learned to big ideas.

Make learning unforgettable

We all have experiences from our schooling days that have stuck. These days and moments stand out to us, typically because we have strong emotion attached or because the events were just so new and/or fascinating to us. I still remember sitting at a computer in our social studies classroom in the basement playing around with Logo on a Commodore 64 computer in the corner.

Can educators make learning like this? I don’t think so, at least not consistently, and not for all children. What makes one child feel a strong emotional response isn’t likely to produce the same response in all children. For every child who learned how to program computers from Logo, there are probably two more who found something else fascinating instead.

But educators can make learning stickier. These instructional strategies, which aim to make mathematical ideas explicit, also make learning sticker. They are almost all free to implement, and while not entirely straightforward, every educator is able to use these kinds of instructional moves.

Revisit ideas learned

Since our brains are optimized to remember things that come up again and again, practicing using ideas is certain to cue the brain as to the importance and relevance of that material. There are some ideal ways to optimize practice, and none of these strategies is especially difficult to implement. They just require some careful planning to coordinate the timing of practicing ideas during the year.

This needs for practice is born out by many different research studies that show that the strength of our memories decays over time and that repetition is crucial to keeping our long term memories fresh.

The problem here is that one cannot practice everything; there just isn’t enough time. Educators have to be selective about what is practiced and what is likely to eventually be forgotten. However, this is where the next idea comes in.

Connect everything to big ideas

Another thing that psychologists have discovered is that the type of information one is trying to learn matters. Random information is much harder to remember than well-structured information, which is similarly harder to remember than information that is well-connected to things one already knows.

Ideally, educators should help students connect what they are learning to other things they already know. One strategy to do this is to connect the small ideas to be learned to bigger ideas already known. One big idea might be: “we can represent mathematical ideas equivalently in different forms”. A smaller idea that can be connected to this big idea is: “we can represent fractions like ¼ visually using a bar model with 4 equal parts and one of them shaded.”

This last idea is less straightforward, unfortunately. Coming up with the right big ideas is nontrivial and can take significant planning from educators. Many curricula out there do not make big ideas explicit, making educators’ work harder. While it is technically free to implement this, the planning load for educators is likely to be large, at least initially. This is where teaming up with other educators is critical to share the planning load.

If these strategies are so good, why aren’t educators already doing these things?

Like most things in education that seem like good ideas that aren’t being used, there are probably a variety of reasons. Using new instructional strategies is great, but educators need time to practice and prepare to use these strategies. Educators feel rushed to complete a long list of things people already think they should be teaching, so building in time for additional practice seems impossible. Educators are already short on time to finish planning all the things they are currently doing that adding one more planning task to the load with uncertain benefits seems unreasonable.

I’m not sure what the answer is here, except that I wish the curricula educators were offered had better support for these strategies. If the curricula offered suggestions for when to use specific instructional strategies, part of the planning load for educators would be already done. If curricula included spaced, interleaved, retrieval practice of the most critical ideas from the curricula, educators could much more easily implement this practice. If curricula explicitly named the big ideas of the curricula that connect together all the small ideas, then educators could spend less time trying to work these out and more time planning how to use these big ideas in their instruction.

This research brief from Vanderbilt University that suggests metacognition in math class is interesting to me for a couple of reasons.

The first is that it includes suggestive evidence this practice works and can be implemented at scale. Of course, like most things that are attempted to be used at scale, educators have to understand why it works and adopt the framework behind the practice. They can’t just dogmatically adopt the practice.

The second is that this is a relatively cheap and easy to implement practice that builds on something many, many educators either already do or know they should be doing, which is using exit tickets. Finding out what children have understood from your class is a pretty crucial practice, and this builds on that practice.

But what does metacognition in the math class look like, and why should educators implement this practice?

Metacognition is “an awareness of one’s thought processes and an understanding of the patterns behind them.” (source) In math class, this looks like students thinking about how they solved problems, looking for generalizations they might apply from those solutions, and reflecting on how they can approach mathematics differently in the future.

Amy Lucenta and Grace Kelemanik taught me the power of metacognition through their Routines for Reasoning. Everyone of their instructional routines includes a meta-reflection at the end and is a necessary part of these powerful routines.

[Memory is] the residue of thought, meaning that the more you think about something, the more likely it is that you’ll remember it later.

Dan Willingham, Why Children Don’t Like School

If what we want children to remember is specific problems that they’ll never see again, we should fill all of their time with endless practice. If we want them to be able to generalize and maybe be able to solve problems of a similar type but which they haven’t seen, we should ask them to reflect (for at least a few minutes) on the thought processes they used.

The goal of a metacognition can be for students to generalize from their experiences. A common metacognitive prompt I use is, “What mathematical thing do you want to remember today that you think might be important tomorrow, given you may never see these exact problems again?“

Crucially, metacognition can turn math class from a place where students solve specific mathematical problems into a place where they learn mathematical principles they can apply to solve problems. The difference is in whether students are just learning procedures or if they are also learning concepts.

There are a number of services claiming to create lessons plans for educators. Just type in your goal, and the generative-AI creates a lesson plan. But is the lesson plan worth using?

In theory, services that aim to save educators time are worth investing in, and educators spend a lot of time researching and planning lesson plans. One solution to this problem aims to address this problem by providing high-quality curriculum resources that include ready-made lesson plans along with student handouts and other resources. Another solution generates these lesson plans utilizing the high number of other lesson plans already shared on the internet as source material. The high-quality curriculum materials tend to be expensive and sometimes hard to use. The AI-generate lesson plans are quick and free, but they are also filled with a lot of nonsense.

The high-quality instructional materials solution is appealing to me because it leverages the collective expertise of thousands of total years of education experience writing and then delivering lessons from the educators who work on the curriculum. This isn’t quite putting the world’s best teachers in front of every student, but it is putting their lesson plans in front of every teacher. My experience shows though that these lesson plans are very solid, but also mostly require the experiences of those educators to come off in the ways they intended. They are hard to use because the instructional moves they demand are not well known or understood outside of a relatively small circle of educators. We created similar lesson plans but experienced a bit more success because we also facilitated workshops for educators, directly aligned to our instructional materials and leveraging our personal experience with the instructional practices embedded within them.

AI generated lesson plans tend to be pretty trashy. I’ve looked at hundreds and not found a single generated example yet that I would personally recommend or use out-of-the-box. This is in part because the inputs contain a lot of simplistic and ready to sell lesson plans and in part because the AI doesn’t have access to some critical information needed to write a good lesson plan. More on that later. These generated lesson plans contain lots of mistakes that I see early career educators make; they assume knowledge the student doesn’t have, they move too quickly, there is no deliberate selection of problems, there are limited opportunities for practice, and they lack instructional guidance on how to actually teach the content.

There’s some magic at work for sure though, when one sees a completely formed lesson plan generated within seconds that purports to help one teach a particular goal, but the shine from this magic wears off pretty quickly as soon as you begin to rub at it.

With some careful review, the AI-generated lesson plans can be improved reasonably quickly. There are many parts of the lesson plan Generative AI does a good job of creating. Given a lesson plan, it can easily and accurately figure out what types of materials would be needed, what key vocabulary is necessary for students to know, a good title for the lesson should be, and other similar metadata about the lesson. It can even generate a relatively simple script for educators to follow, which could be a helpful support for an educator who wants to see the step-by-step version of the lesson plan.

Given all this, it’s helpful to consider what might be done to improve these lesson plans, or alternatively, to consider what role generative AI should play in any lesson plan development.

What the AI doesn’t know is the context in which this lesson plan is intended to be used. It doesn’t know what the children to be taught know and don’t know. It doesn’t know what excites them and what bores them. It doesn’t know what instructional moves the educator is comfortable with. It doesn’t know how the subject matter to be taught hangs together in a deeply interconnected web of knowledge. It doesn’t know what common misconceptions children might have. It doesn’t know a lot of things!

Coincidentally, many of these things the authors of curricula materials also don’t know. The two areas where expert curricula designers do know a lot more than AI are the common misconceptions children might have and how the content in the subject area is connected together. These are critical! The things the expert curricula designers don’t know though are why high-quality instructional materials can be so hard for educators to use.

Given that knowledge of common misconceptions children may have with respect to content is specific to individual topics and not catalogued in detail for every topic, AI-generated lesson plans are doomed to fail. Given that the detailed connections between different topics is also not catalogued in detail for every topic, AI-generated lesson plans are doomed to fail. However, if one invests the time to carefully catalogue this information first, AI-generated lessons can be vastly improved.

Given that no such efforts to record and catalogue common student reasoning across an entire subject area yet exist, I’m not worried about Generative AI taking my job as a curriculum designer just yet.

(An aside: The AI-generated lesson plans are realistic enough that they can pass superficial review, so if I worked for one of those many sites where educators share their lesson plans with each other, I should be deeply worried about my site being flooded with AI-generated lesson plans.)