How can we evaluate our use of educational technology?

At Edcamp Fraser Valley, I facilitated a conversation on the use of educational technology, focusing on the issue of evaluating its use. The framework I shared (created by Bates and Poole, see below) was the SECTIONS framework, which I have found to be a reasonably comprehensive approach to examining educational technology, depending on how you unpack it. This framework reminds us that we need to think about students, ease of use, cost structure, teaching and learning, interactivity, organizational issues, novelty, and speed, when examining educational technology.

At the beginning of the session, I started by recommending starting with students, teaching, and learning when using these frameworks since a primary purpose in using technology should be to improve teaching and learning, with a focus on the effect on student learning. Here is a summary of some of the various points that were brought up during the session.

• Students:

  Is the technology accessible for all of your students? Many (if not most) Flash based applications are nearly useless for your severely visually impaired students, for example. Videos without subtitles are extremely difficult for hearing impaired students to follow. Text based technology might be gibberish to students who are struggling with literacy issues.
   

• Ease of use: 

  How easy is the technology to use? Will learning how to use the technology interfere with the ability to use the technology as a tool? Will students spend ages booting up computers (4 minutes a lesson is 12 hours of wasted instructional time over the course of a year!)?
   

• Cost structure:

  Compare the cost of the technology to what it allows you to do. Some technology is cheap and can be used in a variety of different ways. Some technology is expensive (both in terms of time and money) and can only be used in one or two ways. The benefit to cost ratio of technology should be as high as possible. Further, do not forget to evaluate the environmental cost of the technology. 
   

• Teaching and learning:

  This is what you should look at when evaluating a new technology right after establishing it’s accessibility. What pedagogy am I trying to support with this technology? Does this technology support new modes of pedagogy? Will this technology improve student learning (or at least provide affordances for student learning)? How will I know if this technology is effective? Has any (independent from a vendor) research been done to evaluate this technology?
   

• Interactivity:

  Is this technology interactive? Does it provide opportunities for the learner to get feedback about what they are doing? Is the technology doing something that you can’t already do without it?
   

• Organizational issues:

  Can your organization support this technology? How will you maintain it (if needed)? Does its use fit your organizations mission and values? Do you have sufficient staff to provide training and support for the technology? If not, what staffing ratios will it require? How will you train staff to use the technology (if necessary)? Will the technology even be used? One participant noted that, when cleaning out a closet, he found unopened packs of teacher resources. What a waste of money!
   

• Novelty:

  Choosing a technology because the novelty of it will temporarily engage learners is a bad idea. If this is the primary reason you are using a technology, you’ll find yourself needing to switch to a new technology on a regular basis without sufficient time to evaluate each of the successive technologies. Further, it will make it challenging to evaluate the effectiveness of the technology for long-term use. Was the technology useful for students because of inherent qualities, or because they just hadn’t used it before?
   

• Speed:

  How fast is the technology? Will significant time be spent waiting for the technology to do its job? Is there another equivalent technology that could be faster?
   

After more time considering the issue, I’d add one more item to the SECTIONS framework which I think is hardly ever considered.

• Culture:

  How will the use of this technology affect the culture of your organization? How will this technology affect our greater society? As Neil Postman notes, no technology is culture-neutral. Will our society change as a result of this technology? Is this change desirable? What would we do if we used technology, and found that the changes in our society (or environment) were undesirable? Could we stop using the technology? There is enormous inertia behind the use of technology in society, and technologies are very rarely abandoned.

Technology which is unevaluated before its use could be ineffectively used, wasteful, or even harmful for students. It is irresponsible to use technology with students without some expectation of the outcomes of the technology. While I support experimentation in the use of technology (and maybe not having all of the answers about the effectiveness of the technology in advance), this does not mean that one should not think critically about the use of the technology before using it. 

References:

Bates, A.W., and G. Poole. (2003). Effective teaching with technology in higher education. San Francisco: Jossey-Bas.

Postman, N. (1998). Five Things We Need to Know About Technological Change.