(By Dr. Natalie McKnight)
When Matt Trevithick began thinking about starting the Spaced Repetition Foundation, I decided to see if any colleagues of mine at Boston University (BU) were using spaced repetition in their courses or in their own learning. A quick search of the BU website led me to a PowerPoint presentation by Dr. Kitt Shaffer, Vice-Chair for Education in Radiology at Boston University Medical Center, as well as Chair of the Boston University Medical School Faculty Appointments and Promotions Committee. Kitt has been a Radiology and Anatomy educator since the early 70's, with a PhD in Anatomy and training in Thoracic, Oncologic, and Breast Imaging. She has been Director of Medical Student Education at Brigham and Women's Hospital, Clinical Director of Radiology at Dana-Farber Cancer Institute, Course Director for the Harvard Medical School 1st Year Anatomy Course (Human Body), and Director of Undergraduate Medical Education at Cambridge Health Alliance. She has won many teaching awards, including the Educator of the Year for both the Radiological Society of North America and for the Association of University Radiologists, and she has written widely on digital teaching methods, including the use of spaced education. Her answers to my questions about her use of spaced repetition were extremely generous, as was her list of relevant resources, as you can see below. It quickly became clear that Kitt’s expertise on the subject far exceeds mine, and I promised her if we ever launch a Spaced Repetition Hall of Fame, she’d be one of the first members.
Question: Dr. Shaffer, how did you first discover spaced repetition? Were you using it for your own learning?
Kitt: My first exposure to spaced repetition as a teaching method was during collaboration with Dr. Price Kerfoot, who conducted multi-center trials in Urology education using an innovative web-based email methodology that showed proven efficacy in a manner similar to a drug trial. Those who had repetitions of questions until they had given the correct answer twice had markedly improved long-term retention of information. In reflecting on my own learning style after this experience, I realized that I had made use of a similar (non-digital) method throughout medical school with physical flashcards. I still make flashcards when trying to learn languages, and enjoy the various computer programs that offer 'digital flashcards' and repetition of learning objects to facilitate long-term retention. One in particular that I like is called Duolingo and it presents you with repeated tasks that use different channels of learning (writing, reading, hearing) to help with language advancement.
Question: (As an aside, I also love Duolingo!) I judge by your PowerPoint that you use spaced rep in your classes. Are you still doing so? How do students respond? Have you collected any evidence of its impact?
Kitt: I was a contributor to Dr. Kerfoot's* original Spaced Education website, and designed a number of different types of questions to expand on the usual multiple choice (multiple guess) format. I developed short answer question formats that could be graded by computer, through entering of all possible variations on the correct answer. In fact, I used some crowd-sourcing from participants (who would email me if they thought their answer should have been scored as correct). I also developed methods for displaying stacks of cross-sectional images (like CT and MR) using video and outlined various anatomic structures on each movie for users to identify, using fill-in-the-blank question formats. I think that the combination of short-answer (which requires more sophisticated thought processes than multiple-choice) and spaced repetition is particularly powerful. Feedback from this site was uniformly positive and for a while, it was one of the top-rated courses on the site.
Question: Also, it looks like you were using Spaced ED in some classes--do you still use that? Would you recommend it?
Kitt: As the SpacedEd site went through several technical upgrades, they were no longer able to display the videos I had uploaded, and lost interest in the short-answer formats (which take a bit more effort to input), so I don't think my course is currently available. [Editor’s note: Google searches of “SpacedEd” lead to fairly dated sources, and links within those pages direct searchers to “QStream,” which seems to be an application of spaced repetition to industry needs: https://qstream.com/ ).
Question: Are you aware of other faculty who use it?
Kitt: As the Spaced Ed site has gone through various transformations, I have not kept up with it so am not currently using it. I do know that many of our trainees use other online flashcard programs to assist in study for board exams. I do not currently use any of these in my teaching. I have begun using a free audience response program (Socrative) for short-answer questions as part of my interactive teaching with radiology trainees, but have not tried doing a spaced repetition of the same questions, which is actually a great idea. I may try that going forward. I think it would solidify the things I am trying to teach them in a meaningful way. I love this idea!
Question: Your online PowerPoint lists some very helpful references--do you recommend any others? (One of my recent favorite finds is: Make it Stick: the Science of Successful Learning (2014) by Peter C. Brown, Henry L. Roediger III, Mark A. McDaniel).
Kitt: As references, I would recommend all of Dr. Kerfoot's articles* on his work in Urology. It is really an amazing body of research, and I have not heard of anyone else who has done such sophisticated validation of the spaced repetition concept. The results are very impressive. If spaced education was a drug, we would all want to take it!
Question: Is there anything else you’d like to add?
Kitt: I have also become interested in designing educational websites for a flipped classroom approach that I use in large group teaching for medical students and physician's assistant students, and when you think about it, that is really a form of spaced repetition. Learners are asked to look at information on their own before class, and then it is shown again in class for a second repetition. Learners then have access to the same information on my website going forward, which has potential for a third repetition (or second, if they did not do their initial homework). If you want to see the website, the URL is http://www.shafferradiology.com It would be very interesting to design a study to see what the impact of viewing the material ahead of time is on the actual learning and retention that occurs. A multi-center prospective trial like Dr. Kerfoot has done would be the ideal, but most of us mere humans are not as energetic or dedicated as he is!
*Here is a list of references that Dr. Shaffer recommends:
1: Kerfoot BP, Turchin A, Breydo E, Gagnon D, Conlin PR. An online spaced-education game among clinicians improves their patients' time to blood pressure control: a randomized controlled trial. Circ Cardiovasc Qual Outcomes.
2014 May;7(3):468-74. doi: 10.1161/CIRCOUTCOMES.113.000814. PubMed PMID:
24847084; PubMed Central PMCID: PMC4040124.
2: Kerfoot BP, Baker H, Pangaro L, Agarwal K, Taffet G, Mechaber AJ, Armstrong EG. An online spaced-education game to teach and assess medical students: a multi-institutional prospective trial. Acad Med. 2012 Oct;87(10):1443-9. PubMed
3: Kerfoot BP, Baker H. An online spaced-education game for global continuing medical education: a randomized trial. Ann Surg. 2012 Jul;256(1):33-8. doi:
10.1097/SLA.0b013e31825b3912. Erratum in: Ann Surg. 2012 Oct;256(4):669. PubMed
4: Kerfoot BP, Baker H. An online spaced-education game to teach and assess
residents: a multi-institutional prospective trial. J Am Coll Surg. 2012 Mar;214(3):367-73. doi: 10.1016/j.jamcollsurg.2011.11.009. PubMed PMID: 22225647.
5: Kerfoot BP, Shaffer K, McMahon GT, Baker H, Kirdar J, Kanter S, Corbett EC Jr, Berkow R, Krupat E, Armstrong EG. Online "spaced education progress-testing" of students to confront two upcoming challenges to medical schools. Acad Med. 2011 Mar;86(3):300-6. doi: 10.1097/ACM.0b013e3182087bef. PubMed PMID: 21248600.
6: Kerfoot BP, Lawler EV, Sokolovskaya G, Gagnon D, Conlin PR. Durable improvements in prostate cancer screening from online spaced education a randomized controlled trial. Am J Prev Med. 2010 Nov;39(5):472-8. doi:
10.1016/j.amepre.2010.07.016. PubMed PMID: 20965387; PubMed Central PMCID:
7: Kerfoot BP, Fu Y, Baker H, Connelly D, Ritchey ML, Genega EM. Online spaced education generates transfer and improves long-term retention of diagnostic
skills: a randomized controlled trial. J Am Coll Surg. 2010 Sep;211(3):331-337.e1. doi: 10.1016/j.jamcollsurg.2010.04.023. PubMed PMID:
8: Long A, Kerfoot BP, Chopra S, Shaw T. Online spaced education to supplement live courses. Med Educ. 2010 May;44(5):519-20. doi:
10.1111/j.1365-2923.2010.03678.x. PubMed PMID: 20374457.
9: Kerfoot BP. Adaptive spaced education improves learning efficiency: a randomized controlled trial. J Urol. 2010 Feb;183(2):678-81. doi:
10.1016/j.juro.2009.10.005. PubMed PMID: 20022032.