In order to do research in medical education, we first need to define outcomes.
Stanley Hamstra described at the SAEM 2012 Consensus conference the Kirkpatrick hierarchy for the assessment of learning.
Most studies fall at the bottom of the pyramid, the learner’s reaction to the experience – did they like it? This is where most of our evaluations of our courses fall when looking mainly at student comments.
The next step is to check if actual learning occurred. This is where we can do a pre-test and post-test or look at final exam or USMLE scores in a particular area. Though you can make the argument that USMLE scores do not correlate to knowledge base or professional practice.
Next we assess if a teaching intervention changes a students practice behavior. Do students approach their patients differently based on what they learned?
Finally, results or patient-outcomes is the pinnacle achievement. Did our teaching change patient outcomes. The classic example of this is the Baysuk paper in which he looked at central line placement in a simulation lab. McGaghie and his team did a paper for each of the outcomes:
- Reactions – did students like this method of learning
- Knowledge – can they do well on a test of this information
- Behaviors – did they actually place a line with better technical form (ie, wash hands, flush lines, etc)
- Results – did this decrease the incidence of line infections in patients
Way to milk it. But wait… there’s more. He took it even a few steps further and looked at
- Value – how much money did this decrease in line infections save the hospital (this can be important in justifying the cost of doing such studies)
- Retention – how long did students keep these skills or did they require a refresher course (they did)
The modified Kirkpatrick’s hierarchy is:
- Level 1: participation/satisfaction
- Level 2a: change in attitudes/perceptions
- Level 2b: change in knowledge/skills
- Level 3: change in an individual’s behaviors
- Level 4a: change in an organization’s practice
- Level 4b: benefits to patients
Level 4b is the ultimate goal, of course, of education – to make better doctors.
Dreyfus Levels of Skill Acquisition and Attention
He also discussed the way our attention is allocated. Learners can be subdivided based on their level of expertise: novice, advanced beginner, competent, proficient, expert. The novice and advanced beginner focuses all their effort on the task at hand. They have little spare capacity to handle distractions. And as such, distractions should be kept to a minimum. Experts can handle tasks with little attention paid to them, it’s almost automated. They have plenty of spare attention capacity.
So novices should have little distractions when learning a skill. What sorts of distractions? Maybe we should hold off on presenting case vignettes when learning how to calculate osmolar gaps and instead let them just practice with the numbers… until they get it. Then we can go to applying it in context.
Moral of the story
Try to reach the highest levels of the Kirkpatrick hierarchy as you can, but if you cannot – go for the best you can get. And then milk everything out of it you can.
References:
- Stanley Hamstra. SAEM 2012 Consensus Conference. Chicago. May 9, 2012.
my peripheral brain 
