The Reliability of a Weighted Performance Checklist for Scoring Standardized Patient (SP) Encounters

While the use of standardized patients to evaluate student performance is not new, we have refined the process to provide more objective and reliable determinations of student skills and progress. 

 Most, if not all, standardized patient programs utilize criteria-based tools to evaluate student performance.  Typically these consist of lists of history-taking and physical assessment skills with check boxes to indicate whether the skills were performed correctly, need improvement, or were omitted.  Often these tools are scored based on students’ performance of a percentage of the total skills listed without consideration for the importance of certain specific skills or their difficulty.

We developed and analyzed a weighted evaluation tool that provided numeric scores to reflect students’ ability to perform commonly used or critical skills that are integrated within a complete history and physical examination.  Faculty were satisfied with this new tool and found it to better able to discriminate between successfully and marginally completing a skill.  In addition, faculty found that the new tool allowed them to assign point values to a skill with more objective confidence and less subjectivity.  We presented this tool as a podium presentation at the 39th Annual NONPF Meeting where it was not only well received but it generated a great deal of interest for replication of the tool. 

Too often novel and promising NP educational tools are not formally evaluated for validity or reliability due to competing faculty demands.  Integrating this tool with video recording of the SP sessions provided an opportunity to evaluate faculty inter-rater reliability.  This year we would like to follow-up on last year’s presentation with data on the reliability of the tool.  Fostering confidence in the fidelity of the evaluation process is critical for both faculty and students.