A process of extensive review and investigation has taken place since the release of the October 2021 exam results. We have also met with external assessment experts to discuss this review.
Statistical analysis of candidate scores showed the October 21 cohort were more homogenous than prior cohorts, with an average lower score. Statistical analyses have also been conducted on all recent diets of the OSCE examination to look at internal reliability of the question sets including using the Cronbach Alpha. These were discussed with the external experts who agreed that there were no statistical differences between the sets, including the October 2021 exam; all were within the confidence limits.
The results of statistical analysis show no evidence of a difference in performance caused by the online delivery method for OSCE in October 2021 when comparing this diet to previous online and in-person deliveries. The distribution of scores for two of the four OSCE rounds are at the lower end of the range but are not dissimilar from an in-person sitting in April 2018.
The view of the external assessment experts was that the OSCE Angoff process was likely the most appropriate method of standard setting. The external assessment experts made no criticism of the way the Angoff had been conducted and offered the suggestion as an additional unmeasurable factor influencing the October result. We have therefore determined that the senior examiners and RCoA Exams team could explore the use of other standard setting methods alongside the current (and reviewed) model of Angoff in future exam sittings.
