FICM statement in response to the Independent Review of the Assessment Processes of the RCoA

The Faculty of Intensive Care Medicine welcomes the publication of the independent review conducted by Professor John McLachlan into all examinations delivered by the Royal College of Anaesthetists, which has been published in full today. We will reflect on its findings relevant to the FFICM exam and consider carefully the applicability of all its recommendations. We are committed to delivering fair, robust, and high-quality examinations, and this report will inform our developmental roadmap for the future.  

We have always understood that postgraduate examinations are high stakes assessments that have a significant impact on candidates’ personal and professional lives. For that reason, it was essential for the review to capture the experiences and opinions of our members and other stakeholders, particularly those of current and past candidates. Their input in the review process has been invaluable, as has that of our examiners. We are grateful to everyone who has participated in the review process and thank them for their openness and honesty. We also extend our thanks to Professor McLachlan for giving his perspective. This sits alongside the expertise of our examiners.

We have published an Executive Summary and proposed timetable for implementation of the changes resulting from the review, and we highlight the most significant themes relevant to the FFICM exam here.

Scope of review

The independent review considered all aspects of all exams delivered by the RCoA, including an error in the delivery of the September 2021 FRCA Final Written Exam and the unexpectedly low pass rate for the October 2021 FFICM Objective Structured Clinical Examination (OSCE). The report recommendations have been intentionally written considering all exams delivered by the RCoA, and it will be necessary now to review the recommendations and how they can be applied to the training requirements of each specialty, the stage of training being targeted, and the skills needing to be assessed.

The report covers four separate exams, and not all recommendations apply to the current FFICM exam. For example, the recommendation for change in the multiple-choice question paper to all single best answer questions has already taken place in FFICM, and we do not have unmanned pure knowledge-testing stations in our OSCE. The FFICM SOE includes eight questions already and has a strict question structure to ensure consistency of content and knowledge exploration.

We fully endorse the recommendations from Professor McLachlan to maintain the highest standards of culture and conduct in postgraduate medical examinations and further increase diversity of examiner pools as a candidate, patient and societal benefit. Our examiner Code of Conduct requires behaviour that is non-discriminatory in terms of attitudes, activities, assumptions, beliefs and abilities. We appreciate the professionalism and dedication of our examiners in upholding those standards, and who remain mindful of the need to ensure candidates are not disadvantaged by our exams.

October 2021 FFICM OSCE

We recognise that the low pass rate for the FFICM OSCE in October 2021 was devastating for candidates, examiners, exams staff and trainers. An initial internal review of the October 2021 FFICM OSCE low pass rate took place alongside feedback from listening events held with candidates, examiners, and trainers. Preliminary steps were taken by FICM examiners, FICMTAQ and Board, to address candidate and trainer concerns.

In relation to the low pass rate for the October 2021 FFICM OSCE, the independent review concludes that our decision to allow the OSCE results to stand was the best decision that could be justified at the time and after consultation with the GMC as our regulator. We understand that this decision was disappointing to many candidates. We will closely re-examine our processes should a similarly unexpected result arise again, including use of a secondary standard setting method, as recommended by the independent review. We will continue our work on research into the predictive validity of our assessments to review their alignment with performance in the workplace. 

Our programme of improvement

We are committed to pursuing excellence in all that we do, including the governance and delivery of our examinations. In implementing changes arising from our review of matters identified in both the independent review and our own internal investigations, we will:  

• Invest additional resources (including staff) into our examinations to expand and strengthen our capacity to deliver the exam. An Examinations Development and Assurance Group (EDAG) will be established to lead the development, quality, and alignment of exam processes where appropriate across all three medical specialties whose exams are administered by the RCoA.  
• Give candidates and doctors in training a greater role in our assessment processes, including thorough representation on FICM committees engaged in assessment. These representatives play a central role in supporting communication with candidates and providing stakeholder opinion such as on the FICM StR subcommittee and as elected StR Board members. 
• Continue to review the purpose of our assessments in accordance with the changes made in the curriculum and the changing nature of clinical practice in our specialties. We will consult with members and stakeholders to define this.  
• Work with the EDAG to explore how to implement the best approach to the use of, and training in, standard setting across all RCoA-delivered assessments. We will also work to determine how approaches to all examinations delivered by the RCoA can be best unified. 
• We will also complete the work that was already underway within FICM prior to the independent review to investigate the impact of gender, ethnicity, and educational background on exam performance. 

Your feedback

We understand that members may have questions about the independent review, the applicability to FICM and the actions we are taking as a result. We welcome your feedback and encourage you to attend our Let’s Talk member engagement online event on 8 March at 19.00.  

• Register to attend Let’s Talk

Members can also share their feedback in writing using our online form. This feedback will be shared with the new Examinations Development and Assurance Group. 

Dr Daniele Bryden, Dean of the Faculty of Intensive Care Medicine