FICM Postition Statement on Medical AI

FICM recognises the potential of medical artificial intelligence to improve the way we care for our patients and to achieve the Faculty’s strategic aims. 

 As a data-rich specialty, critical care potentially has much to gain from artificial intelligence (AI) health technologies. New approaches to clinical prediction may improve our ability to intervene and protect patients at risk of deterioration.^1,2 Advances in computer vision already enable automated reporting of certain radiological images³, and innovations in the functionality of electronic health record systems may improve the efficiency of the care we provide. Augmented and virtual reality may enable new ways of training, including acquisition of practical clinical skills.⁴ 

We welcome research and innovation in the field of medical AI to ensure that critically ill patients can benefit from the development of these new technologies.

However, the current state of evidence underpinning these technologies mean that we must proceed with caution before adopting them into practice.

The path from cutting-edge research concept to usable medical product requires robust evaluation. At present, limitations in the quality of reporting for many studies describing AI health technologies means we do not have sufficient information to fully interpret their findings.⁵ There is also evidence that medical AI can cause or contribute to discrimination, bias, and health inequity.^6–11 

In addition, safety and efficacy of AI systems needs to be formally evaluated in the local context.^12,13 Published evidence supporting the use of a particular AI model may not generalise broadly, because of variability in disease and risk factor prevalence, population characteristics, medical practice, sociocultural norms and other factors.

There are also legal considerations, including data protection and cybersecurity obligations. AI models used for medical purposes* will be medical devices, and as such need to be registered with the Medicines and Healthcare products Regulatory Agency (MHRA) and in some cases the Care Quality Commission (CQC).

^{* Medical purposes listed by the MHRA include: diagnosis, monitoring and treatment of diseases, investigation or modification of physiological processes, calculation of clinical risk, and providing clinical decisions.14}

Charting a way forwards for UK critical care

FICM recognises work led by The Health Foundation,^15–17 NHS England,^18,19 The Ada Lovelace Institute,^20–22 the MHRA and other medical regulators,^23,24 the National Institute of Health and Care Excellence,²⁵ and others.

1. As a professional Faculty we commit to align our messages and recommendations regarding medical artificial intelligence with established best practice.
2. Where best practice is yet to be established, FICM will work with external organisations to ensure the views and wishes of intensive care patients and professionals are considered.

In order for our patients to benefit from medical AI, we need models which show safety and efficacy following robust evaluation. 

3. We encourage the transparent and ethical use of healthcare data to develop AI health technologies for the benefit of all critically unwell patients.
4. We support ethically approved and methodologically sound research into the safety and efficacy of medical AI tools in the intensive care context.
5. We recommend that any intensive care department considering deploying an AI model conduct a thorough appraisal of the available evidence supporting its use. They should ensure that the model has appropriate regulatory clearances, and if it is a medical device that the way it is proposed to be used is in keeping with the model’s Intended Purpose. They should also ensure that they have sufficient local expertise to audit the model in use¹³ to detect and mitigate failure which may otherwise harm patients.
6. We also recommend that other non-AI options are explored to ensure that the AI model is the most appropriate solution to the problem in question. 

If we are to use medical AI safely in intensive care, we need our workforce to upskill. 

7. We encourage intensive care professionals to seek training opportunities to learn more about medical AI, including the NHS Fellowship in Clinical Artificial Intelligence programme (https://gstt-csc.github.io/fellowship.html). 

Large language models (LLMs) are a nascent technology that pose several challenges,²⁶ including a tendency to ‘hallucinate’. 

8. We recommend that the use of LLMs for medical purposes in intensive care occurs only as part of a medical device with regulatory clearance, or in the context of ethically approved research. 

We need models to tackle real-world needs, rather than being ‘solutions in search of a problem’. 

9. We commit to helping intensive care professionals, and critically unwell patients and those close to them, to share their expertise and insights with developers of AI solutions.

References

1. ^{Sendak MP, Ratliff W, Sarro D, et al. Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study. JMIR Medical Informatics 2020; 8: e15182.}
2. ^{Liu X, Hu P, Yeung W, et al. Illness severity assessment of older adults in critical illness using machine learning (ELDER-ICU): an international multicentre study with subgroup bias evaluation. The Lancet Digital Health 2023; 5: e657–67.}
3. ^{Plesner LL, Müller FC, Nybing JD, et al. Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact. Radiology 2023; 307: e222268.}
4. ^{Bowness JS, Laurent DB-S, Hernandez N, et al. Assistive artificial intelligence for ultrasound image interpretation in regional anaesthesia: an external validation study. British Journal of Anaesthesia 2023; 130: 217–25.}
5. ^{Martindale APL, Llewellyn CD, de Visser RO, et al. Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat Commun 2024; 15: 1619.}
6. ^{Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366: 447–53.}
7. ^{Seyyed-Kalantari L, Zhang H, McDermott MBA, Chen IY, Ghassemi M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med 2021; 27: 2176–82.}
8. ^{Cao J, Zhang X, Shahinian V, et al. Generalizability of an acute kidney injury prediction model across health systems. Nature Machine Intelligence 2022; 4: 1121–9.}
9. ^{Omiye JA, Lester JC, Spichak S, Rotemberg V, Daneshjou R. Large language models propagate race-based medicine. npj Digit Med 2023; 6: 1–4.}
10. ^{Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature 2024; 630: 625–30.}
11. ^{Hager P, Jungmann F, Holland R, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 2024; : 1–10.}
12. ^{Futoma J, Simons M, Panch T, Doshi-Velez F, Celi LA. The myth of generalisability in clinical research and machine learning in health care. The Lancet Digital Health 2020; 2: e489–92.}
13. ^{Liu X, Glocker B, McCradden MM, Ghassemi M, Denniston AK, Oakden-Rayner L. The medical algorithmic audit. The Lancet Digital Health 2022; 0. DOI:10.1016/S2589-7500(22)00003-6.}
14. ^{Medicines and Healthcare products Regulatory Agency. Medical device stand-alone software including apps (including IVDMDs). https://assets.publishing.service.gov.uk/media/64a7d22d7a4c230013bba33c… (accessed Sept 4, 2024).}
15. ^{Moulds A, Horton T. What do technology and AI mean for the future of work in health care? - The Health Foundation. The Health Foundation https://www.health.org.uk/publications/long-reads/what-do-technology-an… (accessed Sept 6, 2024).}
16. ^{Moulds A, Horton T. Which technologies offer the biggest opportunities to save time in the NHS? The Health Foundation https://www.health.org.uk/publications/long-reads/which-technologies-of… (accessed Sept 6, 2024).}
17. ^{Thornton N, Hardie T, Horton T, Gerhold M. Priorities for an AI in health care strategy. The Health Foundation https://www.health.org.uk/publications/long-reads/priorities-for-an-ai-… (accessed Sept 6, 2024).}
18. ^{Understanding healthcare workers confidence in AI. NHS England, 2022 https://digital-transformation.hee.nhs.uk/building-a-digital-workforce/… (accessed May 22, 2024).}
19. ^{Developing healthcare workers confidence in AI. NHS England, 2022 https://digital-transformation.hee.nhs.uk/building-a-digital-workforce/… (accessed May 22, 2024).}
20. ^{Groves L. Algorithmic impact assessment: a case study in healthcare. The Ada Lovelace Institute, 2022 https://www.adalovelaceinstitute.org/report/algorithmic-impact-assessme… (accessed Sept 6, 2024).}
21. ^{Machirori M. A knotted pipeline. The Ada Lovelace Institute, 2022 https://www.adalovelaceinstitute.org/report/knotted-pipeline-health-dat… (accessed Sept 6, 2024).}
22. ^{Studman A. Access denied? Socioeconomic inequalities in digital health services. The Ada Lovelace Institute, 2023 https://www.adalovelaceinstitute.org/report/healthcare-access-denied/ (accessed Sept 6, 2024).}
23. ^{AI and Digital Regulations Service for health and social care. https://digitalregulations.innovation.nhs.uk/regulations-and-guidance-f… (accessed Sept 19, 2024).}
24. ^{Medicines and Healthcare products Regulatory Agency. Software and AI as a Medical Device Change Programme - Roadmap. MHRA. https://www.gov.uk/government/publications/software-and-ai-as-a-medical… (accessed May 22, 2024).}
25. ^{Evidence standards framework for digital health technologies. National Institute for Health and Care Excellence (NICE). 2018; published online Dec 10. https://www.nice.org.uk/corporate/ecd7 (accessed Sept 6, 2024).}
26. ^{Ordish J. Large Language Models and software as a medical device. MedRegs (Medicines and Healthcare Products Regulatory Agency). 2023; published online March 3. https://medregs.blog.gov.uk/2023/03/03/large-language-models-and-softwa… (accessed Sept 24, 2024).}