Skip to content
HepiusRxivPreprints from early-career health researchers

Do not use it to guide clinical practice or personal health decisions.

Thesis MSc thesis Data available

Evaluation of an AI Primary Eye Care Agent Physician-Validated Multi-LLM-as-a-Judge Assessment Across Frontier and Local Models in Low-Resource Settings

Dagim Amare Worku1 Verified author Corresponding author, Andrew Blakie1 Corresponding author

  1. 1University of St Andrews

Supervisor: Dr Andrew Blakie, University of St Andrews.

PDF, 3.6 MB Cite Share Review or comment Get alerts Verified readers can contact the authors

Summary for everyone

A study tested "Alan," an open-source tool that helps health workers conduct eye exams using AI chatbots (large language models). Researchers ran 1,344 simulated eye consultations across seven AI models—two expensive "frontier" models and five smaller, locally runnable ones. The best local model scored 81.8 out of 100, only 2.4 points below the best frontier model (84.2). However, quality dropped sharply for weaker models. Speed, not cost, was the main practical limit: the top local setup took 65 seconds per reply, while a faster one answered in under six seconds. The authors argue regulators should treat the tool and AI model as one combined product.

Key findings

  • The best frontier configuration scored 84.2 out of 100, while the best open-weight model reached 81.8, a gap of just 2.4 points
  • Quality fell steeply to a floor of 51.4.
  • The LLM judges reproduced the ophthalmologists' model ranking, but their pass rate was substantially lower (54.1% vs 79.8%)

Abstract

BackgroundAn estimated 1.1 billion people live with untreated vision impairment, most of them in lowand middle-income countries where eye specialists are scarce. One response to that scarcity is Alan, an open-source clinical scaffold that can guide a frontline eye consultation on any large language model (LLM). Frontier models are costly and cloud-bound, while small open-weight models can run inside the clinic itself, yet no study had measured what is lost when the model beneath the scaffold changes.

MethodsAlan was evaluated on seven models, two frontier and five open-weight, each with extended reasoning ("thinking") enabled and disabled. Every configuration worked through 48 ophthalmic cases presented by two simulated health-worker personas, yielding 1,344 consultations in total. Local models ran on the University of St Andrews Hypatia HPC cluster with energy measured throughout. A blinded panel of three LLM judges then scored every dialogue on a five-criterion weighted rubric, anchored by two human ophthalmologists.

ResultsThe results were closer than expected. The best frontier configuration scored 84.2 out of 100, while the best open-weight model reached 81.8, a gap of just 2.4 points (95% CI 0.3 to 4.8). Below this leading group, however, quality fell steeply to a floor of 51.4. The LLM judges reproduced the ophthalmologists' model ranking, but their pass rate was substantially lower (54.1% vs 79.8%), which supports their use for comparative ranking rather than as a direct estimate of clinical acceptability. In deployment terms, latency rather than cost emerged as the principal constraint: the highest-scoring local configuration took 64.8 seconds per reply, whereas the fastest deployable one answered in under six seconds while scoring 77.0.

ConclusionsFindings show that Alan produced different clinical behaviour on different hosts. This reinforces the case for regulators to treat the scaffold-model combination as one inseparable product, and for health systems to demand evidence from hardware and configurations their clinics can realistically operate.

Keywords

Alan scaffoldOpen-weight vs. frontier modelsLLM-as-judgeHealth equity

Declarations

Ethics approval
None needed
Consent
Not applicable
Funding
No external funding
Competing interests
None
Data availability
Available here and on a github repo
Author contributions
Research
Use of AI tools
None
Supervisor
Dr Andrew Blakie (MSc, University of St Andrews)

References

  1. Burton MJ, Ramke J, Marques AP, et al. The Lancet Global Health Commission on
  2. Global Eye Health: vision beyond 2020. Lancet Glob Health. 2021;9(4):e489-e551.
  3. doi:10.1016/S2214-109X(20)30488-5
  4. WHO. World Report on Vision. October 2019. 2019.
  5. Lansingh V. Estimated number of ophthalmologists worldwide (International Council of
  6. Ophthalmology update): will we meet the needs? Br J Ophthalmol. Published online
  7. 2019.
  8. Andriamanjato HH, Mathenge W, Kalua K, Courtright P, Lewallen S. Task shifting in
  9. primary eye care: how sensitive and specific are common signs and symptoms to predict
  10. conditions requiring referral to specialist eye personnel? Hum Resour Health.
  11. 2014;12(S1):S3. doi:10.1186/1478-4491-12-S1-S3
  12. Kalua K, Gichangi M, Barassa E, Eliah E, Lewallen S, Courtright P. Skills of general
  13. health workers in primary eye care in Kenya, Malawi and Tanzania. Hum Resour Health.
  14. 2014;12(S1):S2. doi:10.1186/1478-4491-12-S1-S2
  15. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180.
  16. Thirunavukarasu AJ, Mahmood S, Malem A, et al. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digit Health. 2024;3(4):e0000341.
  17. WJW. Alan: Open-source eye, ear and skin learning prompt for concise point-of-care
  18. guidance. Published online 2026. Accessed July 11, 2026. https://github.com/Spider201866/alan
  19. Joanne. Can Artificial Intelligence Platforms Diagnose Eye Disease? An Evaluation of
  20. the Use of Super Prompt Agent (Alan) vs. Large Language Model (ChatGPT-4o Mini).
  21. B.Sc (Honours) Medicine. University of St Andrews, School of Medicine; 2025.
  22. Dennstädt F, Hastings J, Putora PM, Schmerder M, Cihoric N. Implementing large language models in healthcare while balancing control, collaboration, costs and security.
  23. NPJ Digit Med. 2025;8(1):143.
  24. Riedemann L, Labonne M, Gilbert S. The path forward for large language models in medicine is open. Npj Digit Med. 2024;7(1):339.
  25. WHO. Ethics and Governance of Artificial Intelligence for Health: WHO Guidance.
  26. World Health Organization; 2021.
  27. Pente V, Bechange S, Jolley E, et al. Task-shifting eye care to ophthalmic community
  28. health officers (OCHO) in Sierra Leone: A qualitative study. J Glob Health.
  29. 2021;11:07001. doi:10.7189/jogh.11.07001
  30. ______Chapter - References
  31. 59
  32. GD5999_Student-ID:250009792
  33. Blaikie A, Sandford-Smith J, Tuteja SY, Williams CD, O’Callaghan C. Arclight: a pocket
  34. ophthalmoscope for the 21st century. Bmj. 2016;355.
  35. Blundell R, Roberts D, Fioratou E, et al. Comparative evaluation of a novel solar powered
  36. low-cost ophthalmoscope (Arclight) by eye healthcare workers in Malawi. BMJ Innov.
  37. 2018;4(2):98-102. doi:10.1136/bmjinnov-2017-000225
  38. Kousha O, Blaikie A. The Arclight and how to use it. Community Eye Health.
  39. 2019;32(107):50-51.
  40. Malik ANJ, Mnedeme GF, Iriya N, et al. Evaluation of primary healthcare worker training
  41. to screen children under 5 years of age with a low-cost alternative to the direct ophthalmoscope, the “Arclight”, as part of the Integrated Management of Newborn and Childhood Illness (IMNCI) programme in Tanzania. BMJ Paediatr Open. 2025;9(1):e003520.
  42. doi:10.1136/bmjpo-2025-003520
  43. Windle T, Maliko F, Burgiss-Kasthala S, Blaikie A. Long-term impact of Training the
  44. Trainers program on Primary Eye Care networks in Malawi using the Arclight Project
  45. package. medRxiv. Published online 2026:2026.03. 31.26349901.
  46. WHO. WHO Guideline: Recommendations on Digital Interventions for Health System
  47. Strengthening: Web Supplement 2: Summary of Findings and GRADE Tables. World
  48. Health Organization; 2019.
  49. Carchiolo V, Malgeri M. Trends, Challenges, and Applications of Large Language Models in Healthcare: A Bibliometric and Scoping Review. Future Internet. 2025;17(2):76.
  50. doi:10.3390/fi17020076
  51. Chen H, Zeng D, Qin Y, et al. Large language models and global health equity: a roadmap
  52. for equitable adoption in LMICs. Lancet Reg Heal Pac. 2025;63.
  53. El Arab RA, Al Moosa OA. Systematic review of cost effectiveness and budget impact
  54. of artificial intelligence in healthcare. NPJ Digit Med. 2025;8(1):548.
  55. Husereau D, Drummond M, Augustovski F, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance
  56. for health economic evaluations. MDM Policy Pract. 2022;7(1):23814683211061097.
  57. doi:10.1177/23814683211061097
  58. Amugongo LM, Mascheroni P, Brooks S, Doering S, Seidel J. Retrieval augmented generation for large language models in healthcare: A systematic review. Liu X, ed. PLOS
  59. Digit Health. 2025;4(6):e0000877. doi:10.1371/journal.pdig.0000877
  60. Rooney K. OpenEvidence, the “ChatGPT for doctors,” doubles valuation to $12 billion.
  61. CNBC. January 21, 2026. Accessed June 20, 2026.
  62. https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doctors-doubles-valuation-to-12-billion.html
  63. Philip S, Kurain R. OpenEvidence. J Med Libr Assoc. 2026;114(1):86-87.
  64. doi:10.5195/jmla.2026.2247
  65. ______Chapter - References
  66. 60
  67. GD5999_Student-ID:250009792
  68. Wolters Kluwer. Wolters Kluwer releases validation framework for evaluating Clinical
  69. AI at point of care. Published online May 21, 2026.
  70. https://www.wolterskluwer.com/en/news/validation-framework-evaluating-clinical-aipoint-of-care
  71. Vishwanath K, Alyakin A, Ghosh M, et al. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks. Nat Med. 2026;32(7):2405-
  72. 2409. doi:10.1038/s41591-026-04431-5
  73. Sonoda Y, Kurokawa R, Hagiwara A, et al. Structured clinical reasoning prompt enhances
  74. LLM’s diagnostic capabilities in diagnosis please quiz cases. Jpn J Radiol.
  75. 2025;43(4):586-592.
  76. Wang S, Tang Z, Yang H, et al. A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains. Npj Digit Med. 2025;9(1):91.
  77. doi:10.1038/s41746-025-02277-8
  78. Korom R, Kiptinness S, Adan N, et al. AI-based Clinical Decision Support for Primary
  79. Care: A Real-World Study. arXiv. Preprint posted online July 22,
  80. 2025:arXiv:2507.16947. doi:10.48550/arXiv.2507.16947
  81. Rombaut B. Inside the scaffold: A source-code taxonomy of coding agent architectures.
  82. ArXiv Prepr ArXiv260403515. Published online 2026.
  83. Starace J. Scaffold Effects on GAIA: A Controlled Comparison. arXiv. Preprint posted
  84. online June 7, 2026:arXiv:2606.08529. doi:10.48550/arXiv.2606.08529
  85. Laban P, Hayashi H, Zhou Y, Neville J. LLMs Get Lost In Multi-Turn Conversation.
  86. arXiv. Preprint posted online 2025. doi:10.48550/ARXIV.2505.06120
  87. Kruthof G. Models recall what they violate: Constraint adherence in multi-turn LLM ideation. ArXiv Prepr ArXiv260428031. Published online 2026.
  88. Manczak B, Lin E, Eiras F, O’Neill J, Mugunthan V. Shallow robustness, deep vulnerabilities: Multi-turn evaluation of medical llms. In: The Second Workshop on GenAI for
  89. Health: Potential, Trust, and Policy Compliance. 2025.
  90. Mittal S, Aggarwal Y. Evaluation of Large Language Models in the Diagnosis, Urgency
  91. Triage, and Initial Management of Ophthalmic Emergencies. Cureus. 2026;18(1).
  92. Lin X, Yang Y, Ren Y, Wang L, Cai L. Comparative Evaluation of Deep-Reasoning
  93. Large Language Models for Ophthalmic Emergencies. Ophthalmol Sci. Published online
  94. 2026:101158.
  95. Cardoso-Teixeira P, Ambrósio JA, Garcia M, Chibante-Pedro J, Figueiredo L. Comparative Performance of Large Language Models in Ophthalmology Referral Triage. Cureus.
  96. 2026;18(1).
  97. Badshah S, Sajjad H. Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA. In: Proceedings of the 9th Widening NLP Workshop. 2025:251-
  98. ______Chapter - References
  99. 61
  100. GD5999_Student-ID:250009792
  101. Shi L, Ma C, Liang W, Diao X, Ma W, Vosoughi S. Judging the judges: A systematic
  102. study of position bias in llm-as-a-judge. In: Proceedings of the 14th International Joint
  103. Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific
  104. Chapter of the Association for Computational Linguistics. 2025:292-314.
  105. Zheng L, Chiang WL, Sheng Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv. Preprint posted online 2023. doi:10.48550/ARXIV.2306.05685
  106. Gu J, Jiang X, Shi Z, et al. A survey on LLM-as-a-judge. The Innovation.
  107. 2026;7(6):101253. doi:10.1016/j.xinn.2025.101253
  108. Hu Z, Song L, Zhang J, et al. Explaining Length Bias in LLM-Based Preference Evaluations. arXiv. Preprint posted online 2024. doi:10.48550/ARXIV.2407.01085
  109. Blunch NJ. Position bias in multiple-choice questions. J Mark Res. 1984;21(2):216-220.
  110. Li H, Dong Q, Chen J, et al. LLMs-as-Judges: A Comprehensive Survey on LLM-based
  111. Evaluation Methods. arXiv. Preprint posted online December 10,
  112. 2024:arXiv:2412.05579. doi:10.48550/arXiv.2412.05579
  113. Panickssery A, Bowman SR, Feng S. Llm evaluators recognize and favor their own generations. In: The Thirty-Eighth Annual Conference on Neural Information Processing
  114. Systems. 2024.
  115. Li L, Li D, Chen C, et al. Llm-as-a-judge in healthcare: A scoping analysis of applications,
  116. methods, and human alignment. ArXiv Prepr ArXiv260525273. Published online 2026.
  117. Philipp W, Fassbender F, Langer T, et al. Clinician-Level Agreement Without Clinical
  118. Caution: LLM Evaluator Limits in Medical AI Benchmarking. arXiv. Preprint posted
  119. online July 2, 2026. doi:10.48550/ARXIV.2607.01103
  120. DeLucia A, Huang H, Joshi S, Yarmohammadi M, Hassoon A, Dredze M. Same verdict,
  121. different reasons: Llm-as-a-judge and clinician disagreement on medical chatbot completeness. ArXiv Prepr ArXiv260416383. Published online 2026.
  122. Valois PHV, Souza LS, Shimomoto EK, Fukui K. Frame Representation Hypothesis:
  123. Multi-Token LLM Interpretability and Concept-Guided Text Generation. Trans Assoc
  124. Comput Linguist. 2025;13:1436-1458. doi:10.1162/TACL.a.48
  125. Xu F, Hao Q, Zong Z, et al. Towards Large Reasoning Models: A Survey of Reinforced
  126. Reasoning with Large Language Models. arXiv. Preprint posted online 2025.
  127. doi:10.48550/ARXIV.2501.09686
  128. Chen Y, Benton J, Radhakrishnan A, et al. Reasoning models don’t always say what they
  129. think. ArXiv Prepr ArXiv250505410. Published online 2025.
  130. Arcuschin I, Janiak J, Krzyzanowski R, Rajamanoharan S, Nanda N, Conmy A. Chainof-thought reasoning in the wild is not always faithful. ArXiv Prepr ArXiv250308679.
  131. Published online 2025.
  132. ______Chapter - References
  133. 62
  134. GD5999_Student-ID:250009792
  135. WHO. Ethics and Governance of Artificial Intelligence for Health: Large Multi-Modal
  136. Models. WHO Guidance. 1st ed. World Health Organization; 2024.
  137. Malpani R, Moodley K. Large multimodal models: boon or burden for low- and middleincome countries. Int Health. 2025;17(5):853-855. doi:10.1093/inthealth/ihae073
  138. Kapoor S, Bommasani R, Klyman K, et al. On the societal impact of open foundation
  139. models. ArXiv Prepr ArXiv240307918. Published online 2024.
  140. White M, Haddad I, Osborne C, et al. The model openness framework: Promoting completeness and openness for reproducibility, transparency, and usability in artificial intelligence. ArXiv Prepr ArXiv240313784. Published online 2024.
  141. Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3(11):e745-e750.
  142. WHO. Energizing Health: Accelerating Electricity Access in Health-Care Facilities.
  143. World Health Organization; 2023.
  144. Carvalho E, Mascarenhas M, Pinheiro F, et al. Predetermined change control plans: guiding principles for advancing safe, effective, and high-quality AI-ML technologies. JMIR
  145. AI. 2025;4:e76854.
  146. Pandey A, Gupta P, Sehgal NK. Regulatory Requirements and Risk Management for
  147. AI/ML-Enabled Medical Devices. In: Artificial Intelligence in Medical Software.
  148. Springer; 2026:349-384.
  149. OpenAI. Introducing GPT-5.4. OpenAI. July 30, 2026. https://openai.com/index/introducing-gpt-5-4/
  150. Anthropic. Introducing Claude Sonnet 5. 2026. Accessed July 2, 2026. https://www.anthropic.com/news/claude-sonnet-5
  151. Artificial Analysis. Artificial Analysis: AI Model & API Providers Analysis. Artificial
  152. Analysis. Accessed August 1, 2026. https://artificialanalysis.ai
  153. Tu T, Schaekermann M, Palepu A, et al. Towards conversational diagnostic artificial intelligence. Nature. 2025;642(8067):442-450.
  154. Johri S, Jeong J, Tran BA, et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med. 2025;31(1):77-86.
  155. doi:10.1038/s41591-024-03328-5
  156. Hicks RS, Trofimov M, Lim D, et al. HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. ArXiv Prepr ArXiv260427470. Published online
  157. 2026.
  158. Kocaman V, Kaya MA, Feier AM, Talby D. Clinical large language model evaluation by
  159. expert review (CLEVER): framework development and validation. JMIR AI.
  160. 2025;4(1):e72153.
  161. ______Chapter - References
  162. 63
  163. GD5999_Student-ID:250009792
  164. Pemberton JD, Fay A. Oral Exam Study Guide. https://www.researchgate.net/profile/John-Pemberton-4/publication/284732620_Ophthalmology_Clinical_Vignettes_Oral_Exam_Study_Guide/links/5658b11708ae1ef9297df831/OphthalmologyClinical-Vignettes-Oral-Exam-Study-Guide.pdf
  165. Schmidgall S, Ziaei R, Harris C, et al. AgentClinic: a multimodal benchmark for toolusing clinical AI agents. Npj Digit Med. Published online 2026.
  166. Li D, Lebai Lutfi S. Large Language Model–Based Virtual Patient Systems for HistoryTaking in Medical Education: Comprehensive Systematic Review. JMIR Med Inform.
  167. 2026;14:e79039. doi:10.2196/79039
  168. Luo H, Laban G. SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation. ArXiv Prepr ArXiv260409212. Published online 2026.
  169. Peeperkorn M, Kouwenhoven T, Brown D, Jordanous A. Is temperature the creativity
  170. parameter of large language models? ArXiv Prepr ArXiv240500492. Published online
  171. 2024.
  172. Peng D, Wang Y, Schoeffler A, et al. SycoEval-EM: Sycophancy Evaluation of Large
  173. Language Models in Simulated Clinical Encounters for Emergency Care. arXiv. Preprint
  174. posted online July 7, 2026:arXiv:2601.16529. doi:10.48550/arXiv.2601.16529
  175. Verga P, Hofstatter S, Althammer S, et al. Replacing judges with juries: Evaluating llm
  176. generations with a panel of diverse models. ArXiv Prepr ArXiv240418796. Published
  177. online 2024.
  178. Schroeder K, Wood-Doughty Z. Can you trust llm judgments? reliability of llm-as-ajudge. ArXiv Prepr ArXiv241212509. Published online 2024.
  179. Cochran WG. Sampling Techniques. john wiley & sons; 1977.
  180. Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull.
  181. 1979;86(2):420-428.
  182. Cohen J. Weighted kappa: Nominal scale agreement provision for scaled disagreement or
  183. partial credit. Psychol Bull. 1968;70(4):213.
  184. Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007;1(1):77-89.
  185. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543-549.
  186. Gwet KL. Computing inter‐rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61(1):29-48.
  187. Bland JM, Altman DG. Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat. 2007;17(4):571-582.
  188. ______Chapter - References
  189. 64
  190. GD5999_Student-ID:250009792
  191. Renze M. The effect of sampling temperature on problem solving in large language models. In: Findings of the Association for Computational Linguistics: EMNLP 2024.
  192. 2024:7346-7356.
  193. University of St Andrews. High performance computing. April 21, 2026. Accessed April
  194. 21, 2026. https://www.st-andrews.ac.uk/high-performance-computing/
  195. SchedMD LLC. Slurm Workload Manager - Documentation. 2023. Accessed August 1,
  196. 2026. https://slurm.schedmd.com/archive/slurm-23.02.6/
  197. NVIDIA Corporation. CUDA 12.5 Release Notes — Release Notes 12.5 documentation.
  198. 2024. Accessed January 6, 2026. https://docs.nvidia.com/cuda/archive/12.5.0/cudatoolkit-release-notes/index.html
  199. OpenRouter. OpenRouter. OpenRouter. Accessed June 1, 2026. https://openrouter.ai
  200. Google, Abd SE, Aggarwal V, et al. Gemma 4 technical report. ArXiv Prepr
  201. ArXiv260702770. Published online 2026.
  202. Yang A, Li A, Yang B, et al. Qwen3 Technical Report. arXiv. Preprint posted online
  203. 2025. doi:10.48550/ARXIV.2505.09388
  204. Qwen. Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. Qwen. April 22,
  205. 2026. Accessed June 2, 2026. https://qwenlm.github.io/blog/qwen3.6-27b/
  206. Python.org. Python Release Python 3.13.14. Python.org. 2026. Accessed June 9, 2026.
  207. https://www.python.org/downloads/release/python-31314/
  208. Efron B, Narasimhan B. The Automatic Construction of Bootstrap Confidence Intervals.
  209. J Comput Graph Stat. 2020;29(3):608-619. doi:10.1080/10618600.2020.1714633
  210. Woolson RF. Wilcoxon signed‐rank test. Wiley Encycl Clin Trials. Published online
  211. 2007:1-3.
  212. Spearman C. Correlation calculated from faulty data. Br J Psychol. 1910;3(3):271.
  213. Noether GE. Why kendall tau. Teach Stat. 1981;3(2):41-43.
  214. NVIDIA. NVIDIA System Management Interface. NVIDIA Developer. 2026. Accessed
  215. June 2, 2026. https://developer.nvidia.com/system-management-interface
  216. Ofgem. Energy price cap unit rates and standing charges | Ofgem. 2026. Accessed July 2,
  217. 2026. https://www.ofgem.gov.uk/information-consumers/energy-advice-households/energy-price-cap-unit-rates-and-standing-charges
  218. Donnellan D, Lawrence A, Bizo D, et al. Uptime Institute Global Data Center Survey
  219. 2025. Published online 2025.
  220. Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of
  221. large language models: a systematic review. Jama. 2025;333(4):319-328.
  222. ______Chapter - References
  223. 65
  224. GD5999_Student-ID:250009792
  225. Asiedu M, Tomasev N, Ghate C, et al. Contextual evaluation of large language models
  226. for classifying tropical and infectious diseases. ArXiv Prepr ArXiv240909201. Published
  227. online 2024.
  228. Olatunji T, Nimo C, Owodunni A, et al. AfriMed-QA: a Pan-African, multi-specialty,
  229. medical question-answering benchmark dataset. ArXiv Prepr ArXiv241115640. Published
  230. online 2024.
  231. Gao Y, Feng Y, Jahng SG. Implicit bias in digital health: systematic biases in large language models’ representation of global public health attitudes and challenges to health
  232. equity. Front Public Health. 2025;13:1705082.
  233. Pfohl SR, Cole-Lewis H, Sayres R, et al. A toolbox for surfacing health equity harms and
  234. biases in large language models. Nat Med. 2024;30(12):3590-3600.
  235. WHO. Package of Eye Care Interventions. 1st ed. World Health Organization; 2022.