Where human judgement matters
Author: Hanan Khalil1,2
1. La Trobe University
2. JBI Scoping Review Methodology Group
World EBHC Day returns on 20 October for its seventh year, and this year's theme, Evidence and AI: People at the Centre, could not be better timed for those of us who work in evidence synthesis. Over the past two decades, I have led and contributed to evidence syntheses across public health, pharmacy, aged care and health services research, including developing JBI guidance for scoping reviews and contributing to reporting standards. In recent years, artificial intelligence (AI) has become part of almost every review I have undertaken, and I have spent considerable time studying these tools as well as using them. I am not a sceptic. But the campaign asks how we keep people at the centre, and evidence synthesis is a good place to answer because it shows precisely where the machine stops and the person begins.
What automation does well
Searching, deduplication, title-and-abstract screening and extracting study characteristics into a table are steps that review teams traditionally spent months completing. In my own scoping reviews, I have observed AI-assisted tools reduce the time required for these tasks, allowing reviewers to focus more attention on interpretation and stakeholder engagement. Text analysis can tell us, in minutes, which terms dominate a body of literature and how often. For a field that has always been limited by the sheer labour of reading, this is a genuine gain, and I use these tools in almost every review I lead.
Three decisions the machine does not make
What the question is. Before any tool runs, someone decides which population, concept and context matter, which terms to search and what counts as eligible. Those choices determine whose evidence will be visible and whose will be absent. A model handed a corpus will reproduce the assumptions that built it, faithfully and at speed. Automation does not remove that responsibility; it moves it upstream, to design, where it is easier to overlook.
What a concept means. Many reviews now exist because a term is used everywhere and defined nowhere. Deciding that a sentence is a definition rather than a description, noticing that an author has introduced a new definition without explicitly identifying it as such and synthesising a working definition from competing accounts are acts of judgement. The distinction between a description, a formal definition and a working interpretation is often not apparent from the text alone. It depends on how the concept will be used and by whom.
What to do with the answer. A synthesis is a hypothesis about a field until the people it concerns have tested it. Stakeholder consultation is one of the most valuable stages of the scoping review framework, yet research shows that consultation is not routinely undertaken and, when it is included, its process and effects are often incompletely reported (Pollock et al., 2022). As the mechanical stages accelerate, this is the stage that should receive more of our time, not less.
I experienced this in work examining palliative care assessment in residential aged care. Our initial synthesis focused mainly on the clinical indicators and measurable outcomes reported in the literature (Aakattaa and Khalil, 2026). Consultation with aged care providers, clinicians and community representatives drew attention to communication, care transitions and family involvement, issues that were important in practice but less visible in the published studies. Their perspectives did not overturn the findings, but they enriched our interpretation by helping us distinguish between what was most frequently reported and what mattered most in practice. No AI tool could have made that distinction for us; it emerged through conversation with the people who use and experience the evidence.
Keeping people at the centre
For evidence synthesis, I would translate the theme into three commitments. Declare the boundary: state which steps were automated and which were judged, and by whom, as part of the method rather than a footnote. Consult before you conclude: if a concept describes people, they belong in the room before the synthesis is written. And treat the search and eligibility criteria as an argument, not a filter, because they decide whose knowledge counts.
AI will make evidence synthesis faster, cheaper and more comprehensive, and I welcome it. What it will not do is decide what question is worth asking, what a concept means or whether an answer serves the people it is about. Those responsibilities remain ours. That is what I will be arguing on 20 October, and I hope colleagues across the JBI Collaboration and the wider evidence community will join the conversation, share their own experiences and keep people at the centre of it. #WorldEBHCDay
Key messages
1. AI can improve the efficiency of evidence synthesis, but human judgement remains essential for defining questions, interpreting concepts and applying findings
2. Stakeholder consultation becomes more, not less, important in an AI-enabled evidence ecosystem because people provide the context and meaning that automation cannot
References
Aakataa, M., & Khalil, H. (2026). A scoping review of palliative care assessment tools. BMC Palliative Care, 25, 188. https://doi.org/10.1186/s12904-026-02099-9
Pollock, D., Alexander, L., Munn, Z., Peters, M. D. J., Khalil, H., Godfrey, C. M., McInerney, P., Synnot, A., & Tricco, A. C. (2022). Moving from consultation to co-creation with knowledge users in scoping reviews: Guidance from the JBI Scoping Review Methodology Group. JBI Evidence Synthesis, 20(4), 969–979. https://doi.org/10.11124/JBIES-21-00416
To link to this article - DOI: https://doi.org/10.70253/XOBF3426
Disclaimer
The views expressed in this World EBHC Day Blog, as well as any errors or omissions, are the sole responsibility of the author and do not represent the views of the World EBHC Day Steering Committee, Official Partners or Sponsors; nor does it imply endorsement by the aforementioned parties.
