Faster yet Reliable Systematic Reviews with AI: The CESAR Study
Authors: Noosheen R Tahmasebi1, Dr Angelika Eisele-Metzger2, Gerald Gartlehner3
1. Institute for Evidence in Medicine, Medical Faculty and Medical Center - University of Freiburg
2. Angelika Eisele-Metzger, Cochrane Germany
3. University for Continuing Education Krems, Krems an der Donau
Imagine the following situation: Someone notices a new, slightly irregular mole on their skin. They take a photo, upload it to a chatbot and ask if it looks dangerous. The chatbot answers confidently: probably nothing to worry about. They close the app and carry on with their week because the last ten times they used this specific chatbot, they received really good advice. This is a small human moment, repeated countless times a day around the world. It is also, according to the emerging evidence, a genuinely risky one. A 2025 study comparing ChatGPT-4 Omni and ChatGPT-4 Turbo on dermoscopic melanoma images concluded that these general-purpose chatbots cannot yet be relied on to identify skin cancer, and that meaningful improvement is needed before they can be trusted.
AI in Health Care
Depending on where you stand, artificial intelligence (AI) in health care means different things. For a surgeon, it is a robotic arm holding a scalpel steadier than a human hand. For a radiologist, it is software flagging a shadow on a scan at hour nine of a long shift. For a patient, it is an app answering a health question that they're too embarrassed to ask anyone about. Most clinical guidelines, whether it is the blood pressure target a doctor works toward or the vaccination schedule a paediatrician follows, trace back to someone gathering all the relevant studies on a set of questions and working out what the evidence collectively shows. This process is called a systematic review (SR) and is traditionally carried out by human specialists in the field. Conducting SRs, however, is a resource-consuming task that can often take more than a year.
Systematic Reviews and AI
An SR is a structured, pre-registered way of answering a health-related question by combining the high-quality studies that have tested it. Review teams write a protocol in advance, search relevant databases, screen every record that turns up against fixed inclusion criteria, pull the key data out of every study that qualifies, assess the methodological quality of those studies and combine their results statistically. When performed properly, SRs are widely considered the most reliable summary of medical evidence. Cochrane, an international network of researchers, is an organisation closely associated with conducting SRs to the highest standard.
Although SRs are our best option to answer a clinical question, there is a catch: the average SR takes months or even years from registration to publication, much of it spent screening tens of thousands of database results by hand to find the handful of studies that end up qualifying. By the time a review is finished, some of the evidence inside it can already be out of date. Moreover, humans make mistakes. Methodological studies suggest that around 15% of extracted data items in SRs contain errors. Most are minor, such as rounding mistakes, but more consequential errors also occur. It, therefore, seems clear that we turn to AI for help.
AI tools now have the potential to considerably reduce the workload involved. Yet the important question is not simply whether AI can help conduct an SR, but how well it performs and how reliable it is. SRs form part of the evidence base underlying the clinical guidelines and AI-powered medical applications (such as AI-Driven Clinical Decision Support Systems) that health care professionals ultimately rely on. What we, therefore, need to ask is whether the evidence itself still can be trusted once AI helps synthesise it.
Faster, but Still Reliable, or Even Better?
AI-assisted screening and data extraction tools are not new; reviewers have been using machine learning for years, and adoption is already high. A 2022 survey of 253 SRs, health technology assessors and guideline developers found that 89% had used an automation tool, most often for screening, and most reported time savings. Yet the same survey found that the tools were frequently abandoned, and the single biggest barrier was not the tools' accuracy; instead, it was reviewers not knowing enough about how well a given tool actually performed. A separate review of 47 studies on these tools reached a similar conclusion: while abstract screening automation has matured, data extraction automation is still immature, and the evidence behind individual tools tends to be scattered across small studies. This highlights the need for real-world workflow validations rather than relying on fragmented research to set the standard. In other words, the tools are already in reviewers' hands and have the potential to improve efficiency and quality; what is missing is a reliable, continuously updated answer to the question, ‘how well do they actually work under real-world conditions?’
CESAR: Cochrane Evaluation of (Semi-)Automated Review Methods
To close this knowledge gap, we designed the Cochrane CESAR project. Rather than testing an AI review tool once and publishing a single verdict, CESAR borrows a design from clinical drug trials called an adaptive platform trial: several treatments are tested against a shared control group at the same time, with new treatments added or dropped based on their performance during the study, all without restarting the study. CESAR does the same thing, except the ‘treatments’ are AI tools and the ‘control’ is standard, human-only review work. It is all embedded directly inside real, ongoing Cochrane reviews across four parallel study arms. In the first arm, two human reviewers work independently, essentially following the standard double-review process that Cochrane teams already use. The next two arms are semi-automated: two human reviewers, each paired with one of the two AI tools, which have been selected for the study. The fourth arm is fully automated: each AI tool works on its own, with no human involved in the decision-making, to see whether the software alone can match what two trained reviewers produce together. Because the design is adaptive, better versions of these tools, or entirely new ones that clear CESAR's selection process, can be swapped into the semi- and fully automated arms as they become available.
Accuracy, Efficiency, Usability and Response Stability
CESAR tests AI tools on the three most time-consuming tasks in a review: sorting titles and abstracts for relevance, checking full-text articles against the inclusion criteria and extracting the relevant data, like sample sizes, outcomes, definitions, effect estimates, etc., from each study. The results of the CESAR project aim to answer more than ‘was the answer correct?’ CESAR also tracks efficiency (how much reviewer time is genuinely saved), the resulting impact of any errors a tool makes and usability for the reviewers actually working with it. Additionally, it assesses one metric that is particularly important for the reliability of the results: response stability.
Response stability is not about whether the AI screened an abstract correctly; it is about whether it would give the same answer if you asked it again. This matters because, unlike older screening software that always gives the same result, AI tools often involve some randomness. Therefore, the same abstract could be included one time and excluded the next. A tool might score well on a single test but still be unreliable in everyday use. This is an issue that a single accuracy score can hide. Therefore, we must ask the following: When AI is used to build the evidence, can it be trusted to hold up when checked, again?
Best-Case Scenario: From Faster Reviews to Better Care
If CESAR's project design holds up, the most direct payoff will be gains in terms of speed without sacrificing robustness: SRs that currently take well over a year could be completed in a fraction of that time and perhaps even with fewer errors, meaning updated, trustworthy guidelines building on their evidence reach clinicians and their patients sooner. That matters most for the reviews under the most time pressure, like the living and rapid reviews used during disease outbreaks, where a slow, purely manual mode of operation simply is not fast enough to keep guidance current.
It is easy to imagine a future in which AI-powered health care becomes increasingly sophisticated. However, using AI tools requires a level of digital comfort, language fluency or basic internet access that older adults, lower-income communities and people in under-resourced regions often lack. At first, it may seem that the people who could benefit from better health information may be the least able to interact directly with the technology providing it. Yet the impact of AI-assisted evidence synthesis ultimately extends far beyond the review team: evidence feeds into SRs, SRs inform clinical guidelines and those guidelines shape the care that patients receive. A review strengthened by an AI tool that works reliably improves the recommendations that clinicians follow. In that sense, if AI helps a team of reviewers and guideline developers release an update in weeks instead of months or years, every patient downstream could benefit, including those who never interact with an AI tool themselves. This may be the most meaningful role for AI in evidence-based health care: helping trustworthy evidence reach patients sooner, through the clinicians and guidelines that shape their care.
References
Barger, L. K., Ayas, N. T., Cade, B. E., Cronin, J. W., Rosner, B., & Czeisler, C. A. (2006). Impact of extended-duration shifts on medical errors, adverse events, and attentional failures. PLoS Medicine, 3(12), e487. https://doi.org/10.1371/journal.pmed.0030487
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017). Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, 7(2), e012545. https://doi.org/10.1136/bmjopen-2016-012545
Elhaddad, M., & Hamam, S. (2024). AI-driven clinical decision support systems: An ongoing pursuit of potential. Cureus, 16(4). https://doi.org/10.7759/cureus.57728
Gartlehner, G., Banda, S., Callaghan, M., Chase, J. A., Dobrescu, A., Eisele-Metzger, A., Flemyng, E., Gardner, S., Griebler, U., Helfer, B., Jemiolo, P., Macura, B., Minx, J. C., Noel-Storr, A., Tahmasebi, N. R., Sharifan, A., Meerpohl, J. J., & Thomas, J. (2026). Cochrane Evaluation of (Semi-)Automated Review Methods (CESAR): Protocol for an adaptive platform study within reviews. Journal of Clinical Epidemiology, 198, 112390. https://doi.org/10.1016/j.jclinepi.2026.112390
Haddaway, N. R., & Westgate, M. J. (2019). Predicting the time needed for environmental systematic reviews and systematic maps. Conservation Biology, 33(2), 434–443. https://doi.org/10.1111/cobi.13231
Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42. https://doi.org/10.1016/j.jclinepi.2021.12.005
Mathes, T., Klaßen, P. & Pieper, D. (2017). Frequency of data extraction errors and methods to increase data extraction quality: A methodological review. BMC Medical Research Methodology, 17, article number 152. https://doi.org/10.1186/s12874-017-0431-4
Petroccia, S., & Esposito, M. (2026). The digital divide as a determinant of inequality in later life: A systematic review. Salute e Società, 1, 129–142. https://doi.org/10.3280/SES2026-001009
Sattler, S. S., Chetla, N., Chen, M., Hage, T. R., Chang, J., Guo, W. Y., & Hugh, J. (2025). Evaluating the diagnostic accuracy of ChatGPT-4 Omni and ChatGPT-4 Turbo in identifying melanoma: Comparative study. JMIR Dermatology, 8, e67551. https://doi.org/10.2196/67551
Scott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2022). Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: A survey. Journal of Clinical Epidemiology, 138, 80–94. https://doi.org/10.1016/j.jclinepi.2021.06.030
To link to this article - DOI: https://doi.org/10.70253/QTIY7441
Disclaimer
The views expressed in this World EBHC Day Blog, as well as any errors or omissions, are the sole responsibility of the author and do not represent the views of the World EBHC Day Steering Committee, Official Partners or Sponsors; nor does it imply endorsement by the aforementioned parties.
