AI can’t lead evidence synthesis (yet) – perspectives from Cochrane Netherlands
Author: Dr Kim van der Braak1,2 , Hanna J de Vrueh1
1. Cochrane Netherlands
2. Julius Center for Health Sciences and Primary Care
Artificial intelligence (AI) is changing the way we, as healthcare professionals, analyse, summarise and present (health) information. As a national group of Cochrane, evidence syntheses (systematic reviews) that support evidence-based healthcare are at the heart of our daily work. For example, we produce evidence syntheses for the Dutch National Healthcare Institute, informing decisions on which healthcare services should be reimbursed at a minimum. Other evidence syntheses projects directly inform clinical guidelines and impact health decisions involving people’s lives. The increasing demand for timely and trustworthy health information far exceeds the resources available to evidence synthesis groups worldwide. This highlights the need for more efficient approaches to evidence synthesis and dissemination. It is, therefore, not surprising that AI is being introduced to nearly every step of evidence synthesis. Currently, AI tools exist for several (semi-)automated processes, like searching, screening, data extraction and quality assessment.
However, it is questionable whether we can trust the information generated by these tools, particularly generative models, because it is often unclear how they arrive at their output. The need for transparency and validation increases with the potential impact of the decision informed by the AI(-supported) information. Requirements for using an AI model in evidence synthesis may, therefore, differ depending on its application. AI models in scoping reviews require different safeguards than AI used to synthesise information that is meant to directly inform health decisions involving people’s lives.
For example, using a language model to help produce synonyms and terminology for additional search terms is very helpful to gain a comprehensive search strategy, provided that human experts decide on the value of adding suggested terms. In contrast, allowing AI tools to develop and execute an entire search strategy with little or no human involvement remains challenging, as the process is often not fully reproducible or transparent. Asking generative AI for relevant articles can even provide you with citations of research that does not exist, also referred to as hallucinated citations.
In the last few years, AI tools for (semi-)automated screening have received significant interest. There is no wonder, as reference screening takes up a lot of time and effort for reviewers. We sometimes have thousands of references for a single review. These AI screening tools usually ask reviewers to provide them with a set of eligible and ineligible references and then prioritise the other references. There are, however, different ways to then use this prioritised list. You can screen all references, which will not reduce the screening load. Or you can stop screening after a certain criterium. However, setting these criteria, also known as stopping rules, will introduce the potential for missing some relevant studies. How many relevant studies will be missed differs per review. It depends on how strict these rules are and the relative number of relevant records within the retrieved references. Missing a few relevant studies might be acceptable, if the missed studies are a random set. Yet, in practice, they often are not, because priority screening tends to surface the most easily identifiable studies first, so the studies that are missed can be systematically different. To date, however, there is no consensus on stopping rules and when and how they should be used. In our experience, defining good stopping rules often costs more time than simply screening all the references. For these reasons, we refrain from (semi-)automated screening within our review processes that directly inform health decisions.
An example of a machine learning tool that we do use within our evidence synthesis projects is the ‘Cochrane RCT-classifier’. This tool scans retrieved references and can filter out references that are not randomised controlled trials (RCTs). It has been trained on high-quality data, reduces the screening load (even on top of using the Cochrane RCT search filter) and has a low risk of missing relevant studies. Because its performance has been formally validated and is reproducible across reviews, we are comfortable letting it filter references automatically, even though we would not yet do the same for full screening. Unfortunately, no such tools yet exist for other study designs.
We, at Cochrane Netherlands, are advocates for making use of tools that improve the efficiency and quality of evidence syntheses. For most tools and technology, human involvement is key to their safe and good use. When an AI model makes decisions autonomously, the methods should still be reproducible and transparent, and they should not introduce bias. With regards to screening, we most often perform our reviews in duplicate to ensure quality and consistency, but this also doubles our workload. While we believe that human supervision will always be needed, we can see a future where an AI model could assist in screening and thereby reduce our workload per review. Ideally, these screening tools would skip the titles and abstracts and base their judgement on the full texts of relevant studies. However, this requires all scientific literature to be openly accessible, which is a topic for another discussion.
Key messages
- The risks and rewards of using AI tools within evidence syntheses should be carefully examined and considered before their actual application
- Some level of human involvement is key to the safe and good use of AI models within evidence syntheses
Acknowledgements
The authors thank René Spijker for the discussions that inspired this piece.
References
Affengruber, L., van der Maten, M. M., Spiero, I., Nussbaumer-Streit, B., Mahmić-Kaknjo, M., Ellen, M. E., Goossen, K., Kantorova, L., Hooft, L., Riva, N., Poulentzas, G., Lalagkas, P. N., Silva, A. G., Sassano, M., Sfetcu, R., Marqués, M. E., Friessova, T., Baladia, E., Pezzullo, A. M., Martniez, P., Gartlehner, G., & Spijker, R. (2024). An exploration of available methods and tools to improve the efficiency of systematic review production: A scoping review. BMC Medical Research Methodology, 24(1), 210. https://doi.org/10.1186/s12874-024-02320-4
Nature. (2025, September 8). Can researchers stop AI making up citations? https://www.nature.com/articles/d41586-025-02853-8
Repke, T., Tinsdeall, F., Danilenko, D., Graziosi, S., Müller-Hansen, F., Schmidt, L., Thomas J., van Valkenhoef, G. (2026). Don't stop me now, 'cause I'm having a good time screening: Evaluation of stopping methods for safe use of priority screening in systematic reviews. Cochrane Evidence Synthesis Methods, 4(1). https://doi.org/10.1002/cesm.70068
The Systematic Review Toolbox. (2026). https://systematicreviewtools.com/
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021). Machine learning reduced workload with minimal risk of missing studies: Development and evaluation of a randomized controlled trial classifier for Cochrane Reviews. Journal of Clinical Epidemiology, 133, 140–151. https://doi.org/10.1016/j.jclinepi.2020.11.003
To link to this article - DOI: https://doi.org/10.70253/MDBM1712
Disclaimer
The views expressed in this World EBHC Day Blog, as well as any errors or omissions, are the sole responsibility of the author and do not represent the views of the World EBHC Day Steering Committee, Official Partners or Sponsors; nor does it imply endorsement by the aforementioned parties.
