When AI Meets Systematic Reviews: What My Students Taught Me
Authors: Dr Hanieh Salehi-Pourmehr1,2, Dr Neda Kabiri1,2, Dr Sakineh Hajebrahimi1,2
1. Research center for Evidence-based Medicine
2. Tabriz University of Medical Sciences
As a teacher and researcher in evidence-based healthcare, I have conducted, supervised and taught systematic review methods for many years. Increasingly, my students ask me whether artificial intelligence (AI) can assist them in conducting a systematic review.
My answer is yes – but only if we are very clear about what ‘assist’ means.
A recent experience with a PhD student in neuroimmunology made this distinction particularly clear. The student had conducted a systematic review on AI in emergency stroke neuroradiology. The review ultimately included 59 studies, and the student had used the Pro version of an AI tool across much of the review process. Remarkably, the work had been completed in approximately two days.
When I first saw the review, I was impressed. The manuscript was well organised and the tables were attractive. The search strategy looked systematic, the data extraction tables were neatly presented and the Summary of Findings table appeared professional.
Then, I started checking the evidence behind the presentation.
A search strategy can look systematic without being comprehensive
The first problem was the search strategy.
The AI-generated strategy was not sufficiently aligned with the concepts expressed in the review title and question. Although it looked technically sophisticated, I was not confident that it could identify the relevant evidence comprehensively.
I could not reliably determine how many eligible studies had been missed by the original search, so I do not want to assign a number retrospectively. Instead, we took a practical step: we asked an experienced librarian to develop and conduct a revised search.
For me, this was an important lesson. A search strategy is not good simply because it contains Boolean operators, multiple databases and a long list of search terms. Its value depends on whether it represents the concepts of the question adequately and retrieves the relevant evidence.
Research evaluating AI-assisted screening has shown that performance can vary across models, review topics and eligibility criteria. Accordingly, Delgado-Chaves et al. found that the overlap between large language model (LLM)-selected and human-selected records varied across three systematic reviews. A separate evaluation of a human–AI evidence-synthesis pipeline reported promising results for searching, screening and extraction, but these findings related to a specific system and benchmark rather than proving that AI can replace human review. According to Wang et al., AI may reduce the workload, but it should not be treated as a standalone replacement for human review.
Data extraction: The table looked complete, but it was not
The second problem emerged during data extraction.
The extraction was incomplete, particularly when information was not clearly presented in the abstract and required examination of the full-text article. Important information may be distributed across tables, figures, supplementary material and different sections of a paper. In this case, some of that information was not captured.
What made this particularly concerning was the appearance of the final table. It looked complete.
This is one of the risks I now associate most strongly with AI-assisted evidence synthesis: the output can look more reliable than it actually is.
Note that a recent evaluation by Wang et al. of a human–AI evidence synthesis pipeline reported improved extraction accuracy in its benchmark and pilot workflow, but those results relate to the specific system and study conditions employed. Hill et al., meanwhile, describe a practical method for using Bing’s AI-powered search engine as a second reviewer to check data items extracted by a human reviewer; they caution that this should not replace established independent double extraction without further evaluation. Together, these sources support using AI as an assistive and verifiable tool, not assuming that its output is complete.
AI can, therefore, be useful for organising and supporting extraction, but the reviewer still needs to verify the extracted information against the original articles.
Risk of bias cannot simply be delegated
There was another important lesson to be learned from this review.
The student had not asked the AI tool to perform a risk-of-bias assessment, so I cannot say whether it would have completed this task well or poorly. Instead, what I observed was that the student was moving towards an assessment of certainty without having built the full methodological foundation that should precede it.
For me, this reinforced a basic principle of systematic review training: we need to understand the methodological purpose of each stage of the review. Using AI does not take away that need.
GRADE: The most important lesson
The most instructive issue emerged when I examined the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) assessment.
I found that the evidence base consisted of observational studies. And the Summary of Findings table itself looked professional, but the methodological rationale underlying the assessment of certainty was not sufficiently clear.
This is where I think AI can be particularly misleading. GRADE is not a scoring system in which an algorithm simply counts methodological problems and produces a final label. Instead, it requires structured judgement about the body of evidence.
For a conventional GRADE assessment of intervention effects, evidence from randomised trials starts at high certainty, whereas evidence from non-randomised studies of interventions starts at low certainty. Certainty may be rated down for risk of bias, inconsistency, indirectness, imprecision and publication bias; under specified conditions, observational evidence may be rated up, for example, when there is a large effect (Guyatt et al., 2011, 2025).
The approach must be matched to the study design and applied transparently. And the Core GRADE guidance sets out explicit considerations for risk of bias and rating up certainty, while the earlier GRADE guidance on inconsistency explains how variability in relative treatment effects should inform our judgements (Guyatt et al., 2011, 2025).
The problem I encountered was, therefore, not simply that the final table looked wrong. It was that the reasoning behind the assessment of certainty was not explicit.
That distinction matters.
A beautifully formatted Summary of Findings table does not demonstrate that GRADE has been applied correctly.
What I would do differently as a supervisor
This experience has changed how I approach AI-assisted systematic reviews with students.
Going forwards, I will not tell students to avoid AI. Instead, I will ask them to use it under supervision and to document what it is being used for.
Before accepting an AI-generated search strategy, I will test it against known relevant studies and examine whether its concepts actually reflect the review question. If the search is not sufficiently reliable, I will involve an information specialist or librarian.
For data extraction, I will require students to check the AI output against the full text, rather than assuming that a complete-looking table represents complete extraction.
Then, for critical appraisal, I will first make sure that the student understands the appropriate appraisal framework and can justify their judgements independently.
And before using AI to assist with GRADE, I will ask a simple question:
Can you explain why this body of evidence starts at this level of certainty, and can you defend every decision to rate it down or up?
If the answer is no, the problem is not the AI. The problem is that the student’s methodological understanding is not yet sufficient.
What this experience taught me
The most important lesson for me was that AI can make an incomplete or insufficiently reasoned systematic review look remarkably complete.
That creates a particular risk for novice researchers. An experienced reviewer may notice that a search strategy is conceptually weak, that an extracted value does not fit the original paper or that a GRADE judgement requires further justification. Yet a novice may see only a polished table and assume that the difficult methodological work has already been done.
AI can help with repetitive tasks, organise information and support screening and extraction. In this regard, Ge et al. describe potential benefits of AI in evidence synthesis (while emphasising the need for human oversight). Moreover, evaluations by Delgado-Chaves et al. and Wang et al. of LLM screening and a human–AI evidence synthesis pipeline produced promising but task- and system-specific results. Hill et al., meanwhile, provide a worked method for AI-assisted extraction verification, though they caution against treating it as a replacement for independent extraction.
But speed should not be confused with methodological quality.
My experience with this 59-study review can be distilled into a simple rule for supervising AI-assisted systematic reviews: Make sure students use AI to assist the process, but never outsource the reasoning.
The researcher remains responsible for knowing whether the search was comprehensive, whether the extracted information is correct, whether the methodological judgements are defensible and whether the certainty of the evidence has been appropriately assessed.
For me, that is the real challenge of AI in evidence synthesis: not learning how to instruct AI to produce a systematic review, but learning how to recognise when the review it produces needs to be questioned.
Key messages
1. A polished AI-assisted systematic review can still have an incomplete evidence base or incomplete data extraction
2. AI-generated outputs should be verified against the original evidence, particularly for searching, full-text extraction and methodological judgements
3. GRADE requires methodological reasoning and cannot be reduced to an automated scoring exercise
References
Delgado-Chaves, F. M., Jennings, M. J., & Atalaia, A. (2025). Transforming literature screening: The emerging role of large language models in systematic reviews. Proceedings of the National Academy of Sciences, 122(2), e2411962122. https://doi.org/10.1073/pnas.2411962122
Ge, L., Agrawal, R., Singer, M., Kannapiran, P., De Castro Molina, J. A., Teow, K. L., Yap, C. W., & Abisheganaden, J. A. (2024). Leveraging artificial intelligence to enhance systematic reviews in health research: Advanced tools and challenges. Systematic Reviews, 13, 269. https://doi.org/10.1186/s13643-024-02682-2
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., Norris, S., Vist, G., Dahm, P., Shukla, V. K., Higgins, J., Falck-Ytter, Y., & Schünemann, H. J. (2011). GRADE guidelines: Rating the quality of evidence—Inconsistency. Journal of Clinical Epidemiology, 64(12), 1294–1302. https://doi.org/10.1016/j.jclinepi.2011.03.017
Guyatt, G., Wang, Y., Eachempati, P., Iorio, A., Murad, M. H., Hultcrantz, M., Chu, D. K., Florez, I. D., Hemkens, L. G., Agoritsas, T., Yao, L., Vandvik, P. O., Montori, V. M., & Brignardello-Petersen, R. (2025). Core GRADE 4: Rating certainty of evidence—Risk of bias, publication bias, and reasons for rating up certainty. BMJ, 389, e083864. https://doi.org/10.1136/bmj-2024-083864
Hill, J. E., Harris, C., & Clegg, A. (2024). Methods for using Bing’s AI-powered search engine for data extraction for a systematic review. Research Synthesis Methods, 15(2), 347–353. https://doi.org/10.1002/jrsm.1689
Wang, Z., Cao, L., Danek, B., Jin, Q., Lu, Z., & Sun, J. (2025). Accelerating clinical evidence synthesis with large language models. npj Digital Medicine, 8, 509. https://doi.org/10.1038/s41746-025-01840-7
To link to this article - DOI: https://doi.org/10.70253/IODN3854
Disclaimer
The views expressed in this World EBHC Day Blog, as well as any errors or omissions, are the sole responsibility of the author and do not represent the views of the World EBHC Day Steering Committee, Official Partners or Sponsors; nor does it imply endorsement by the aforementioned parties.
