Three Moments That Made Me Say: Wait…
Author: Dr Nonglak Chaloumsuk1,2
1. Faculty of Nursing, Chiang Mai University
2. JBI Thailand Centre for Evidence-Based Health Care
The first made me laugh.
The second made me question the answer.
The third made me question what we mean by understanding.
All three happened while teaching and supervising nursing research at the Faculty of Nursing, Chiang Mai University, Thailand. They involved students using artificial intelligence (AI) in different parts of the research process. Each made me realise that as AI becomes more convincing, human judgement becomes even more important in evidence-based healthcare.
Is it real?
The first moment happened while I was reading a student’s systematic review proposal on psychosocial interventions for postpartum depression among adolescent mothers. I was the major advisor, and another faculty member was the co-advisor. Both of us had published research related to postpartum depression.
We noticed something rather funny.
Our own names appeared in the student’s writing as authors of studies that supposedly supported particular claims. There was just one problem. Neither of us had ever conducted those studies.
My co-advisor looked at me and laughed.
‘Apparently, we’ve already published studies we haven’t even done yet!’
We both laughed. The citations sounded entirely plausible because both of us actually worked in the field. AI had connected the right names with the right general topic but with research that did not exist.
We detected the error only because the people being falsely cited happened to be the people reading the proposal.
Then came the less amusing question: What if we had not been the authors whose names appeared there? What if the reviewer did not know the literature well enough to recognise that something was wrong?
A fluent sentence, a familiar author’s name and a citation that looks academic can easily create an impression of credibility. But plausible information is not necessarily trustworthy evidence.
For a systematic review, that distinction matters greatly. A fabricated citation in the background section may be embarrassing. But if unverified information enters study identification, data extraction, synthesis or interpretation, the consequences can become much more serious. (For methodological guidance, see the JBI Manual for Evidence Synthesis and PRISMA 2020.)
A review could appear methodologically sound while being influenced by information that was never true. And that led me to a troubling irony. We could follow the language of evidence-based healthcare, produce recommendations that appear to be based on evidence and still move healthcare in the wrong direction if the evidence we started with was not grounded in reality.
Ultimately, the consequences would not fall on AI. They would fall on patients and the public.
The experience made me reconsider what we need to teach students in this rapidly changing environment. Access to information is no longer enough. They also need the ability to distinguish plausible information from trustworthy evidence.
So, rather than telling students, ‘do not use AI’, perhaps the more important lesson is: never outsource verification and judgement to AI.
Is it right, or simply agreeable?
My second ‘wait’ moment came during research supervision for a student’s study, A Causal Model of Postpartum Depression among Thai Mothers.
The student had completed the analysis in Mplus, interpreted the output and brought the results to me for review. As part of checking the analysis together, I also asked AI to examine the output and help interpret the results.
At first, its response sounded convincing. AI worked through the output and proceeded to discuss the model and its findings in a way that appeared methodologically reasonable.
Then, one value caught my attention. The residual variance for one of the observed variables was negative.
I stopped there. A negative residual variance is a warning sign that something in the model needs closer examination. Yet AI had passed over it and continued with the interpretation as though the model could be interpreted in the usual way.
So, I challenged the answer.
AI changed course immediately. It agreed that the negative residual variance was problematic and explained that such an estimate can arise during iterative estimation and may indicate an improper solution requiring further investigation.
The revised explanation was more appropriate, but it appeared only after I had noticed the problem and challenged the first response.
What if I had not noticed that single line in the output?
The original interpretation was fluent, confident and plausible enough that someone without sufficient methodological knowledge might simply have accepted it and moved on.
That left me with a simple question: Was the answer correct, or was it simply agreeable?
Scientific reasoning should change when new information or stronger arguments are introduced. The difficulty comes when the user cannot independently judge whether the revised answer is methodologically stronger than the original one.
AI can present different analytical approaches with remarkable fluency and confidence. Methodological knowledge is what allows the researcher to ask whether the assumptions are appropriate, whether the data support the proposed analysis and whether another explanation should be considered.
This experience changed how I think about competence in using AI for research. Responsible use requires more than knowing how to formulate a good prompt. Researchers need enough methodological understanding to recognise when an answer deserves to be challenged.
AI may help us reach an answer more quickly. Judging whether that answer is appropriate remains a human responsibility.
Does it mean what we think it means?
The third moment happened in a class on qualitative data collection and analysis, where students were practising thematic analysis through a case study.
One group was exploring the experiences of first-year international PhD students.
During the presentation of their analysis, one student raised a question: ‘Could we give the interview transcript to AI and ask it to analyse the data?’
I did not answer yes or no. Instead, I suggested that we try it and see what happened.
The students uploaded a de-identified transcript and asked AI to analyse it using thematic analysis. We then compared the AI-generated interpretation with the analysis the students had developed through their own engagement with the interviews.
AI worked quickly. It organised the text, identified recurring ideas and generated polished, coherent themes.
One participant had described receiving a heavily commented file back from a supervisor. The experience was emotionally overwhelming: the participant cried in front of the computer and began questioning whether they were capable of continuing their doctoral study.
AI interpreted this account through the theme: Imposter Syndrome and Academic Attrition Risk.
It sounded plausible. It also sounded impressively academic.
The students who had conducted the interviews and immersed themselves in the full accounts saw something more contextual. Their interpretation captured a moment in which confidence had been crushed, leaving a profound sense of incompetence and powerlessness shaped by the emotional weight of academic feedback and the student–supervisor relationship.
The words were the same, but the interpretation was not. That difference stopped us.
What exactly had AI understood? And what had the researchers understood through listening to the participant, following the whole account and repeatedly returning to the data?
The deeper question became: What does it mean to understand qualitative data?
Meaning in qualitative research does not sit in individual words alone. Researchers interpret participants’ experiences in relation to the whole account, the phenomenon being studied and their sustained engagement with the data. A sentence may take on a different meaning when it is considered alongside what came before it, what followed it and what the participant’s broader experience revealed. (For a methodological discussion of reflexive interpretation in thematic analysis, see Braun and Clarke.)
AI had access to the text. The researchers had access to the text together with an accumulated understanding of its context.
That distinction matters.
AI can still play a useful role in qualitative analysis. It can help organise data, explore patterns and prompt researchers to consider interpretations they may not initially have considered. Researchers, however, remain responsible for deciding whether an interpretation is supported by the dataset and genuinely represents participants’ experiences.
When research is about human experience, keeping people at the centre requires thoughtful, reflexive and context-sensitive human interpretation.
Three questions and three take-home messages
These three experiences taught me that AI can sound convincing in very different ways.
It can make something that does not exist look real. It can make more than one analytical choice sound equally reasonable. And it can turn words into coherent themes without necessarily sharing the contextual understanding through which those words gained meaning.
So, when I now encounter AI-supported research, three questions come to mind:
Is it real?
Is it right?
Does it mean what we think it means?
Keeping people at the centre of evidence and AI, therefore, means more than keeping a human ‘in the loop’. It means keeping humans capable of questioning what AI produces, competent enough to verify it and accountable for deciding whether it is trustworthy and appropriate. This emphasis on human accountability is reflected in the World EBHC Day 2026 campaign.
Perhaps AI will make many parts of research faster. But speed does not reduce our responsibility for evidence. In fact, it may make that responsibility even greater.
References
Aromataris, E., Lockwood, C., Porritt, K., Pilla, B., & Jordan, Z. (Eds.) (2024). JBI manual for evidence synthesis. https://doi.org/10.46658/JBIMES-24-01
Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.1769238
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
To link to this article - DOI: https://doi.org/10.70253/WZNN5070
Disclaimer
The views expressed in this World EBHC Day Blog, as well as any errors or omissions, are the sole responsibility of the author and do not represent the views of the World EBHC Day Steering Committee, Official Partners or Sponsors; nor does it imply endorsement by the aforementioned parties.
