{"id":874993,"date":"2025-10-02T21:29:06","date_gmt":"2025-10-03T02:29:06","guid":{"rendered":"https:\/\/newsycanuse.com\/index.php\/2025\/10\/02\/ai-models-are-using-material-from-retracted-scientific-papers\/"},"modified":"2025-10-02T21:29:06","modified_gmt":"2025-10-03T02:29:06","slug":"ai-models-are-using-material-from-retracted-scientific-papers","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2025\/10\/02\/ai-models-are-using-material-from-retracted-scientific-papers\/","title":{"rendered":"AI models are using material from retracted scientific papers"},"content":{"rendered":"
\n

Some AI chatbots rely on flawed research from retracted scientific papers to answer questions, according to recent studies. The findings, confirmed by MIT Technology Review<\/em>, raise questions about how reliable AI tools are at evaluating scientific research and could complicate efforts by countries and industries seeking to invest in AI tools for scientists.<\/p>\n

AI search tools and chatbots are already known to fabricate links<\/a> and references. But answers based on the material from actual papers can mislead as well if those papers have been retracted.\u00a0The chatbot is \u201cusing a real paper, real material, to tell you something,\u201d says Weikuan Gu, a medical researcher at the University of Tennessee in Memphis and an author of one of the recent studies<\/a>. But, he says, if people only look at the content of the answer and do not click through to the paper and see that it\u2019s been retracted, that\u2019s really a problem.\u00a0<\/p>\n<\/p><\/div>\n

\n

Gu and his team asked OpenAI\u2019s ChatGPT, running on the GPT-4o model, questions based on information from 21 retracted papers about medical imaging. The chatbot\u2019s answers referenced retracted papers in five cases but advised caution in only three. While it cited non-retracted papers for other questions, the authors note that it may not have recognized the retraction status of the articles. In a study from August<\/a>, a different group of researchers used ChatGPT-4o mini to evaluate the quality of 217 retracted and low-quality papers from different scientific fields; they found that none of the chatbot\u2019s responses mentioned retractions or other concerns. (No similar studies have been released on GPT-5, which came out in August.)<\/p>\n

The public uses AI chatbots to ask for medical advice<\/a> and diagnose health conditions<\/a>. Students and scientists increasingly use<\/a> science-focused AI tools<\/a> to review existing scientific literature and summarize papers. That kind of usage is likely to increase. The US National Science Foundation, for instance, invested $75 million in building AI models for science research this August.<\/p>\n<\/div>\n

\n

\u201cIf [a tool is] facing the general public, then using retraction as a kind of quality indicator is very important,\u201d says Yuanxi Fu, an information science researcher at the University of Illinois Urbana-Champaign. There\u2019s \u201ckind of an agreement that retracted papers have been struck off the record of science,\u201d she says, \u201cand the people who are outside of science\u2014they should be warned that these are retracted papers.\u201d OpenAI did not provide a response to a request for comment about the paper results.<\/p>\n

The problem is not limited to ChatGPT. In June, <\/strong>MIT Technology Review<\/em> tested AI tools specifically advertised for research work, such as Elicit, Ai2 ScholarQA (now part of the Allen Institute for Artificial Intelligence\u2019s Asta tool), Perplexity, and Consensus, using questions based on the 21 retracted papers in Gu\u2019s study. Elicit referenced five of the retracted papers in its answers, while Ai2 ScholarQA referenced 17, Perplexity 11, and Consensus 18\u2014all without noting the retractions.<\/p>\n

Some companies have since made moves to correct the issue. \u201cUntil recently, we didn\u2019t have great retraction data in our search engine,\u201d says Christian Salem, cofounder of Consensus. His company has now started using retraction data from a combination of sources, including publishers and data aggregators, independent web crawling, and Retraction Watch<\/a>, which manually curates and maintains a database<\/a> of retractions. In a test of the same papers in August, Consensus cited only five retracted papers.\u00a0<\/strong><\/p>\n

Elicit told MIT Technology Review<\/em> that it removes retracted papers flagged by the scholarly research catalogue OpenAlex from its database and is \u201cstill working on aggregating sources of retractions.\u201d Ai2 told us that its tool does not automatically detect or remove retracted papers currently. Perplexity said that it \u201c[does] not ever claim to be 100% accurate.\u201d\u00a0<\/p>\n

However, relying on retraction databases may not be enough. Ivan Oransky, the cofounder of Retraction Watch, is careful not to describe it as a comprehensive database, saying that creating one would require more resources than anyone has: \u201cThe reason it\u2019s resource intensive is because someone has to do it all by hand if you want it to be accurate.\u201d<\/p>\n

Further complicating the matter is that publishers don\u2019t share a uniform approach to retraction notices. \u201cWhere things are retracted, they can be marked as such in very different ways,\u201d says Caitlin Bakker from University of Regina, Canada, an expert in research and discovery tools. \u201cCorrection,\u201d \u201cexpression of concern,\u201d \u201cerratum,\u201d and \u201cretracted\u201d are among some labels publishers may add to research papers\u2014and these labels can be added for many reasons, including concerns about the content, methodology, and data or the presence of conflicts of interest.\u00a0<\/p>\n<\/p><\/div>\n

\n

Some researchers distribute their papers on preprint servers, paper repositories, and other websites, causing copies to be scattered around the web. Moreover, the data used to train AI models may not be up to date. If a paper is retracted after the model\u2019s training cutoff date, its responses might not instantaneously reflect what’s going on, says Fu. Most academic search engines don\u2019t do a real-time check against retraction data, so you are at the mercy of how accurate their corpus is, says Aaron Tay, a librarian at Singapore Management University.<\/p>\n

Oransky and other experts advocate making more context available for models to use when creating a response. This could mean publishing information that already exists, like peer reviews commissioned by journals and critiques from the review site PubPeer, alongside the published paper.\u00a0\u00a0<\/p>\n

Many publishers, such as Nature<\/em> and the BMJ<\/em>, publish retraction notices as separate articles linked to the paper, outside paywalls. Fu says companies need to effectively make use of such information, as well as any news articles in a model\u2019s training data that mention a paper\u2019s retraction.\u00a0<\/p>\n

The users and creators of AI tools need to do their due diligence. \u201cWe are at the very, very early stages, and essentially you have to be skeptical,\u201d says Tay.<\/p>\n

Ananya<\/em><\/a> is a freelance science and technology journalist based in Bengaluru, India.<\/em> <\/p>\n<\/div>\n

Read More<\/a>
\n Ananya<\/p>\n","protected":false},"excerpt":{"rendered":"

Some AI chatbots rely on flawed research from retracted scientific papers to answer questions, according to recent studies. The findings, confirmed by MIT Technology Review, raise questions about how reliable AI tools are at evaluating scientific research and could complicate efforts by countries and industries seeking to invest in AI tools for scientists. AI search<\/p>\n","protected":false},"author":1,"featured_media":874994,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2483,46,151],"tags":[],"class_list":{"0":"post-874993","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-models","8":"category-technology","9":"category-using"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/874993","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=874993"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/874993\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/874994"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=874993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=874993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=874993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}