Relevance Score: 98/100

Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models

Author: Juraj Vladika
AI ResearchLitigation
View Original Research on arXiv →

Original Paper: Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models

Authors: Juraj Vladika, Mahdi Dhaini, Florian Matthes Technical University of Munich, Germany School of Computation, Information and Technology Department of Computer Science {juraj.vladika, mahdi.dhaini, matthes}@tum.de

New Evidence of LLM Memorization Challenges the AI Transformative Use Defense

Original Paper: Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models Authors: Juraj Vladika, Mahdi Dhaini, Florian Matthes

Executive Summary

A new study from the Technical University of Munich provides a powerful evidentiary basis for challenging the “transformative use” defense in AI copyright cases. The research demonstrates that prominent large language models (LLMs) systematically memorize and reproduce specific, outdated facts from their training data, proving direct, non-transformative copying.

What the Research Shows

The authors developed two novel question-answering datasets to test the currency of medical knowledge in eight leading LLMs. The key dataset, MedChangeQA, contains 512 questions where the established medical consensus has changed over time. This allowed the researchers to precisely measure whether an LLM was providing the current, correct answer or regurgitating an outdated one learned from its static pre-training data.

The results were conclusive and consistent: every model tested demonstrated a significant tendency to rely on and reproduce outdated medical knowledge. The study establishes a clear, traceable link between specific information in the training corpus and the model’s output. This is not a matter of inference or emergent knowledge; it is an act of direct recall. The LLMs are not “learning” in the human sense but are instead engaging in large-scale memorization and regurgitation of the data they were fed, including specific, identifiable, and now-obsolete facts.

Why This Matters for Your Case

  • Directly Counters the Transformative Use Defense: The core of the fair use defense for AI developers rests on the argument that their models transform ingested data into something new. This research provides concrete evidence to the contrary. By showing that LLMs reproduce specific, verbatim facts—even harmful, outdated ones—you can argue the use is merely reproductive and derivative, not transformative, directly attacking Factor 1 of the fair use analysis.

  • Establishes a Causal Link Between Training Data and Output: This paper moves the argument from a theoretical claim to a provable one. It provides a methodology for demonstrating that a specific output is the direct result of ingesting a specific piece of copyrighted material. This is crucial for proving the element of copying and for defeating arguments that the model’s output is an original creation.

  • Provides a Template for Evidentiary Discovery: The study’s design—using questions with known-obsolete answers—creates a blueprint for your own discovery and expert analysis. You can design similar tests using your client’s copyrighted material to demonstrate that a defendant’s model has memorized and can reproduce protected content on demand, thereby proving infringement.

Litigation Strategy

  • Targeted Discovery Requests: Use this study as a basis to draft discovery requests demanding information on training datasets known to contain outdated or superseded information. Compel the production of model outputs in response to prompts designed to elicit specific, memorized facts from your client’s work, mirroring the paper’s methodology.

  • Leverage Expert Witness Testimony: Retain a data scientist or AI researcher as an expert witness. They can replicate the study’s findings on the defendant’s model, using your client’s copyrighted material as the source data. Their testimony can establish that the model is not “creating” but is performing an act of high-fidelity, verbatim recall, which is functionally equivalent to direct copying.

  • Frame the Argument for the Court: In motions and at trial, use the paper’s findings to frame the LLM not as an intelligent author but as a flawed, large-scale database. The analogy of an LLM providing dangerous, outdated medical advice is a powerful and intuitive way to explain to a judge or jury why non-transformative memorization is not a benign technical process but a core aspect of infringement.

Key Takeaway

This research is more than an academic paper; it is a tactical tool for copyright litigators. It provides a clear, scientific framework for proving that LLMs engage in direct, non-transformative reproduction of their training data, directly undermining the central legal defense used by AI companies.