Generative AI Hallucinations Could Mislead Biological Research

Generative AI may assist scientists in finding a promising new medicine, but it could equally persuade them that a biological effect is real when it is not.

Generative AI produces fresh material by learning the recurring characteristics and relationships within existing examples.

Generative AI in biological research

While the technology is most widely recognised for generating text and images, scientists are investigating how it could be used for protein design, cell simulation, completing missing experimental results, and producing synthetic biological data.

Yet these systems can hallucinate.

In biological science, that might involve creating a convincing molecular pattern or conclusion that fails to represent the biology beneath it.

The consequences of this type of mistake could be substantial.

An AI system could reject a drug candidate that might have succeeded, steer scientists towards a treatment that does not work, obscure a real biological effect, or present a disease mechanism that does not exist as a discovery.

Computational biologist Thomas Burger, of Grenoble Alpes University in France, examines this issue across 10 possible applications of generative AI in an Opinion article published in Patterns.

Omics experiments can produce enormous datasets measuring genes, proteins and other molecules. Although AI may help researchers interpret this huge amount of information, minor alterations introduced into data this complex could be hard to spot.

Burger argues that the risks are not equal across all of these uses.

The crucial distinction is whether an AI result is simply an idea to be tested later in a physical experiment, or synthetic data that is used directly as evidence.

"I have never thought about that so far, but I guess it is possible to have hallucinations that lead to genuine discoveries." – computational biologist Thomas Burger

When AI outputs become scientific evidence

Screening prospective drugs or proteins is one of the relatively lower-risk applications. A model might quickly evaluate many candidates before identifying a smaller selection for laboratory tests.

Should the AI make an error, scientists could overlook a candidate that would have worked, or spend time and money studying one that eventually fails. Still, any chosen candidate would need to show its effects in an actual experiment before it could be regarded as a discovery.

The risk rises when AI-generated data starts to stand in for experimental measurements.

Synthetic biological data may be useful for completing missing measurements, safeguarding patient privacy, generating comparison groups, cutting research costs, or reducing the number of animals used in experiments.

However, if an AI system introduces a feature that was never there, scientists might conclude that they had found a biological effect that did not occur. Rather than merely making a wrong prediction about an experiment, the system's invention would then have become part of the evidence behind a scientific claim.

"Most of the time, the problem is not about comparing a hallucination and a genuine biological discovery side by side," Burger told ScienceAlert.

"It is more about real data having been corrupted by hallucination along the course of the complex computational (genAI-aided) workflow that makes it possible to turn raw signals acquired with complex biotechnologies into biologically valid descriptions of molecular mechanisms."

Put another way, AI could modify a signal while handling genuine data in a manner that is not easy to detect. This alteration could shape the scientists' conclusion without producing a distinct, plainly fabricated result.

"If, along the process, some signals are distorted, amplified, or changed in any direction that may lead to different final biological conclusions, the investigator will have trouble noticing it unless they have a deep understanding about how the genAI has worked," Burger said.

AlphaFold 3 and the risk of distorted signals

AlphaFold 3 has provided a real-world example.

In a 2024 paper in Nature, the developers of AlphaFold 3 said that the model can produce "hallucinated structures" in disordered regions of proteins, though low confidence scores can warn researchers about this issue.

AI mistakes will not necessarily make an absent effect seem genuine. Instead, a model could introduce enough distortion into data for scientists to miss a true effect, potentially overlooking evidence that a treatment really works.

Burger had not formerly considered the possibility that an AI hallucination might result in an authentic discovery.

"I have never thought about that so far, but I guess it is possible to have hallucinations that lead to genuine discoveries."

He likened the prospect to unforeseen discoveries produced by mistakes in the laboratory.

"Serendipity has long been acknowledged; whether it originates from genAI hallucination or any other wet-lab mistake should not matter in the end, both from a moral viewpoint and from the expected posterior validation level," Burger said.

What counts is the way scientists handle the output. When it is regarded as an idea for testing, a hallucination might remain no more than an unsuccessful hypothesis. When it is accepted as a real observation, though, a persuasive fabrication may enter the evidence and be confused with biological reality.

Even AI's most compelling proposed result does not become a discovery unless it is independently confirmed through a real experiment.

The article was published in Patterns.

This article was fact-checked by Rebecca Dyer and edited by Rebecca Dyer. While we take pride in our process, we are only human. If you notice an error, please let us know.