In a groundbreaking study published in Nature Human Behaviour, researchers demonstrate that artificial intelligence can predict scientific results better than human experts

In an era where scientists struggle to keep pace with the exponential growth of research publications, artificial intelligence may offer a solution. A landmark study published in Nature Human Behaviour reveals that large language models (LLMs) significantly outperform human experts in predicting the outcomes of neuroscience experiments, potentially revolutionizing how scientific discoveries are made.

The study, led by researchers from University College London, introduces “BrainBench,” a novel benchmark designed to test whether LLMs can forecast experimental results in neuroscience—a field characterized by complex, multi-level research spanning from molecular mechanisms to human behavior.

“Scientific discoveries often hinge on synthesizing decades of research, a task that potentially outstrips human information processing capacities,” the authors write. Their findings suggest that AI systems trained on vast scientific literature can integrate patterns across thousands of studies to make predictions that exceed human capabilities.

Beyond human capabilities: The performance gap

The results are striking: LLMs achieved an average accuracy of 81.4% on the BrainBench test, compared to human neuroscience experts’ 63.4%. This performance gap held true across all neuroscience subfields tested, including behavioral/cognitive, cellular/molecular, systems/circuits, neurobiology of disease, and development/plasticity/repair.

Perhaps most surprising was that smaller models with just 7 billion parameters (like Llama2-7B and Mistral-7B) performed comparably to much larger models, suggesting that raw size isn’t the only factor in an AI’s predictive abilities. Interestingly, the base versions of models outperformed their chat-optimized counterparts, indicating that conversational tuning might actually hinder scientific reasoning.

The researchers took their work a step further by creating BrainGPT, a specialized model fine-tuned on neuroscience literature. This customized model showed even better performance, improving accuracy by an additional 3%.

How BrainBench works

The BrainBench benchmark presents a deceptively simple challenge: given two versions of a neuroscience abstract—one real and one altered to change the results while maintaining coherence—can the test-taker identify which is authentic?

For human participants, the test involved making a choice between these versions and rating their confidence and expertise. For LLMs, the models processed both versions, with the choice determined by which abstract had lower “perplexity”—essentially, which text the model found less surprising based on its training.

This approach represents what the authors call “forward-looking” evaluation, testing the ability to predict novel outcomes rather than simply retrieving known facts—the latter being the focus of most existing AI benchmarks.

Confidence and calibration

A particularly important discovery was that both humans and LLMs showed good “calibration”—when they expressed higher confidence in their predictions, they were indeed more likely to be correct. This characteristic is crucial for scientific applications, as it allows researchers to trust AI predictions when the system expresses high certainty.

The researchers found that LLMs’ superior performance stemmed from their ability to integrate information throughout the abstract, particularly connecting methodological details with likely outcomes. When limited to analyzing only the results section of abstracts in isolation, LLM performance dropped dramatically.

Not just memorization

To address concerns that LLMs might simply be regurgitating memorized information, the researchers conducted extensive tests confirming that the models hadn’t been exposed to the test cases during training. Instead, the evidence suggests that LLMs had learned fundamental patterns underlying neuroscience research, enabling them to generalize to novel situations.

Further supporting this conclusion, the team created a smaller LLM trained from scratch exclusively on published neuroscience literature (excluding any material in BrainBench), which still achieved superhuman performance.

Implications for scientific discovery

The study’s implications extend far beyond neuroscience. As lead author Xiaoliang Luo and colleagues write: “We foresee a future in which LLMs serve as forward-looking generative models of the scientific literature. LLMs can be part of larger systems that assist researchers in determining the best experiment to conduct next.”

This vision suggests a transformation in scientific practice, with AI systems helping to predict experimental outcomes before studies are conducted, potentially accelerating discovery and reducing wasted resources on less promising research directions.

However, the authors acknowledge potential risks: “One risk is that scientists do not pursue studies when their predictions run counter to those of an LLM.” They note that sometimes these contradictions might represent crucial breakthrough opportunities where the existing literature contains gaps or errors.

Beyond neuroscience

While the current study focuses on neuroscience, the researchers emphasize that their approach could be applied to any knowledge-intensive field. The techniques they developed aren’t domain-specific and could potentially transform how research is conducted across scientific disciplines.

The authors advocate for democratizing these AI tools by highlighting the impressive performance of smaller, open-source models that can be run locally, contrasting with proprietary commercial systems that may be less accessible to the broader scientific community.

Human-AI Collaboration

Rather than suggesting AI will replace human scientists, the researchers envision a collaborative future where LLMs assist humans in making discoveries. The complementary nature of human and machine strengths—with LLMs excelling at pattern recognition across vast datasets and humans providing creative insights and explanations—points toward a new paradigm for accelerating scientific progress.

As the authors conclude: “Prediction is very important, but not everything.” Human experts will continue to play crucial roles in framing research questions, designing experiments, and providing the scientific explanations that give meaning to results.

This groundbreaking study demonstrates that we may be entering a new era of scientific discovery—one where human creativity combines with artificial intelligence’s pattern-recognition capabilities to navigate the ever-expanding universe of scientific knowledge more effectively than either could alone.