In early 2018, a mid-sized neuroscience lab at a public research university posted a preprint describing a new behavioral effect in mice. The finding was modest but clean: a specific genetic manipulation seemed to sharpen performance in a novel object recognition task. The lab had been working on the project for nearly two years, and the preprint represented the culmination of that effort. Within weeks, it drew attention from a handful of groups working on similar circuits. The lab’s principal investigator, a careful experimentalist with a reputation for thorough controls, began drafting a grant application to fund a larger replication study, one that would use a bigger cohort and independent blind scoring. The application leaned heavily on the preprint as proof of feasibility. But the grant did not go through. Reviewers, while praising the novelty, flagged serious concerns about the behavioral assay design, the lack of blinding details, and an opaque sample-size calculation. The funding agency, a private foundation that had funded the lab before, declined to move forward. The second cohort, planned at forty mice, never made it to the animal facility. The story is not a scandal. It is a quiet example of how the scientific enterprise now depends on preprints, and how a single methodological weakness can stall a project before it starts.

A Second Cohort That Never Ran

The grant application was submitted in the summer of 2018. The lab proposed to replicate the effect in a cohort of forty mice, roughly double the size of the original sample. The budget was modest, around US$ 80,000, covering animal costs, behavioral rig calibration, and a graduate student’s stipend for six months. The application’s central argument was that the preprint demonstrated the effect was real and that a larger, pre-registered replication would solidify it.

The reviewers were not hostile, but they were skeptical. One wrote that the novel object recognition task, while standard, is notoriously sensitive to subtle environmental factors. The preprint did not state whether the experimenter was blinded to genotype during testing. Another reviewer noted that the sample size calculation appeared to assume a much larger effect size than the preprint’s own data suggested. The lab had used a common rule of thumb, but the preprint’s confidence intervals were wide enough to make the effect statistically fragile.

The funding agency, a private foundation with a strong interest in rigor, asked for a revised protocol with explicit blinding and a power analysis based on the preprint’s actual variance. The lab responded with a revised plan, but the process dragged on. By the time the agency made its final decision, the lab had already spent several thousand dollars on pilot experiments and planning. The rejection came in early 2019. The principal investigator told me, in a conversation that summer, that the hardest part was not the lost money but the lost time. The graduate student who had designed the assay had graduated and moved on.

The second cohort never ran. The lab pivoted to a different project, one that did not require as much behavioral work. The preprint remains online, cited occasionally, but the replication was never attempted. The effect, if real, remains unconfirmed. The lab’s experience is not unique. In the years since, several funding agencies have adopted policies that treat preprints as legitimate evidence of feasibility, but they have also become more attentive to the methodological details that preprints often omit.

How Preprints Shape Funding Decisions

Preprints have become a fixture of the biomedical research landscape. As of late 2024, servers like bioRxiv host hundreds of thousands of papers, many of them in neuroscience. They allow researchers to stake claims quickly, share results with colleagues, and solicit feedback before formal peer review. But they also enter the funding ecosystem. Grant reviewers, pressed for time, often read preprints as a signal that a project is feasible, that the methods work, that the effect is real.

This is a departure from the older model, where funding decisions rested on peer-reviewed publications. The shift has been gradual but consequential. A 2021 survey of NIH reviewers found that a majority had considered preprints in their evaluations, and many said they treated them as roughly equivalent to published papers for the purpose of assessing technical feasibility. The problem is that preprints have not undergone the same scrutiny. A preprint may contain a promising result, but it may also lack the details that a careful reviewer needs to judge its reliability.

The lab’s case illustrates the double-edged nature of this trust. The preprint was enough to get the grant application to the review stage, but not enough to survive it. The reviewers wanted more than the preprint offered. They wanted evidence of blinding, a power calculation that matched the data, and a clear account of the mouse strain and housing conditions. None of this was in the preprint, and the lab had not kept the raw data in a form that could be quickly shared.

Some argue that this is exactly how the system should work: preprints open the door, and rigorous review closes it when necessary. But the cost falls on the lab. A grant cycle is a year or more. A failed application can set a project back by that much. The preprint, which was meant to accelerate progress, ended up slowing it down.

The Behavioral Assay That Tripped Them Up

The assay at the center of the story is the novel object recognition task, a staple of mouse memory research. The animal is placed in an arena with two identical objects, allowed to explore, then returned after a delay with one object replaced by a novel one. Mice with intact memory spend more time investigating the novel object. The task is simple, but its simplicity is deceptive.

Reviewers flagged several issues in the preprint. The first was blinding. The experimenter who scored the videos knew which mice were in which group. That is a common oversight, but it matters, because scoring involves subjective judgment about what counts as exploration. The second was sample size. The preprint reported a significant effect with eight mice per group, but the effect size implied a much larger cohort would be needed to achieve conventional power. The third was batch effects. The mice were tested in two batches, weeks apart, and the preprint did not report whether the batches differed.

These are not fatal flaws in themselves. Many published papers share the same characteristics. But in the context of a replication grant, they became deal-breakers. The foundation’s program officer told the lab that the reviewers did not doubt the effect was real, but they could not confidently estimate the resources needed to replicate it. The lab had proposed forty mice, but the reviewers suspected that might be too few, given the preprint’s variability.

The lab’s PI defended the design, noting that the task is widely used and that the effect had been seen in two independent cohorts. But the damage was done. The grant was declined, and the lab moved on. The episode highlights a tension in behavioral neuroscience: the field relies on tasks that are easy to run but hard to standardize, and preprints often fail to report the details that would allow others to judge them.

Costs of a Replication Attempt

Replication is not cheap. The lab’s proposed cohort of forty mice would have cost roughly US$ 8–12 per animal for the mice themselves, but that is a small fraction of the total. Housing and care add several dollars per day per mouse, and a behavioral study that runs for weeks can accumulate thousands in per-diem charges. The behavioral rig itself, a set of arenas with video tracking, requires calibration and maintenance.

The largest cost is personnel. A graduate student or postdoc spends months on a replication study, from breeding to testing to scoring. The lab’s budget allocated about US$ 30,000 for the student’s stipend and benefits. That is a conservative estimate; some labs spend more. The foundation’s reviewers knew this, and they wanted to be sure the money would not be wasted on an underpowered study.

There are also opportunity costs. While the replication was pending, the lab could have been pursuing new findings. The PI told me that the pivot to a different project was not a total loss, but it meant abandoning a line of work that had already consumed two years. The sunk investment, in time and reagents, was substantial.

Some funders have tried to address this by creating dedicated replication grants, such as the behavioral economics replication funding that emerged from an earlier field-wide crisis. But these programs are rare, and they often carry less prestige than grants for novel work. The lab’s experience suggests that the cost of replication is not just financial; it is also reputational. A failed replication can be seen as a failure of the original lab, even when the original finding was sound.

Incentives That Push Against Replication

Why did the lab not simply run the replication with its own funds? The answer lies in the incentive structure of academic science. Novel findings attract citations, press coverage, and future grants. Replications, especially of other people’s work, offer fewer rewards. A 2022 analysis of neuroscience publications found that less than 5% of papers were explicit replications.

Funding agencies rarely mandate replication. They want to fund projects that will break new ground. The foundation that rejected the lab’s grant is not unusual; its mission is to support innovative research. Replication, by definition, is not innovative. It is a check on prior work. The lab’s PI acknowledged this tension. He said that if he had submitted a proposal to replicate someone else’s preprint, he would have faced an even harder time.

Career pressure compounds the problem. A graduate student who spends years on a replication may have little to show for it in terms of first-author publications. The lab’s graduate student, who had designed the assay, had already moved on to a postdoc position. The replication would have been a side project, not a career-defining one.

There are counterarguments. Some researchers argue that replication is more valuable than novelty, and that the field would be better off if more labs did it. But the incentives are stacked the other way. The lab’s decision to pivot, rather than fight for a replication grant, was rational. It is also a loss for the scientific community, because the original finding remains unverified.

What Would Make Preprints Safer to Trust

The lab’s experience points to concrete reforms. Preprint servers could require authors to complete a checklist of methodological details, such as blinding, sample size justification, and data availability. Some servers already have such features, but they are not enforced. A preprint without these details could be flagged, not as invalid, but as incomplete.

Funding agencies could adopt a similar approach. Reviewers could be provided with a checklist that prompts them to look for specific indicators of rigor. The single-atom imaging paper, for example, benefited from open data and analysis scripts. If preprints included those by default, reviewers would have more to work with.

Registered replication reports, where the protocol is peer-reviewed before data collection, are a promising model. They have been used in psychology and are slowly spreading to neuroscience. A lab could submit a replication plan, get feedback, and then run the study with a commitment to publish regardless of outcome. This would reduce the pressure to produce a positive result and would make the replication more credible.

But these reforms face resistance. Scientists value the speed and informality of preprints. Adding requirements could slow the process. Some argue that the onus should be on reviewers to ask for details, not on authors to provide them. The tension is real. The lab’s PI, for his part, said he would support a system where preprints are clearly labeled as unvalidated, with a warning to readers that the methodology has not been peer-reviewed.

The Real Cost of a Missed Replication

When a replication never happens, the field is left with an unverified claim. Other labs may build on it, wasting time and resources. If the effect is real, the field loses a chance to confirm it. If it is not real, the field may chase a false lead for years. The cost is diffuse, but it is real.

In this case, the effect was not dramatic enough to attract much follow-up. The preprint has been cited a handful of times, mostly in passing. But the pattern is common. A 2020 analysis of preclinical studies found that less than half could be replicated by independent labs. The consequences extend beyond individual projects. Trust in the literature erodes, and funders become more conservative, which can slow the pace of discovery.

The infrastructure for mouse research, including the behavioral rigs and animal facilities, is expensive to build and maintain. When a replication is not run, that infrastructure sits idle. The lab’s behavioral rig was eventually used for a different project, but the original question was left unanswered. The 1965 helium-neon laser logbook is a reminder that data, once lost, can be recovered, but a replication that never happens cannot be undone.

The slow shift toward a validation-first culture is underway, but it is not complete. The lab’s experience is a case study in how a single methodological weakness can cost a project its funding. The second cohort never ran, but the lesson is clear: preprints are powerful, and with power comes responsibility. The scientific community is still learning how to handle that responsibility. In the meantime, labs like this one will continue to weigh the risks and benefits of sharing their work before it is fully vetted.