On a tangent, this idea of reputation keeps on coming up in this whole discussion and I am burdened by it in a way I don't fully understand. The way people have talked, if this LK-99 doesn't work out, then it is almost as if those who published this did something _morally_ wrong. Well, morally wrong is not quite true, but the way people talk about it tanking their reputation it feels like such a strong statement. Is there some way we can focus on the science and not get bogged down in the very human reputational part of this whole thing? It's almost as if a good chunk of the scientific community don't care about the benefits the science brings but the reputational benefits.
Of course they care about science itself, but there's a limit to what risks they'll be willing to take when it affects them personally.
For people with a relatively low reputation (or no reputation, i.e. unknown), taking a risk is not a bad move. They have less opportunity, and there's a chance the risk might pay off and boost their reputation.
For people whose reputation is already good, the risk is less worth it. They don't stand to gain as much, and they could lose a lot. So they're less likely to do it.
During the 'golden age' of science, the time of the Royal Society the fields weren't specialized at all and the publication mechanism was scientists sending each other interesting stuff by post. At that time there was no meta analysis at all and there was so much low hanging fruit that the 'gentleman scientist' could make big breakthroughs in their home laboratories. But as that low hanging fruit decreased the educational paths required before being able to do meaningful science became longer and longer, then specialization set in and the costs of doing science went up. That's how we arrived at grants used to fund science.
A lot of these gentleman scientists were independently wealthy aristocrats that didn't need hand-outs. The fact that we don't to a meaningful extent have that sort of leisure class anymore is arguably a much bigger reason we need grant funded science these days.
It could be argued there a bit of a replication of the pattern in the space race between Musk and Bezos, but they're missing the sort of well education the aristocrats of yore would have had[1]. They employ a lot of people to do the actual dirty work, but that's not really a big difference from back then either.
This sheds some light on it to me. I guess what partly surprises me is that people seem to care more about reputation than just a means for improving the signal to noise ratio in papers or as a estimate on what will give you your biggest bang for your buck.
The other issue I see come up is the idea that if there is no signal to noise filter, then a scientist might "waste their time," either reading the paper or trying to replicate. But to me, it sounds a little bit like trying to avoid actually doing science. And peer reviewed papers don't imply excellent quality either. You should evaluate papers on their merits. It is your job, as a scientist, to evaluate the most productive approaches based on the merits of the science being done, not based on reputation.
This means it's really easy to just claim something, that will be really hard for others to verify.
And wrong claims are incredibly common. It's easy to delude yourself through all sorts of biases or good old sloppy work.
That's why, when scientists talk to each other, they need to know that the other person is a serious scientist and won't pollute their mind with nonsense.
If you develop a reputation for making baseless claims, people will stop including your claims in their own thoughts.
But there are costs to this. There are big gaps between what people discuss with colleagues and what gets published, and the is no forum to publish partial or negative results, except maybe conferences. Ideally published papers stay at a very high bar, but there are other forums to publicly share work in progress. In a way Twitter is becoming this.
