Nutrition science is filled to the brim with embarrassingly low quality research. The modal nutritional epidemiological study is a heavily confounded cross-sectional one with results that don't hold up and don't reflect causal effects, but rather, that certain people opt into having or reporting having certain diets. So if you're going to do nutrition studies, at least do them in the multiverse. The way this works is basically by exploiting disagreement about the proper way to specify a model. As an example, imagine there are two people, Cynthia and Bob. Cynthia thinks that to properly analyze the effect of a vegan diet on cardiovascular health, we just need to control for education, sex, and age and then we can compute a hazard ratio for heart attacks. Bob thinks we need to control for sex, age, and we need to use a different measure of cardiovascular health. So in the multiverse analysis, we run both models and report both results. But not only that, we run every set of analyses in between. Here's a recent empirical example of how this plays out in practice: assessing the impact of red meat consumption on all-cause mortality. Wang et al. used the National Health and Nutrition Examination Survey to check the relationship with every possible combination of these variables: They used outright different models, operationalized red meat consumption in different ways, looked within different groups, and used a lot of different controls, like smoking status, pregnancy, BMI, etc. When you plot the coefficients from all of these more than one thousand different specifications on a curve from the lowest to the highest effect size, it looks like this: The blue lines are significant, negative estimates, and the red lines are significant, positive ones. Overall, there's just not much there. If you want to look at your favored specification, you can check that out in the paper, but because there's so little there, I'm just going to focus on the aggregate result: 4% of specifications had significant results (roughly the 5% expected by chance), the median effect size was a hazard ratio of 0.94 (i.e., a nonsignificant 6% reduction in all-cause mortality), and the average of all specifications' Z-values was not more extreme than would be expected if red meat didn't do anything to all-cause mortality. Looking at every result, it seems like there's nothing worth caring about here. The specifications with significant results looked basically random and marginal, but if you were motivated, you could probably write a paper on them. You might even be able to get it published, and it would look like a typical nutrition epidemiology paper. The meta point here is less about red meat and more about how we ought to do cross-sectional research without causal designs. We cannot and should not get rid of this sort of research. We need to know things are related, so when we do it, we should do it in multiverse analyses that show that our results hold up in reasonable alternative models, as with the example of red meat and death. If you want to read the paper, it's here: jclinepi.com/article/S0895-4…

It's widely known that there's a lot of fraud in cancer research. A new study looked at misspelled cell lines and found that some subsequent studies (mostly by Chinese authors) claimed to use and described experiments with those nonexistent cell lines. Hundreds of fake studies!

Fraud is much more common in science than you might think. A recent preprint provided some stunning evidence for this in a systematic review of preclinical depression studies. Specifically, the paper's authors were looking for fraudulent image duplication. Before looking at their results, we have to understand the taxonomy of image duplications. This comes from @MicrobiomDigest, and it's pictured here: The first type in the taxonomy is simply repeating a picture. This often happens for innocuous reasons, but it's still frequently suspect. When this involves duplicating parts of images within other images, such as what often happens with Western blots, it's probably a sign of fraud. The second type is less, but still, sometimes justifiable. This involves taking an image and duplicating it while changing its orientation, zooming in, or just generally doing anything that mildly obscures the fact that the image was repeated. A zoomed in image is justifiable if the point is to see something requiring a zoom, but usually zooming is just done to obscure that an image was repeated for more nefarious reasons. If it's not problematic, it will be stated in the text in some clear way like "Picture B is a zoomed in version of Picture A." Unmarked zooming is unfortunately common. The third type of duplication is pretty much unjustifiable. It involves duplicating an image, but editing some aspect of it. This is generally done, not so much to, say, make parts of an image more visible, but to mislead, to push a fake finding. So, out of 1,035 papers screened for a systematic review of preclinical depression work, the authors of this study found that 588 of those studies included images. Well, of those 588 papers, at least 19% had photo duplication issues: That's 19% with a very clear issue. Now maybe some of those are innocuous accidental duplications of the Type I variety. But if you look at the breakdown, it actually seems that there are more Type II and III, severe duplication events. That's bad, and this is a lower bound on how bad things are. Before going on, do note: 17 "Other" problems, not fitting into the typical taxonomy. These were described as issues like images being switched around, leading to mismatched descriptions, and cases where images appeared to come from a "paper mill" that was fabricating research at the request of paper authors. Now OK, this is bad. But what does it do? For that question, the answer is tough. The studies cited in the review which had detectable issues also reported 0.81 g larger effects than studies that weren't flagged, but they fit in with plenty of potentially non-problematic studies: Even when these problematic papers were stratified by their checklist-based "quality of reporting," they seemed to be indistinguishable from the papers that weren't obviously fraudulent. Their "quality" came out at 5.8(/10) versus 5.1 for all of the studies with a quality rating. The "risk of bias" wasn't much better, showing a score of 3.2(/11) for the potentially fraudulent studies and 2.9 across every study where the risk of bias was reported. That's extremely unfortunate, as it indicates commonly-used metrics for indicating study quality don't pick up outright, visually-detectable problems. Or maybe they do detect problems, and there are just huge numbers of undetected problems among remaining studies. That is the best case scenario, but it's not a happy one. But surely peer reviewers, editors, and subsequent readers of studies will identify and penalize papers with image duplication issues, right? Well, the fact that these were published speaks against some interpretations of that belief, so the real question regarding publishing is do high-impact journals, with their high-powered reviewers, editors, and readers, recognize problematic papers? The answer is no: studies with problematic image duplication are published in journals with comparable impacts to the ones non-problematic papers are published in, and they're cited just as often too. The authors had this to say: "We would suspect that the reason why most researchers would agree with the statement that research fraud is rare, all the while data seems to point to the contrary, is that they have not encountered it personally." Fraud is very prevalent in science, and there's every incentive for it to be, and there are relatively few tools for figuring it out. Fraud is rewarded, and not caught, so of course it's around, and that means many conclusions that might be mainstream might be completely wrong and we may just not know it, yet, or even ever. But there's a potentially happy note: frauds seem to be productive. Some of the time, they're productive through paper mills, and much of the time through simply rushing out papers at implausible rates. This means that a few people can corrupt entire scientific literatures, and that's bad, but it also means that, like violent and property crimes outside of academia's ivory tower, we can stop it. We just have to crack down on the few people who engage in extremely disproportionate levels of scientific malpractice. Right now, it doesn't seem like that will happen: "Journal retractions happen at a glacial speed, and author protests can sometimes be enough to prevent them wholesale." But the "scientific community cannot afford to act as slowly as the industry that publishes our reports." While science currently isn't equipped to deal with large-scale fraud, that doesn't mean it has to be. This problem is fixable, it'll just involve a lot of hammers coming down on a lot of fraudsters' necks. We just have to commit, because the problem won't fix itself. Give the paper a read. It's good: biorxiv.org/content/10.1101/…