{"id":18770,"date":"2019-07-08T14:49:35","date_gmt":"2019-07-08T21:49:35","guid":{"rendered":"https:\/\/www.podfeet.com\/blog\/?p=18770"},"modified":"2019-07-08T14:49:35","modified_gmt":"2019-07-08T21:49:35","slug":"ccatp-601","status":"publish","type":"post","link":"https:\/\/www.podfeet.com\/blog\/2019\/07\/ccatp-601\/","title":{"rendered":"CCATP #601 \u2013\u00a0Dr. Maryanne Garry on Why Science is Broken"},"content":{"rendered":"

Dr. Garry, self-proclaimed Crusher of Dreams, joins us again on Chit Chat Across the Pond, this time to talk about how she thinks science is broken and why. Dr. Garry is a professor of Psychology at the University of Waikato, and also professor of the NZ institute of Security & Crime Science. In this episode, she takes on the topic of how science journals choose which papers should be published, and the flaws she sees in that process that cause the extraordinary claims to be published, rather than those efforts with less flashy premises. She backs up her thoughts with specific research that has been conducted in reviewing how well these claims of amazing success pan out over time.<\/p>\n

Please follow Dr. Garry on Twitter at @drlambchop<\/a>.<\/p>\nhttps:\/\/media.blubrry.com\/nosillacast\/traffic.libsyn.com\/nosillacast\/CCATP_2019_07_08.mp3<\/a><\/audio>\n

mp3 download<\/a><\/p>\n

<\/p>\n

Rough notes for the discussion with links to sources:<\/p>\n

BACKGROUND<\/p>\n

It seems like every week or so, we read another headline or hear another story about some \u201cdiscovery,\u201d such as:
\na. \u201cUp to 25 cups of coffee a day still safe for your heart\u201d
\nb. \u201cfish oil reduces [whatever]\u201d
\nc. Power posing makes you feel powerful<\/p>\n

These stories come from various places:
\n\u2014the press releases scientists and their universities send to the media,
\n\u2014 journals that publish a specific paper and want to publicize those findings<\/p>\n

And not just about coffee or fish oil or a posture. You can find the same kinds of things about the results of clinical drug trials\/or how your brain responds to some situation\/or how coconut oil is going to save you\/carbohydrates are bad\/you\u2019re a nicer person when you hold a warm beverage than a cold beverage…. whatever.<\/p>\n

…and then maybe a few years later, you hear: oh wait, coffee….just kidding. Fish oil? Uh, we were wrong. This clinical drug trial? Didn\u2019t pan out.<\/p>\n

In the late 1980s, John Ioannidis (pronounced you-need-ees), a medical researcher, started to wonder about this roller coaster ride. He showed pretty clearly that the whole landscape\/the whole structure\/ that surrounds research inadvertently rewards the wrong things. It rewards surprising results; results that would have a big financial payoff for the university or drug company. It rewards scientists who publish many high-profile papers in the \u201ctop tier\u201d marquee journals. Those are the papers the media covers most, that other scientists and doctors and clinical psychologists and various practitioners read the most.<\/p>\n

But if you step back and think about it, science is not splashy. When you stand on the shoulders of giants and build on their work, most of science is probably incremental. You have a hypothesis, you make a prediction, because it\u2019s the next logical step. It isn\u2019t usually surprising. That doesn\u2019t mean it\u2019s not cool.<\/p>\n

Also, think it through: when you find out you\u2019re wrong, it\u2019s not great, right? Science is supposed to be self-correcting. But the people who do science are….people. People don\u2019t like to say \u201coh wow, so you know that really surprising finding I just published in a really prestigious journal ? It was wrong.\u201d And the really prestigious journal doesn\u2019t like to advertise \u201coh hey, that paper? Nope. Wrong.\u201d<\/p>\n

So Ioannidis focused on a set of articles that had been published over 10 years in medical journals and found that \u2014as a result of further research\u201441% were wrong or exaggerated. But of course, the further research doesn\u2019t make as big a splash as the original research.<\/p>\n

And that, Ioannidis showed, is just the tip of the iceberg. He wrote a well known paper called \u201cWhy Most Published Research Findings Are False\u201d (show notes at 1), a real nerd-out paper.<\/p>\n

So: Ioannidis wasn\u2019t the first one to voice this concern, but he did it so effectively that he kicked in motion what we now sometime call the \u201creplication crisis\u201d in science.
\n\u2014-
\nIn 2012, scientists followed up on 53 \u201clandmark\u201d cancer studies and found that 11% of them held up. [show notes at 2]
\n\u2014\u2014-
\nIn 2018, scientists followed up on a big bunch of high-profile papers from psychology, economics published in Nature or Science…the \u201csaw off your arm to get your paper published in there\u201d journals. They showed ~2\/3 of those findings were reproducible.
\n\u2014-
\nHow did we get here? Science isn\u2019t broken…but the system has moved away from the fundamental principles of science.
\n\u2014science is self-correcting only if scientists correct the science.<\/p>\n

What happens now is that research is so expensive, so high-stakes, that if you do some research and don\u2019t get the results you expect, there are often real risks to your reputation. Sometimes to your job.<\/p>\n

Around the world, universities have been put under so much pressure to justify their existence, to bring in external grants to fund what legislators used to fund, that it\u2019s completely unsurprising the pressure trickles down to those of us who work at universities.<\/p>\n

\u2014you can promoted or not based on, among other things, the kinds of journals you publish in\/how much external grant money you bring in.<\/p>\n

\u2014in the US, if you\u2019re at an early part of your career, you can lose your job if you fail on these counts. So let\u2019s put that another way: You\u2019ve got 6 years to prove yourself as a good scientist…but really the criteria your university applies to measure how good a scientist you are ..are the very criteria that helped get us to this crisis in scientific research.<\/p>\n

\u2014So if you do a study, and you don\u2019t get the results you expected, now all of a sudden journals don\u2019t care. They don\u2019t want to hear \u201cI had this cool idea for surprising findings but it didn\u2019t work…so hey.\u201d Now what do you do instead if you\u2019re a scientist? You think \u201cWell, there must be something in this dataset that would be interesting. I can\u2019t WASTE these data.\u201d There\u2019s even a fear that it\u2019s unethical to spend taxpayer money and waste the time of the people who take part in our studies….and then just toss this study in the trash can.<\/p>\n

BUT we also know that one of the principles of science is that you have to generate your hypotheses ahead of time. You can\u2019t generate hypotheses after you know your results.<\/p>\n

Also, the system as it is encourages scientists to be protective of their methods, data, materials….all the things that do not help the scientific community advance the science.<\/p>\n

FIXES
\nCreate and then lodge an analysis plan that specifies, ahead of time, \u201chere are my hypotheses; here are the analyses we\u2019ll do\u201d
\nMake your data and research plan and the materials available to other scientists so they can check your work.<\/p>\n

My colleague Simine Vazire says if you\u2019re a scientist, you have to keep asking yourself this question: \u201cHow will I know if I am wrong?\u201d One of the best ways to answer this question is to, as she says, \u201cgive your critics the ammunition they need to show you that you\u2019re wrong.\u201d That\u2019s a hard thing to do when the stakes are so high.<\/p>\n

We also have to change the system of incentives and rewards so that we don\u2019t make rock stars out of the people who publish an unrealistic number of papers, an implausible number of those \u201choly cow\u201d surprising findings, and instead go back to focusing on solid research….recognising that much of science is slow and incremental and not going to drive a lot of traffic to your university website or Buzzfeed or whatever.<\/p>\n

Reference Papers:<\/p>\n