« The Weeds

#FakeScience

2018-08-10

Senior health correspondent Julia Belluz and science reporter Brian Resnick join Dara and Matt to discuss the replication crisis in scientific studies. References and further reading: Brian's piece on the marshmallow test John P. A. Ioannidis' "Most Published Research Findings Are False," mentioned by Julia Scientists tired to replicate 100 psychological studies, 40% passed, mentioned by Brian Scientists also tried to replicate 100 economic studies, 60% passed Vox's survey of 270 scientists about the biggest problems in science, mentioned by Brian Julia's piece on Amy Cuddy and power posing NY Times magazine feature on Amy Cuddy, mentioned by Dara Julia's piece on 'big paper towels' campaign against hand dryers Mars chocolate study, mentioned by Julia 

Learn more about your ad choices. Visit podcastchoices.com/adchoices

This is an unofficial transcript meant for reference. Accuracy is not guaranteed.
Support for this episode? Country clicker lose an average of three hours every day, switching between all our work apps, but you can get them back with click on a flexible platform that brings all your central tools in one place: people prioritize tasks, collaborate on dogs, chat with your team and track gums. It's why companies like Hoover and web flow use click up is that Mission Control Centre, placing every other after we're using before. Click up even guarantees to help you save one day a week and get more done, it's completely customizable and it's free forever to dry click up today. It click up a dot com, slash weeds, this episode is brought to you by own up, every bank says they ve got great mortgage rates. So why are people
paying billions of dollars and extra interest every year, because it's too hard to know what a great deal looks like for. You are unique scenario. Own up makes it easy to get the personalized data to help you make a smarter home financing decision, learn when a great deal looks like for your mortgage, so you never over pay bills. Your profile today at o, not dot com equal how, opportunity and am, alas, idea number one. Four five await o five. Who was the answer? Not that how what the hell is that aspirational might place, but he can't contrast hello. What to another episode of the weeds on the box media Potass network here today with Julie balloons and bright Resnick from the science team, as was darlin, it has definitely been like a number of days since we last recorded
Yes, definitely how much time has passed? We're not you know just banking up. it's I'm on evergreen top. Meanwhile, amazingly, both man and I are recording that's podcast, despite the fact that both of us are on vacation right now. Yes, it's it's Moroccans! No, so we have been talking about broaching this this subject for awhile, because, that means we often talk about different kinds of of studies and papers that things in such a science. We get a lot of sort of interesting empirical results that you know. Smart people can cook up with different experimental designs and we talk about them and then we can just like move on you now. We like have good discussion about like what's true and then we're just gonna drop the subject, and I guess this turned out to be like not going
scientific practice to raise our oftentimes things are discovered, an academic studies and then, when people look back is maybe not true, and undiscovered right, and this is what does it only asterisk asterisk asterisk hostile tag, fake news, fakes, vague music, so I mean this. It was called replication right. It's like, I do an experiment and say have a result and in theory like you, should go, be able to do it and get the same, try to reproduce the result railing if it's a real thing that exists in the world and not just like a thing that I observed in the past tense during this one time that I ran my study using this. These pretty give set of methods on a particular group of people, yeah. You want to try to see whether you couldn't reproduce, The findings are not an that's kind of you know, one of them, but
rock ideas in the scientific method that you don't this man once you try to run it multiple times, multiple different in different settings and see whether the results reproduce, as others have decided idea that there is a replication crisis. I have heard that phrase, yes, rather earlier Knox, I've got you probably reserve until it a little. What that might mean, but like can be exploited boiling, how does it how? How did we either? If, if this is one of the better our principles of the scientific method, like his Julia was saying? How did we get to a point where the scientific community has like suddenly started in old, like decided that it The problem here is, I think, what we are hearing about today, a lot of it. You are hearing about the power. opposed studies not reproducing, or the Martian the tests which Brian his as written so nicely about recently, but a lot of them be a lot, we think about now, is a replication crisis like it its origins in discussions that have been happening for decades in medicine, so a kind of hit them
in stream more recently, but like you know in the Eightys and Ninetys people in the medical care. You were talking about how you know you we can trust the result of single studies in that we needed to build systems around how to bring together the results of many studies and systematic reviews and make sure our decisions were based on matters to us the named John Unities, has he con mainstream, I do not has taken our liberties, like the rock star replication in publishes seminal paper in the two thousand two thousand and five called why most publish research findings are wrong and uses a mathematical model to show why? Because of reason, Tobias, because of publicity. Unbiased or the fact that look, it's published tens to be positive in splash? Findings are not necessarily negative findings. Are you know an overall? The story is that researchers have had too much freedom,
in constructing the tests and analyzing the result. So a lot of things that used to be pretty commonplace and social science are now seen as being really bad practice like history, pretty common, where, if you were doing a study and for each participant in oh you're, just looking to see when you reach statistically significant result, and you can stop the study once you ve, reached a statistically significant result, That is the term you might have heard sometimes copy hacking, and it's like these are all sorts attack. Except you can use implicitly in a lot of these. Things happens like not, because people want to do fraudulent work just because they have been common practice in and they kind of feel intuitively crackling. Why would you add more participants to study. Where have you got the result and alive These things crystallized like round twenty ten, twenty eleven specially. There is this paper that was published in psychological journal that found evidence for pre cognition, so evidence that people observe something before it happened, and this
paper used all the commonly used methods and in psychology like there's, nothing like methodological wrong with it in terms of like what everyone in the field is doing, but yet this this finding this ridiculous and that kind of inspired like theirs, threats. The story in immediate is one of them. Biomedicine is one of them social science, like that paper kind of inspired this interest. action and looking at methods and looking at how common practices like stopping experiments once you ve reached certificates, significance how like just using forty participants from her, Bird and in an experimental study, is actually a recipe for false positive, not specific from Harvard. But just forty is like a commentary saying, might so yeah you're an academic girl, you wanna, do an experiment. You would like to buy, get the experiment right and you don't want to have to pay like a good Julian dollars to people who are going to have to like take time off work and like finding people
hard so studying the undergrad at the institution where you happen to work is like the easiest way to get an expert yeah there's other things too, like you can collect data on multiple variables. Let's say I study you: are I'm setting this class of college and then you know we're studying both like how they feel about legs. The steam and were studying like how they are happy? They let's say our tasks like improves self esteem, but doesn't do anything for happiness in the past. unlike okay, to forget about happiness, indices, I go. We found a statistically significant results for self esteem. If you had gone in saying G self Esteem is really what were interested in here, rather than just kind of like data mining, Jaso like when they said freedom before, like there, has been a lot of freedom and in science and social science and biomedicine took kind of choose your own path through both designing exe and and data collection and at each point,
your choosing, something like that you're introducing the risk of finding a fault, positive, you're, probably increasing the risk of finding a false positive, and so, like there's just been this movement, several years, also really Chris living around at twenty fifteen art, Call science, where a hundred psychological papers were attempted replication, only forty percent of them. The really crystallizing. This idea that we get science and assure footing and is not like a lot of these studies are wrong. There is not necessarily that in everything you had hit and scientific magazine or newspaper is wrong, but just There is a movement now to re, evaluate and reassess now that the fields have realised that there methods haven't been as rigorous as they could then now it's time to bring that rigour, highly kind of want to dig into like why this might be happening. Lately, aid mention publication, biased, like my
understanding. Is that the reason that scientific publications are so lake it takes so long to get a study published in all that is that they're trying to make sure that everything is as rigorous as possible like kind of sounds like the story you guys are telling is that a researcher feels pressured to find like a really sexy, splashing finding and is therefore cutting. Of these scientific corners that they might not otherwise cut. How did that happen there so so many factors that play a role and you have to come to think about the broader incentive structure in science. So if you want a career as an academic. You need be published in high impact journals. High impact journals tend to prioritize findings that our novel, that our splashing some are. Especially now they're publishing replication studies, but in general they want something it's you know, spicy in new So if you are trying to get a job at a university in your evaluated, that job you have to have a certain publication track record theirs.
incentive till I keep going on a path that has shown some positive results, so you get a positive result in many keep getting more and more and more it's funny. To that I mean it's not funny had even some mega analyses. Perhaps you ve heard that a good way to evaluate science to get all the studies done, topic and group them altogether and to see where the evidence lies in a we talk about systematic reviews, but if they're not done correctly, and if you don't account for tat for publication bias, which is this thing that you know what goes on two journalists tend to only be there like the happy result and the results they make. You look good as the scientists and all the other ones. You can just stashed away in a file jar those met ever a lot of them. Are they were? They were reveal the truth that may be the effect that researchers have been chasing, aren't there, because the literature buyers, to deposit the surface it. This is like. If I may, steady and, unlike ok, do almonds, cure cancer
I run it and like the people, a diamond like it, didn't care cancer. That's like that's boring. I and I can't get that published, but if enough people do enough little almond experiments and one of the it does cure cancer, or I can get that Renard and is like, oh, my god, I'm insecure cancer and then your men analysis of the Ottomans literature, you might find three studies that showed that the almonds cure cancer. Because the seven million studies in which they didn't, like nobody, publish or Three studies in which, like almonds, are shown to give you lake stronger hair and nails and to improve digestive functioning and what won't tell you is that all of those studies started out, as can almonds cure cancer, and those are the few positives they fat went well guess what brain, where you are talking about this way, it's a setting more than one. Very long ride. So it's like, if you feed almonds, to a control group
And then you study a million different variables, yeah right, like its there's, gonna be Sumter, and this is big in clinical trials tour. I think that's one thing that is imported differentiating. Psychology is probably now talking about pre right. Stirring. Yes, I'm your protocol. So, like you before you run. It said you have to say this is what I'm studying these that would put Brown was just driving the said that they'll come that I'm most interested in and so medicine again was kind of has been doing this for years so there's something called clinical trials dark of. Where are we It is our supposed to register their. You know their power for whatever the studies gonna be for all the world to see. However, it still then happen in many cases, there's this me little thing, you can see on their website. You can added history of these registries, and you see that go back and added, but their protocols were, after the fact so
on that still happens await you certainly merits it is usually register, but you're allowed to actually change it to change their fellow. That's it. I got a few things that purpose, but I've got it right, but I should say that their yeah, like as part of this whole reform movement. There is a big push tat, ten move things in a lot of people. Are doing better now than they have so that think that But one thing that's important to remember this crisis is actually good thing so we're seeing like science has an impact. Two system and there are increasingly different. efforts and measures being put in place to have these checks and balances that we need so a lot of what you're talking about a kind of these broad trends. But it seems lake in how this has come to light. It's been kind of it does it. It's happened in kind of one area of science and then another like how is that progression happened and We had a point now, where, like everybody and and Social sciences dealing with their own replication crises are their places where it still yet to head the replication crisis to replicate
mobile, he's all right, son infinite earths. One thing I'd think just my opinion and like reading a lot of this research farming land. This news that Social Sciences, the least for me, seem to be like a really transparent and public and really like kind of like the the outward face of this replication, and we can also called the open science movement. Like opens This is the answer to replication problems and so, like a lot of the studies, have been hearing about marshmallow test ego depletion, which is like this idea that our willpower is a finite source. These ideas, I haven't really endured replication, like they're kind of big p. cultural ideas, you probably read them about and like big got a magazine through the tops of things at work, their way into self help, bookselling, they really prove TED tied, it ted and also education, education policy, He has been really powerfully shaped by psychological studies, so these things
We encounter them all the time and so so again, maybe a little misleading to think like there's an extra problems. Psychology, just because we just hear about it So imagine there seems to be like a real effort in psychology. It enthusiasm around, particularly some younger psychologists? You really want to do this work and correcting the record, which is really cool, to see and report on, but the other there have other replication crises in other areas of science. sell studies, animal studies, obviously medical psychology, the many many areas there is even bomb. I remember there was another paper looking out trying to replicate a hundred studies and economics. Economics did a little bit better in psychology, think sixty percent, papers in that effort replicated, but I think does that going back to Julia was talking about Johnny Neediest? Who was writing about this in two thousand five? Just like these? cultural practices and the pressures that me,
get more likely that false positives get into the literature are are just in academia. You know maybe and there's no one story in the application crisis, as we can call it. You know it different in medicine, looks different and in psychology looks different in own, whatever I'm sure chemistry or in studies Julian I a years ago, we did this survey. We heard from two hundred fifty scientists across country actually probably cross the world just asking them were them big problems in science and the biggest thing about we came across as that institutions to science, don't verwoerd failures, they don't reward rigorous failures. People are afraid of rigorous failures. I remember one of humanism, rigour, failure. So, like you ve done all the work you ve registered, you're you're, study Your study is not subjected to pee
hacking in oh, you dotted all the eyes you cross the T S, but sometimes when you do that the thing you learn, the truth you learn, as is that there is nothing back and shouldn't. Even call those failures bud nightly. Isn't that the whole point of like the way you freezer hypothesis? Is that either lake you do see an effect or you don't either way you I identified what's going on there and he comes back this incentive and rewards terrain around the other, we're humans and you know Hake, should its you watching. Psychology go through this because I think a lot of true ASP, of human psychology are revealed in a lot of these com some debates around which papers replicating which are not, and in a we its really hard to see your work be challenge, even though you know Sciences kind of loftier than like ego. You know it's like there's a numbers and were supposed to test them dispassionately, but it gets really hard because people
it also has the incentive structure works now, like your name, is your working in you sums? Singular people are attached to ideas, certain labs kind of get a lot of money. First single idea that becomes like their trade mark Fang in his really hard to separate yourself from it? In people books in businesses, sometimes consulting businesses, built up around whatever this special. ideas and then, and then you solely arising within it. What's unexampled hovel there, cutty at once lots neighbour, whose antiquary shoes Harvard Ablutions a Harvard psychologist, and this is you ve heard of power opposing this is. This is like the classic example, and I actually think she can have gets an undue amount of attention because I guess it alive of these practices were really commonplace in the papers. dinner our colleagues published on power, posing which, as they did, you make this big expansive gesture. You actually feel more powerful.
Even more than that, they perform to find evidence at levels of hormone would would surge in relation to that- and this was the I dunno about forty college students and just kind of like a classic example of Small small studies can find false positives and she got on the TED Talk Circuit. She wrote a book. bout this like she made a brand out of this, and I think she attracted allowed negative attention now because these mature psychological study that she did failed and application. Continually fails and application, but because you like drew our attention for it in. I suspect there some sergeant involved. Who have you noticed? Singling out this one woman, who's gonna, leave you some attacks or not attacks, but a wearisome criticisms of Amy cut his work some blog like that. They really do get very mean, and the do like assuming a lot of arm excuses, site trying to make my He offered this idea, which might be true but near
So if we allowed a debate about how to talk about replication issues, the others are really dead. Feature that the New York Times magazine wrote last fall on Amy today that open the show nuts and is one of the things that it really illustrates. Is that as a lot of conversations that society or particular groups of people have had about systemic problems. Those usually start with pointing out particularly egregious examples of failure and kind of go. after those you know it's there. It's the Harvey wines, dean thing right, we're like it takes one particularly egregious example for people to start asking questions about what are the structures that make this possible, but the difference in this case from air the me to movement or from for that matter in our systemic, looks into a kind of policing, after you have cases of you. No cops using Rachel slurs or that kind of thing is that, instead of the people who are under attack, first being exponents of being like the people in power who took advantage of the system,
There are people who were playing by what were the rules at the time and its take in kind of point to them is failures to get those rules changed, although I feel I mean in her case right. I mean like one reason: this blood is that, like, I think you can sort of understand with any body like you do in an experiment that you can do right with the resources that are available. Maybe the samples not great. Maybe it's not huge. You find a result. You try to get it published right, but there. I think I got a sane person, even when you like believe in your work, and I think you you ve done it well, you ve play by the rules, would still have a certain man of humility about a result right and it's like to go to our in the way that certain people do based on one particular experiment. It is
seem like there's something a little egregious about that to me. as a journalist I have Sometimes you know gone to press with, like one random. Steady are right up of this. Where thing that, like I thought I had a good headline on and did not do like the greatest amount of due diligence imaginable for it, but like before I like wrote a book I would try to like dig in alone harder than than I one post, and unless it struck me about some of these settings, you know linkage, It's one thing to like get a study published without having replicated it twelve times or whatever
it's another thing when you see like endless mountains of like hot, takes piled upon each other all based on one thing when, like you know, you never gone back and checked it. I mean I met really like I feel like our experience. Is journalists actually puts us in a very good position to understand how, when you're, making a kind of drawing conclusions on your own. Your calculus may be different from when you know, there's a particularly strong professional incentive to develop a particularly strong formulation of that opinion. Did I totally hundreds and why it happens that I'm not like baffle, To give one example also, I recently read about marshmallow test, which actually has origins and the nineteen seventy is where. put a marshmallow in front of a kid in the basic task is like is tell the kid you can get. Two marshmallows of uterus patient, of your way by fifteen minutes so far, of work in the nineties, found like a really striking correlation kids, who are really good at this test in the sun
these are the seventies and eighties. there was a minister tended to get better as a test scores. I kinda like all these benefits of this being patient as a five year old raw, now kind of inspired. This idea that we can just teach kids, patience, character, education yielded They will yield benefits later on. a great early intervention to help kids be better adults and. What was most amazing to me, so this big blockbuster paper him out nineties hadn't been replicated until now in others is nearly thirty years later, so Walter Michel is a psychologist to spearhead of those areas on paper is like the heated things. Five if anyone help people followed up on his work I basically anyone listening to this right now. I'm willing to bet that you are you're dealing with stress, maybe there's it of it like an overwhelming amount, or maybe it's more like a low. But steady, drumbeat background stress. No, how you are experiencing stress, it's likely effect.
who'd you energy in so many other areas of your life, you feel, like stress, is starting to take over strain your relationships and shorten your temper probably tend to unload and better help is perfect, for that better help is customized online therapy that offers video phony live chat. Sessions with your therapists should hope to see anyone on camera. If you don't want you it's much more affordable than in person therapy and you could start communicating with a therapist none forty eight hours and distressing. It's an unbiased feedback. You be pretty surprise when you can gain for it see it's for you. The weeds sponsored by better help and listeners get ten percent off the first month, better help dot com such weeds, that's, B, e t, T, R, H, E lp, dot com, slash weeds.
It feels like you, don't even of hours in the day to get everything done. He might because you're missing out on three of them were those three. I was up how he fell into a deep dark abyss that opens up, amongst which, between working hours had those three hours to all the productive time we miss out on thanks to at home, distractions, disorganization, fatigue and some wonder the days future work should work with clear up. It does look up as a flexible productivity platform. That brings all you work at one place. That's all you chats apps docks and ass. One centralized place like mission control companies like we were in Google use click up to make the days more productive, managed projects, people and calls more effectively, but for teams of all sizes industries cook up to pleasingly fast feature One thousand plus integrations must have for anyone wanting to trap, manage and tackle their work in one place. In your hours, back with clear up trade free today took up the dot com. Slash weeds,
With the with the marshmallow paper, nothing, they me Cutty example. I look. I remember seeing the marshmallow study on like sixty minutes or something when I was a kid and I think one thing that they speak to his eye, the hype we create around them, the dutch media machinery and abuse these findings. With maybe more significance than they deserve? firstly, mad and graces his hard takes on single studies. Although we kind of build this pipe around their men, dumb and yet definitely not always deserve. This is hold changed. The entire way you report studies or your reporting is completely influence by alive these changes going on science, allow this recognition of many a year you're, very ethical. How do we respond Milburgh? Actually, diversity of reporting on evidence based medicine movement was kind of a wake up to realize. Ike woman, Do these single study reports in health liking
in fine studies, it say watermelons cause cancer. They prevent cancer of cargoes are like Felix for a long way. I hundreds of super food right. It's a super food. There also so you have a have vocs started. The series called show me: the evidence which was kind of trying to take more systematic approach to looking at risk. search evidence and incentive, relying on single studies trying to look for a system Dick reviews were char, I'm sure. Member of the weeds audience will know their collections of studies on a certain questions come to more fully supported conclusions, studies are weighted by their rigour. We ve been trying to do that. Have oxen. A direct response to that problem of you know. If you look at like the New York Times, archival, like vitamin e you'll, see, Finally, many you know prevents this annex This is bad, and maybe you should take more take lesson. That's the result of reporting on
the studies have been built is an area where, like I have come to appreciate some of the virtues of like partisan politics and people having their heels again like on certain issues right like a minimum wage and background on jobs and employment, like precisely because such a hot button, political controversy every time there is a new study. We dislike another counter who study and there's lots and lots and lots of work on this and, like I think, as a fair minded person is a little bit hard to know like what the truth is. But you can rest assured that it's not like a unexplored topic, or, like one guy, came up with a single, clever experiment. Sometime and then we all just ran with it because it sounded cool. But like what are the issues with the power poses right? Is it's like there's no one invested in like Anti power power
right. So it's like one thing comes up: it generates a couple like high be journalism stories. It catches some people's fancy. It please to like a certain kind of pressure This is right. There's like we would all love it to be the case that there was like an answer as imports. Thy right. Am I it's not naturally anybody's incentive to be like no, like I'm fanatically committed to like slouching body posture and I'm gonna through all my work at it. The way there is more politically contested type issues right, as you like, you really have this in health right words like every food group, has its own like lobby right I find me some reason why this thing is also nutrition. Research is notorious for this, and also the research into drugs. Where you have got big incentive structure, from the drug companies are from the blueberry council like this is
you remember when we probably all teenagers blueberries, became a super food was because blueberry. Sales were flagging in the blue by thing it's called the blueberry council. This to invest in antioxidant research and get out idea D, especially in the media? producing these studies that if you eat blueberries, you know you have more exposure to anti oxidants and therefore you gonna live longer and have better health outcomes. I'm very convinced of this is true, and I once I dont know what anti accidents are, but I'm Viewed from accidents yeah, they keep my oxygen away where confine, but you can find like the tree, not council, and the one that council and Grand Byerley humbly. Ninety part tat. Our ties are taper tallow, that's very fair of paper, towels verses hand dryers where, where does what big paper towel has been invested in research, to show that Andrew Iris disseminate alot of bacteria when you use them.
say the M Automatic Andrews and ban a rate of their blow hot hot air yet, and this little separate then than some of the replication issues that we ve been talking about, but now that the source of funding for a study is also a source of by and is a source that It's a sign that, whatever thing you're setting there may not replicate, goal of this whole movement. As has been explained to me, it's not like NASA they go through study by study and, like yeah, shame people ashamed studies, but is defined Alec. What makes for a rap couple science like water, the components of science, The studies that, when you of all these components and place like that. You feel good that this result is more likely to replicate. Then, if you had just used old ways and what is that look like they ate and Cypriot enthused about like getting into like some of the kind of what? What are the new structures that people are building to increase?
the rigor in their fields, so one is pre registration, and this is a really simple idea that you just commit to a plan. So it's saying I'm going to recruit this many people we're going to do this test. Study this variables, this is how I plan to analyze it and you submit that to a public forum for people to move regions nice and and its there. You know you ve limited your freedom in making a false result. More Missouri, so what kind of specific things is that that will happen? I've pre register my experiments now like white, but can I not differences that problem is like you're studying ten variables, and you can't just pick whichever one has the favourable finding in the end and say that was the important thing after the fact. So, for example, gonna run a study on Alzheimer's you're, going to say that you know this certain a fact in the MRI is what's important to me, and maybe this result on some test
cognition is important to me and the other things, secondary. Comes there less important, really going to focus on those two gonna, look at those two after a certain number of months. So what matters is the measure of acts and why, at twelve noon, because in my view, then you don't. You can't run in a study out to eighteen months and find positive back then in say? Ok! No! This is now well really after you have to determine in advance what a meaning are you trying to find what's a meaningful test and declare that openly and have it on the record, so that other people can hold you to account, and if you run twenty tersely, you have become a probability of each test, giving your false positives so if you the more tests your unlike like likely, more likely you're going to this fine one, that's given your false positives, though, is the way to constrain you to say like this is: do this is exactly what Julia said So, where does like in a standard likes? It is little significance test. What you're saying like this less than a one in twenty chance that this could be a random
Do you want I'll drop you value of that? That is not what does not work, that they have all been able to its here. They give. I have infinite degrees of freedom and, yes, the dependent variable is right. It's like, as the number of dependable I look at explodes yeah, like the odds of finding spurious correlations, go. Away in these registries you're supposed to say what the primary outcomes are see if the pick a couple, they are focusing on rights and then say that the outcomes, the matter and we're looking for result in those outcomes, but these Reggie These are just one of the solutions. There are many who, what other or the other many sources, this episode is brought. You buy fender football is back and the best bet you make is downloading the fan dual sports book app. It doesn't matter if new to gambling or an old pro fan. Dual has something for everyone and as an official,
sports betting partner of the NFL. You know, you're, better, safe, there's, also never been a better time to use. Fan do because, right now, you'll, get up to one thousand dollars back. If your first bet doesn't win, you can even turn small wager into a big payday with the same game, parlay that just sign up with a promo code Spotify to place. Your first bet risk free on fan, dual sports book Download Vanderpool today, twins. when plus and present in Pennsylvania, don't mind real money. Wager only refund she does not withdrawal site credit that expires in fourteen days. Restrictions apply, see terms at sports booked outfit it will not come. Gambling problem call one eight hundred gambler this happen It has brought to you by the rise in her eyes, and small business days are happening this Friday and every Friday this July come in and Verizon's team of train business specialists will give you a complimentary tech evaluation to help jump start your communications, connectivity and security plus limited
I'm offers like a new findings phone on the horizon. Not a customer, though even help cover the cost of which come to a participating store on Friday in July for Verizon small business days. Finally, ultra wide band available. Only in parts of select cities, five g nationwide available in twenty seven hundred plus cities offers available would select Sweden and business. Unlimited plan terms apply limited time, offer shop in store safely. Yes, I guess another big thing is trying to do again: systematic review so, instead of relying on sing, studies on a certain clinical question nor the desert systematic of user, also happening in social sciences. You're gonna take all the studies that you can find. Some of them also look for unpublished, research as well and you're. Gonna take All of that and see where it is, the truth lie like. Where would weigh more? direction are most of these studies. Leaning than these
These men are reviews of their done right. They can account for that publication by us. We're talking to release, assess it. So there has been cut off of statistical significance at put people's or less than point five and very strangely, a lot of studies tend to have p values of just under the hammer. You see here quantify ninety five percent a year and that's really suspicious, because if you try to figure out like how often you should find a p value of point of five, if your result is correct, We shall find that very awesome. You should find like much much lower is it it's really weird and science? When all of that p value reported, p values tend hover around the cut off is like in everyone's training. Like right under the limbo Paul zero. Using like that. It suggests that in the aggregate there is massaging guess of the debt write that, like a good experiment, might be right under the first
saw the petition it could be over. They do. They have had plenty of good experiments, should be way past, right or nowhere in the neighborhood. But if everything is like Joe, stun one side of the dividing line that suggests A lot of things were may be coming out on the wrong side, and then your tweaking tweaking we gang and as soon as you get across the cut off your garden, I'm done too weak. That's exactly right and in the good men analyses will like really try to dig into that to see, if that, if there is that suspicious numbers of echo, there's a suspicious number of studies. Or just they just crossed the threshold, and that's that's a reducing like any good men, and I guess you don't just like summarize. The conclusions of literature, but you might say, looking at His whole literature economy looks problems and then there are also like here the particular studies in this literature that lake are the gold standard,
verses Here- is a lot of dross roylake their clustering around a particular thing I dealt with this years ago, and I was writing about false rape reports. There is so much range in that because there are so many different forms. The protocols and like good layer views were going well. You can trust the study that highly asked seven Jeanne rape, victims in England or you trust the setting leg. Actually you no surveys, lotta, feeble, totally and Tom there The one thing that they're supposed to be doing is looking at the research methods, which is what you're getting at an end since There has been more discussion about the replication crisis, a need for reproducible, research, there's a whole field, that's been spy, called met a research and their whole rise on that I was just doing. Research on research and running statistic hotel on hundreds of studies and trying see you know, is the random, station in these studies plausible or not. Does it look too good to be true that wasn't big replication crisis,
happened recently with the Mediterranean diet. Studies on out even the Mediterranean Diet, as are noted, I think it's still generally agreed to be a pretty healthy diet, but there was a big randomize control, title of the Mediterranean diet that was considered like the gold standard testing this diet and it invites loved randomized people to either the turanian diet with extra nuts I spoke to you more nuts. The training died with extra olive oil or control group. And on this anesthesiologist in Britain he ran a statistical test on forming like five thousand studies, including this gold standard The training died, study called problematic and he found that it and a bunch of others had flaws in the random mediation process he found other problems to, anyway. It ended up leading to a hall re analysis of the pretty med study. This big Mediterranean diet study and
researchers spend months poring over their data. They found, indeed that there are massive problem. The two way people were randomized and so the New England Journal of Medicine ends up attracting the study and, pushing a re analysis of the results? Anything gets to O Brien was talking about earlier, with Ike. happens when you're so invested in an idea like the psychological care said, it might reveal like that. This was a landmark study from the New England Journal There is a lot of I guess. Criticism about how they handled the pretty med retraction in rebuilding. whether they were a bit tyrrhenian cause. They allowed the study to be republished with some financing of the. We are basically in and I have said the conclusion was still these results hold up, and this is still in all robust study and their raises. she's about when a journal is really invested in article and did so much to put it out there in the world and to promote it. Would work
happens that falls down a silly given all of that, you know what can like weeds listeners who probably of whom are not on the border. The New England Journal of Medicine, If you are, you are email, yeah totally loading of you bet. You know as kind of yours of knowledge, broadly people who, like might hear about scientific paper. From journalists who might like occasionally look at one once in awhile who might assiduously download every weeds white paper every week and now is going to be like bombarding us with e mails about the various ways that the White paper massage their results in a wider, the kind of take a ways about what you should be trusting as good science or not so bad. I just read: Julia's work, I don't know, I think one one thing is to understand: the breakthroughs are really really rare and that money here is something you know. This is a breakthrough, Sultan like we might have found. I don't wanna, be to sound super cynical, but against
everything goes back to join you needed, but you bitterly studied this. He looked out like papers in the top fighting five high impact journals. They declare that a drug that had been discovered was a breakthrough and then he track Moreover, I think something like ten years to see what actually ended up getting. onto the market and becoming widely used like what actually was a breakthrough. So out of twenty five thousand studies it? He identified only twenty seven of these drugs that were supposed to be breakthroughs, actually ended up going through clinical trials and only one ended up being used by a lot of people. So when you hear about a big breakthrough put on your bullshit detector, because this a very good chance that it won't under book one, and then God don't rely on the results of single studies. Look for systematic review, whose try to get a sense of where does The truth on this matter lie like, whereas the research, an agreement or not, and then, if you really dig into a study of there, are some things to look forward to that like why
in another solution to this whole replication crisis that we are talking to is publicly available data. So it's posts to be in. This will be good for science of more and more researchers is put whatever date, they had collected just on the ends that for anyone to access or if the, if there's some privacy concerns like they should make it accessible in a with with few hoops to jump through an like as a consumer. I think you don't actually need to go that data, but is nice, knowing that, like anyone, can analyze realise the David said, and he You have a sense that, like the researcher is like trying to hide anything as much. clear as a consumer who, like is not first and in statistics, he probably should not be going into our scientific data like this is a thing that I thinking go a little. You far interns scepticism like just like its unlike They first scientists to have a breakthrough super unlikely that Lake Joe acts will be able to interpret stuff bet,
Then the people who were oh yeah wrong thing is like it is nice to the issues, is a good sign. My people, good till. I know that other people who do know things can check their work and then probably gonna, look at vested interests behind a study or a claim that's being made. So he knows big. Brutal, really bank or who has studied here, you know, is it we did a review of chocolate and cocoa research, was funded by more you about this found that ninety five percent of these studies came to you have conclusions were favourable to chocolate or cocoa and no orphans tomorrow's, but they do. a very strong vested interest in finding that cocoa. Make you live longer and then you know make your hair shiny on your nails, stronger, Sears enemies to be a bunch more research into Let figure only we're, not letting you is what if you'd see sharply covered blueberries, then it will probably be very helpful.
oh, how I'm gonna have no no accident. I would actually highly here for the systematic review diet in which Julian houses, which particular foods and lake of her Julie is giving me a look right now. It is literally every time Julia rates at something like this. The basis is there is no all the easy way out like what you call a skeleton key We sometimes sometimes call ourselves on the signs team. The wet blanket pod weapon blanket Griffith Vocs. Does it feel like a guy after every headline could be like on palm. I think this is also why we don't let you on lake Stare Tuesday, we it's because like in a week, recently about a view, amazing gains of iodine, and that was based on census. Data like that was you know, pretty strong pay
you know all it takes is like one methodological. No at all and we'd oils are going about American. Probably look good in terms of a during journalism. Is that one I dont know is a completely acceptable answer in a lot more common one, then than you read about in science. News like that. It's ok be the questions are sometimes so much more fascinating. Many answers and if you look at how I've been trying to answer this question. You see how much effort, and so Julia like the basin question that you know is in nutrition. Researches like what is the optimal diet for a human being. Is such a simple question is taken decades barely now, however, much about it and that's cool and, like thousands of people have worked on the simple question and spectacle pillar that that we still have no great answer and isn't because science is gonna, fallacies, causing incremental and you know, like every true finding we have, I get. You know it's just like a gift we give to the future, and then they can better.
Assess it in and no go from there. I hope that I get one taken from the replication crisis that it's not that science. Is broken and let you know what's in crisis, scientific studies are by. by their nature, broken because their made by humans and they're going to flawed and the whole purpose of the replication crisis is to make things better, yeah we're gonna, learn things: it's not actually hashtag fake newspeak, science and, in any case like when somebody amidst faults, Elise, personally about a person, I trust, so I trust people who are committing faults and science fantastic and with, you know, if you have any faults to admit out their eyes listeners, you should check out the weeds. Facebook group a place to convene an early. He has said- and you know whatever else tell us: why are to say white papers are nonsense, other things like that. So really, thanks
and Brian for joining us and humming sound of this? Our engineer, Griffin Tanner, are producer bridge armstrongs of you out there in weeds listening land, and we will be back on Tuesday accessible, affordable broadband helps communities. stored their american dream for students. Lectures on a Chavez means rising above the poverty line and becoming valedictorian of international High School Langley Park, and thanks to access from eighteen t, it can help these dreams turn into reality. That's my eighteen tease, making two billion dollar three year. Commitment to help close the digital divide to more Americans, have a chance to succeed. To learn more is eighty t, dot com, slash connecting communities.
Transcript generated on 2021-09-12.