Here’s an example of the kind of thing Christina was talking about in Tuesday’s lecture – only worse, really.
The link here is to a post by John Grohol on the PsychCentral blog (http://psychcentral.com/blog/).
It’s about a paper in Pediatrics (a peer-reviewed journal) about a study on whether pre-natal exposure to a possibly damaging chemical, BPA (used in plastics production, I think), affects hyperactivity and aggression in 2-year-olds. The headline for the paper on PsychCentral (probably based on a press release from the journal or university) says it does:
BPA Prenatal Exposure Linked to Behavioral Problems in Kids
- then the first paragraph says it doesn’t, really:
New research suggests fetal exposure to a chemical used to make plastic containers and other consumer goods called BPA is associated with a slight but nonsignificant increase in behavioral and emotional problems in young girls.
‘slight, but non-significant’ means that we can’t be confident that the difference didn’t arise by chance, and as scientists, we shouldn’t find that evidence convincing (or conversely, we should be fairly confident that the null hypothesis has been supported). So this is really, an example of the kind of negative result Christina was talking about.
So, how to deal with it? It could be stuffed away in the bottom drawer, though it would be worth publishing as a negative finding:
“We know that BDA has been shown to have neurotoxic effects elsewhere, but it doesn’t look as though moderate levels of prenatal exposure has much effect on hyperactivity and aggression in toddlers, so that’s one thing less to worry about.”
– no! Much better to publish it as a positive finding, even though it isn’t: ‘Deadly chemical poisons our kids’ brains!’
That gets round Christina’s negative result problem nicely*.
If you look at the detail Grohol gives further down the blog, it actually gets a bit worse. Grohol points out that the accepted level of difference for significance (established in the norming of the original scales) on the scores for the scales the researchers used is 10 points, and only two of the published 40 differences (for different age groups, boys and girls, etc) reach or exceed that level. Grohol comments:
Here’s a study that looked at a total of 44 variables (when you count the analysis of gestational BPA versus childhood BPA levels) and found significance in only 2 of them.
To me, that’s an interesting correlation.
Hang on: if we’re talking about significant at the 0.05 level, that means we would expect a result like this to arise by chance in one out of 25 trials. So, we find two results significant at this level in 44 trials? Isn’t that really very close to what we’d expect to find by chance? Don’t we teach you that if you do lots and lots of comparisons, you have to allow for the odd apparently significant result which WILL pop up, just by chance? Anyone who thinks it’s positive evidence is a statistical ignoramus. Here’s a recipe for scientific success, kids – do lots and lots of comparisons: some of them are bound to be significant – and statistical significance is the only thing that counts, right? (Actually, wrong – but maybe that’s another blog post.)
It seems like a month doesn’t go by when this journal is publishing more crappy science, and then draping it in a public relations campaign that gets everyone’s attention. (Actually, to be fair, the science is sometimes fine; it’s the over-reaching conclusions drawn by the researchers and the PR media machine that is truly vomit-inducing.)
I think that’s a good summary (though ‘vomit-inducing’ is both a bit strong and wimpy: this stuff doesn’t make me want to throw up: it makes me want to put my fist through the computer screen). Always suspend judgement on the headline, until you’ve read down to the the 27th paragraph – the truth is often down there in the details. Better still, look at the original paper, if you can. In this case, it’s at
Braun, Kalkbrenner, Calafat, Yolton, Ye, Dietrich & Lanphear (2011) Impact of Early-Life Bisphenol A Exposure on Behavior and Executive Function in Children †
*Actually, my analysis above is over-simplified. The results do look non-significant and unconvincing, but they are (mostly) in a negative direction. So we might say ‘it looks a bit as though there might be a negative effect but we can’t be (scientifically) sure about that’. But hang on: these are our children’s lives we’re talking about! Do you mean that there’s even a slight risk that exposure to BPA would lead my child to grow up to be a London rioter or to be like Paris Hilton? Shouldn’t we think about banning it right away, just to be safe? (review Christina’s points about large and small effects here). Maybe this study is at least a basis for further research after all.
…but it’s also politically complicated. Let’s say someone, somewhere, tried to make a court case about BDA and psychological damage. This might be energetically opposed by companies who find using BDA in their products convenient or profitable, or who might be liable for damages (it’s happened with asbestos and tobacco like this). so the plastics company calls an expert witness:
Counsel: One of the pieces of evidence the opposition has produced is the paper by Braun & al, which shows that two results out of 44 apparently showed a significant negative effect of my client’s product. Dr Miller, how would you comment on such an interpretation?
Miller: Errr, it’s not very convincing
Counsel: Didn’t you write, in 2011, that anyone who thought that such a result was convincing was a ‘statistical ignoramus’?
Miller: OK, yes
Counsel: So we’re dealing here with a case based on statistical ignoramicity?
Miller: You could say that, yes
So dodgy results like this are both (strictly) scientifically worthless, potentially significant (in the non-statistical sense) pointers, fodder for misleading scare stories, and hostages to fortune if used to support a case in the real world – all at the same time. This stuff is complicated (one of the messages of Schools of Thought, after all).
† If you look at the original paper, the conclusion is:
The results of this study suggest that gestational BPA exposure might be associated with anxious, depressive, and hyperactive behaviors related to impaired behavioral regulation at 3 years of age. This pattern was more pronounced for girls, which suggests that they might be more vulnerable to gestational BPA exposure than boys. In contrast, childhood BPA exposure did not exhibit associations with behavior and executive function at 3 years of age. There is considerable debate regarding the toxicity of low-level BPA exposure, and the findings presented here warrant additional research.
…which is pretty much what I was suggesting (note the use of words like ‘suggest’ and ‘might’), so maybe we (and Grohol) shouldn’t be too hard on the authors.