Imagine this: You have bulimia nervosa, a psychiatric condition that traps you in an unhealthy cycle of binge eating and purging. You’ve been recruited to participate in a functional MRI experiment on this devastating illness. As you lie in the scanner, you are shown pictures of pizza, chocolate and other high-calorie foods and you’re told to imagine eating them. You do this for 72 pictures of delicious, fatty foods. At other points in the experiment, you see pictures of bodies (sans heads) of models clipped from a women’s magazine. You are told to compare your body to each of the bodies in the pictures. You do this 72 times, once for each skinny (and probably retouched) model’s body. The experience would have been unsettling enough for normal women trying to eat healthier or feel happier with their not-so-super-model bodies. But for women with bulimia, it must have truly been a hoot and a half.
Luckily, the misery was worth it. When the researchers publish their findings, they claim to have shown that patients with bulimia process body images differently. In their conclusions, they say that their results can inform how psychotherapists should treat patients with the illness. They even suggest that it might someday lead to direct interventions, such as a targeted zap to the head using transcranial magnetic stimulation.
My recommendation? Cover your therapist’s ears and stay away from the head zapper. This study shows nothing of the sort.
Functional MRI is a widely used and quite powerful method of probing the brain, but it is only useful for experiments that are thoughtfully conceived and carefully interpreted. Unfortunately, many fMRI papers that make it to publication are neither.
One of the most common problems in fMRI is making bad comparisons. All fMRI studies rely on comparisons because brains are all different and scanners are all different. If you are going to say that Region X becomes active when you see a picture of chocolate, you first have to answer that crucial question: compared to what? If you’re interested in how the brain reacts to unhealthy food in particular, you might compare looking at pictures of chocolate with looking at pictures of raisins or eggplant. And if you’re comparing these comparisons across subject groups (such as patients versus non-patients), both groups had better have the same the control condition. Otherwise, you’re not even comparing apples to oranges. You’re comparing apples to gym socks.
Sadly, that is just what these experimenters did. They compared brain blood flow when the subjects looked either at junk food or skinny women with blood flow during 36-second stretches of time when subjects just stared at a small, white ‘+’ on the screen. The authors say that using a more similar control condition (say, imagining using non-food objects like a lamp or a door) would be bad because patients with bulimia might respond to these objects differently than healthy subjects. This argument is nonsensical. There’s no reason to believe that people with bulimia feel any differently about doors or lamps than anyone else, but there’s plenty of reason to believe that they would spend 36-second moments of downtime before or after comparing their bodies to those of models either obsessing or trying not to obsess about how their bodies ‘measure up.’
In fact, I
suspect that could not help but wonder if the authors didn’t originally intend to use this ‘+’ as the control condition. They actually had less crappy control conditions built into the experiment. As a control for imagining eating pizza and chocolate, the participants were also shown non-food objects like tools and told to imagine using them. They also saw interior décor photos and had to compare the furniture to those in their own homes – a control for comparing each model’s body to one’s own.
When the authors did their analyses using these (better) control conditions, they found very few differences between patients and non-patients. None, in fact, for the imagine-eating-junk-food portion of the study. For the comparing-oneself-to-models portion, they only found that patients showed less activation than controls in two regions of visual cortex. These regions may correspond to areas that specifically process body images. But would less activation in these regions mean that patients with bulimia process body images differently than other people? Not at all. If the patients were not looking at the pictures as much as non-patients or were more distracted/less attentive to them, you would see the same pattern of results. In short, the authors had no story to tell when they used the better controls. They had a ‘null result’ that would not get published.
Based on the design of their experiment, I
suspect that find myself wondering if this was how they originally intended to analyze their data.* And it’s really the only sensible way to analyze these data. Experiments like these include the ‘+’ condition to establish a baseline (essentially, what you’re going to call ‘zero’). These ‘+’ blocks also correct for an unfortunate phenomenon called scanner drift that adds noise to the data.
It’s possible that
I have to wonder if the authors decided to use the ‘+’ for their comparisons because they didn’t get any exciting results with the actual control conditions. If so, it unfortunately worked. Using the baseline condition, they found two differences between patient and non-patient activations in the food task and even more differences between the groups in the body task. Ultimately, the authors got their significant results and they got them published. But those differences have nothing to do with the causes of bulimia and everything to do with what flits through people’s minds while they stare at a plus sign.
Unfortunately, this is just one example from a growing sea of bad fMRI studies out there. And while many people do wonderful work with the technique and advance the field, others do it a disservice and set us all back. From researchers to reviewers, publishers, science writers and reporters, we all need to proceed with caution and evaluate papers with a critical eye. The participants in our experiments deserve it. The public deserves it. Most of all, patients deserve the best information we can give them. Science done well and served to them straight.
Update: I’ve made a few small changes to this post to clarify my intent. I don’t personally know the study’s authors and have no insight into their actions, intentions, or motivations. In writing the piece, I hoped to bring attention to a widespread problem in fMRI research. Of the study’s authors I can only say that they did some seriously flawed research. Why, when, or how is as much your guess as mine.
Since posting this piece, I’ve contacted the editor of BMC Psychiatry regarding my concerns with the paper. Not only have I received no reply from her, but this paper is still listed as one of the ‘Editor’s Picks’ on their website as of 1/5/14.
*For curious fMRI folk: each run contained 6 food/body blocks, 6 non-food/décor blocks, and only 3 baseline ‘+’ blocks. That means they collected twice the data for the control conditions that they supposedly didn’t intend to use than for the ones that they did.
Van den Eynde F, Giampietro V, Simmons A, Uher R, Andrew CM, Harvey PO, Campbell IC, & Schmidt U (2013). Brain responses to body image stimuli but not food are altered in women with bulimia nervosa. BMC Psychiatry, 13 (1) PMID: 24238299