3 Reasons to Ignore (Many) Randomized Trials

Randomized trials are worshiped as the “gold standard” of health research. Personally, I think they’re the gold standard example of science losing its grip on reality at times.

Last week, researchers published a randomized trial in which low-carbohydrate diets were reported to be more effective than low-fat diets at increasing weight loss and decreasing cardiovascular risk factors. The study was widely publicized and widely criticized because whenever you combine “diet,” “weight loss,” and “randomized” in one sentence, mass hysteria ensues.

I’m not going to rip the study itself; enough people have done that already. My issue is – why do we worship randomized, controlled trials (RCTs) in the first place?

Many scientists argue that randomization is the only way to prove causation. The rationale is that you create 2 treatment groups that are identical in every respect, aside from the treatment that is being studied. Controlling everything else allows you to conclude that the treatment caused any differences between groups.

In this case, participants were randomized to 2 diets, low-fat or low-carb, but otherwise followed the  same regimen. Their carefully-designed regimen included weekly dietitian counseling; nutrition education materials, cookbooks, shopping lists, and free meal replacements provided by the research team; and instructions to not change their physical activity.

In other words, they followed a regimen that nobody on Earth follows.

That’s what RCTs are about – disconnecting from reality. I’ve worked on randomized trials, and I understand that this approach has a lot of scientific value in a lot of contexts. But I think it is vastly overrated in nutrition and physical activity research for several reasons:

REASON #1: We don’t eat and play in a randomized, controlled world

Let’s assume that the study was “correct.” People who were randomly assigned to the low-carb diet lost more weight.

What’s your point? They lost more weight when everything else in their life was being controlled. The investigators were not controlling participants’ every move, but they were striving to create an artificial world in which all men and women are created equal except for their diet.  A world nobody actually lives in.

When RCT proponents brag that this study design can prove causality, what they really mean is that it can prove causality in a reductionist sense that has zero applicability to our daily life. An average person doesn’t follow a low-carb diet like they do in a study protocol or the pages of an academic journal. An average person on a low-carb diet is not receiving weekly counseling, shopping lists, and free meal replacements.

As countless experts have said, the best diet is the one you can stick to over time. Not the one that works when a team of MDs and PhDs is closely monitoring your life.

Then there’s another matter of fact that RCT proponents rarely acknowledge …

REASON #2: A lot of treatments cannot be randomized

Soda taxes, portion size limits, marketing regulations – these are the “treatments” that policymakers (a.k.a., people who make real decisions) discuss while scientists are feuding about carbs and fats. Some RCT-obsessed scientists oppose taxes or other policy proposals on the rationale that no RCT has proven that these policies work.

Again, what world do the RCT proponents live in? How would you do a randomized trial to determine if, say, state soda taxes lower sugar consumption? Would you gather 50 governors in a room and say:

“OK, people, we’re going to randomize the 50 of you. Twenty-five of you will have a penny-per-ounce soda tax applied to your state, regardless of what you or your voters think. Twenty-five of you won’t have a tax, regardless of what you or your voters think. Sound good? …”

“What? That’s illegal?? …”

“But this is a randomized, controlled trial!!”

I’ve seen plenty of scientists insist that we need RCTs before we implement any of the policies listed above. I’ve never seen a scientist explain how he or she would do that.

REASON #3: RCT’s are not always studying what they say they are

Reason #3 isn’t about integrity. Again, it’s about reality.

RCTs essentially pull people from their normal lives and stick them in an alternate universe, with the goal of studying that alternate universe. But how do you know you’re not studying the effect of pulling people from their normal lives?

In May, researchers reported that drinking diet soda helped people lose weight, based on results of a randomized trial. But the “treatment” they studied wasn’t people drinking diet soda per se. It was actually the opposite – the researchers recruited habitual diet soda drinkers and forced half of them to stop drinking diet soda. This “treatment” was implemented in the midst of a weight loss intervention, and people who were forced to kick their diet soda habit didn’t fare as well in the intervention.

Does this show that drinking diet soda helps people lose weight? Or does it show that forcing people to abandon a habit makes it more difficult to lose weight? Those are two different things. Personally, I’d lean toward the latter explanation, but I don’t know. More importantly, the study’s lead author didn’t know, as he conceded on Twitter.

In short, randomized trials make great fiction but they aren’t always the best science. The harder you try to construct an alternate universe by controlling everything, the less you know what you’re looking at. Or if you even care what you’re looking at.

Then why do we care about them so much?

(Click here to read more about the randomized trial that compared low-fat and low-carb diets. Several excellent commentaries on it were written, including Dr. David Katz, Ted Kyle, and Andy Bellatti.)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s