Devor and Smith Get the Band Back Together
If you want to study the effects of the Paleo diet, the Paleo intervention group should follow the Paleo diet. This shouldn’t be necessary to state. But it is.
Remember Drs. Steven Devor and Michael Smith? After failing to substantiate their first CrossFit study and getting hit with multiple lawsuits for fraudulent data, they’ve published another study on the athletes at CrossFit 614.
Their first study on CrossFit 614 alleged that 9 subjects dropped out “citing overuse or injury.” But the subjects, the gym owner, and the study’s own data coordinator all say the 9 subjects weren’t injured. And the researchers had no way of contacting the absent subjects to find out why they didn’t show up.
Devor and Smith’s second study is “Unrestricted Paleolithic Diet is Associated with Unfavorable Changes to Blood Lipids in Healthy Subjects.” It may be even worse science than the first.
Under “Dietary Intervention,” the study claims,
A Paleolithic diet, as first described by Eaton and Konner, was implemented for all study participants (11). Subjects were advised to increase their consumption of lean meat, fish, eggs, nuts, fruit, and vegetables and were instructed to strictly avoid all grains, dairy products, and legumes. All modern, processed foods including any form of processed sugar, soft drinks, and coffees were also excluded from the diets of the subjects.
Every line above is false. Let’s start with Line 1:
False Claim #1 “A Paleolithic diet, as first described by Eaton and Konner, was implemented for all study participants”
CrossFit 614’s challenge did not require full compliance with the Paleo diet. I asked CrossFit 614 owner Mitch Potterf if the subjects followed a strict Paleo diet for 10 weeks. He replied,
No. We gave points for several different behaviors ( men will die for points) and took away points for negatives. Ex)
1 Point-Paleo meal/snack (Pick it or kill it, no dairy, grain or processed food)
1 Point Zone Meal/snack ( We used a few CFJ articles to explain as well as books)
3 Point Paleo/Zone meal/snack
Negative 1 point for each Non Paleo/Zone meal
Negative 1 point for more than 2 Alcoholic drinks/week
Negative 2-5 pts don’t remember exactly for tobacco use
So the subjects could have followed the Zone diet only, and not Paleo. Or they could have done neither and simply took the hit in points.
False Line #2
“Subjects were advised to increase their consumption of lean meat, fish, eggs, nuts, fruit, and vegetables and were instructed to strictly avoid all grains, dairy products, and legumes.”
As seen above, CrossFit 614 did not instruct its athletes to “strictly avoid” non-Paleo food. In fact, the challenge encouraged consumption of grains, dairy, sugar, and other foods in cheat meals. Potterf states, “We also gave bonuses for the most epic cheat meal (AMRAP90min) caught on camera (winner got bonus points losers gambled and got negative points).”
False Line #3
“All modern, processed foods including any form of processed sugar, soft drinks, and coffees were also excluded from the diets of the subjects.”
Even the athletes who did try to stick to mostly Paleo foods did not exclude coffee from their diets. It is strange that the researchers would even assume that Paleo dieters don’t drink coffee, the choice beverage of the Paleosphere.
Potterf’s own noble policy is that he “would never allow coffee to be removed from the diet of anyone at my gym.” As for the other fun foods, they were also allowed in the challenge, as long as the athlete admitted it and lost points.
In summary, Potterf doesn’t think a single subject followed the dietary intervention that Devor alleges:
I would say we probably didn’t have anyone go strict for more than a few weeks before they cheated. It really wasn’t a Paleo Challenge. The truth is nobody really knows what people actually did because they didn’t write down what they ate, just whether it could score them points or not.
How Did Devor and Smith Get it Wrong?
At this point it’s not clear why the researchers completely misrepresented this dietary intervention. I asked Potterf if they observed any of the subjects eating, ever. He responded, “No, not unless they were hiding in the bushes with a camera.”
If the researchers didn’t watch any of the subjects eating, perhaps they could’ve compiled diet logs during the intervention, right?
Nope. The researchers didn’t make that effort, either. They collected only 8 of the 44 subjects diet logs. But it gets worse. Those diet logs didn’t come from the actual intervention period.
A few months after the whole challenge Mike Smith asked me to hand out some diet logs and get people to record what they ate during the challenge. A few returned their sheets but they couldn’t really remember exactly what they ate so they ( Smith) just told them to write down a sample of an average day and they would use that for the whole thing.
So, will the researchers ever explain why they made these false claims? If Devor’s history is any sign, it may take a lawsuit.
So what did Devor conclude about the Paleo diet based on subjects on who did not follow the Paleo diet?
In conclusion, our study shows that the Paleo diet was significantly deleterious to blood lipid profiles in healthy subjects concurrently participating in a CrossFit- based, high-intensity circuit training program. Specifically, subjects with optimal initial blood lipid values demonstrated the greatest increase in LDL, TC/HDL, TC, and n-HDL values, along with the greatest decline in HDL values, following the 10- week Paleo diet intervention.
We don’t yet have any evidence, besides precedent, to doubt the blood lipid levels Devor and Smith collected. That said, this statement is also false. It asserts that the Paleo diet worsened the blood lipid levels of the subjects in that study. This could only be true if the subjects all followed the Paleo diet as described in this study. They didn’t.
Therefore, we can’t say what caused the admittedly surprising deterioration in blood lipid levels. Did they make too many attempts at winning the 90-minute AMRAP cheat meal challenge? One of the subjects admitted to eating 1-1.5 pounds of bacon a day, because it was Paleo. Perhaps this was a common behavior among the subjects, and it affected their blood lipid levels.
We can guess. But we really have no idea what was responsible for those changes. And neither do the researchers, though you’d never know that from reading the study. The accurate title for this study would have been “Varied Nutritional Approaches are Associated with Unfavorable Changes to Blood Lipids in Healthy Subjects.” But who would publish that?
Section 2 of the ACSM’s Code of Ethics states, “Members should maintain high professional and scientific standards and should not voluntarily collaborate professionally with anyone who violates this principle.”
I’m not aware of any scientific standard that tolerates repeatedly publishing inaccurate and unsubstantiated claims. If this behavior is not enough for the ACSM to revoke Steven Devor’s fellowship, what does it take? Where does the ACSM draw the line on scientific misconduct?