A Rate without Time?
Twice before (Part 1, Part 2) we have covered Dr. Brian Giordano’s study, Injury Rate and Patterns Among CrossFit Athletes, on our blog. To recap, the study used an online survey in an effort to find the injury rate in CrossFit. They found,
“75 participants (19.4%) had experienced at least 1 injury resulting from a CrossFit workout in the 6 months prior to filling out the survey that met the defined injury criteria. Of this population, 63 participants experienced 1 injury (84%), 10 experienced (13.3%) 2 injuries, and 2 experienced 3 injuries (2.7%).”
The phrase “injury rate” implies that there is a definite time duration of CrossFit training for which 19.4% of the subjects had at least one injury. But over what time period did those injuries occur? Is “6 months” the time frame? It would appear so, but some of the athletes had trained CrossFit for less than that. 35.2% had done CrossFit for 0-6 months and “average (±SD) participation over the 6-month period was 5.5 ± 1.2 months.”
So what we are left with is a percentage of injury, but an unclear picture of the time period over which that percentage occurred. As we’ve covered often, a percentage of injury is meaningless without a specified time frame. A 20% injury rate would be very high for 1 week of training, or very low for 10 years of training.
Russell Berger emailed Giordano and expressed these concerns. Giordano was unable to explain why he used “injury rate” without a time frame.
The errors continue. Giordano refers to “power lifting” movements, but includes the “squat, deadlift, press/push press, bench press.” The press and push press are not powerlifting movements.
He also concludes that “This indicates that CrossFit is a workout program that can function safely for athletes across all represented age groups if performed in a safe environment.”
That’s like saying driving is safe if you drive safely. It commits the begging the question fallacy; it presumes its conclusion (safe) in its premise (safe environment).
Yuri Feito Addresses Prevalence vs. Incidence:
The Orthopaedic Journal of Sports Medicine published a letter by Dr. Yuri Feito of Kennesaw State University entitled, Prevalence and Incidence Rates Are Not the Same.
Feito also emphasized Giordano’s unclear language:
“In their study, even though the authors refer to ‘injury rates’ throughout the manuscript, what they actually report are prevalence rates, which cannot be interpreted the same way as an incidence rate …
Per its definition, prevalence ‘represents the number of individuals in a population (group) that exhibits the outcome of interest at a specified period in time.’ On the other hand, incidence ‘provides a measure of the rate at which people without a condition develop such a condition (number of new cases) over a specified time interval (eg, 1 year).'”
Feito is too kind to point it out, but Giordano’s study doesn’t even meet a strict definition of “prevalence” since there is no “specified period in time,” as covered above.
In addition, Feito warned that Giordano’s “data could be misinterpreted by other readers and the media.” Given the media’s abhorrent record of inaccuracy covering CrossFit, this is certainly a concern.
Fortunately Giordano’s piece hasn’t received as much media attention as other flawed studies on CrossFit have. Perhaps the OJSM’s publicist isn’t as effective as the NSCA’s.
Giordano Responds to Feito
To his credit Giordano responded formally to Feito’s concerns, as he also did to Berger’s.
Giordano agrees with Feito that his study is discussing “prevalence,” not incidence, but doesn’t explain why his study failed to make this distinction. To reiterate Berger’s question, why didn’t he just say “prevalence rate” instead of the more ambiguous “injury rate? Furthermore, Giordano made confusion more likely by comparing his rate with other studies with different time rates and methods.
Giordano defends his ambiguous use of “injury rate,” vice prevalence rate, saying “it has been reported similarly in other sports studies.” He cites just one example.
In response to Dr. Feito’s note, Giordano finally does calculate an injury rate, “2.4 injures per 1000 hours of CrossFit.” He carefully notes that the injury rate his survey found is a bit higher than the injury rate for “walking, cycling to work, gardening, home repair, hunting and fishing, golf,” and dancing.
Nonetheless this rate is comparable to long distance running and triathlons and lower than that found for middle distance running, sprinting, and team sports. No surprise. Paul Hak and Yuri Feito have found similar results.
Have Giordano and Hak finally vindicated CrossFit? I don’t think so. The real world results achieved in CrossFit affiliates every day already vindicated CrossFit. If CrossFit really was less effective than traditional training and much more dangerous, the past five years of CrossFit growth are nearly impossible to explain. Why is the military still CrossFit’s largest single customer of CrossFit L1 seminars? Keep in mind that they hold the health records of every single service-member who has taken the CrossFit L1 Seminar.
Survey data is a decent first step, but it is admittedly flawed and less rigorous than lab experiments. There are two main issues with surveys: faulty memories and selection bias.
As Giordano says, “potential recall bias cannot be excluded from those results, as other studies have demonstrated a 61% accuracy of self-reported diagnosis in participants reporting over 12 months.” In other words, people have a hard time remembering how much they trained, or their injury history.
As for selection bias, Giordano stated, “We distributed our survey electronically to current participants who then chose whether to complete it. This introduces sampling bias because participants who have had injuries may be more likely to complete the survey.” So, it’s possible that the real world injury rate is substantially lower than the 2.4 per 1000 hour that Giordano found.
Academic Science: Post-Modern Priesthood or Fallible Laypersons
When we have criticized published research, one of the most common responses is “when it comes to matters of science, most are going to side with the men in white coats.” This may be true, but that doesn’t mean it should be.
In reality, scientists are fallible like we all are. Criticism and doubt are cornerstones of science. In fact, to believe current science uncritically implies a belief that we’ve already reached the pinnacle of knowledge. This wasn’t true of Newton or Einstein, and it’s certainly not true of Giordano or Kraemer.
The good news is that the OSJM printed Feito’s criticism, and Giordano responded intelligently to it. The result is better science and more information.