Irregularities in a Study on Soccer Headgear And Concussions

This week the British Journal of Sports Medicine (BJSM) published the results of a randomized trial of headgear to reduce concussions in soccer players.

Here’s an example of the type of headgear being investigated, in case you’re curious:

One of five different types of headgear tested in the study.

Before I jump in to my criticisms, I do want to commend the authors for running a randomized trial on an important question. Preventing concussions is a matter of physics. It’s all about dissipating collision forces so less of it transmits to your skull and shakes your brain inside of it. Helmets in sports like football were designed for a totally different engineering task: preventing skull fractures, which they’re great at. And as big of a concern as concussions have become, we only know of one surefire way to prevent them: stop collisions that shake people’s heads. All that is to say rigorous studies on new technologies that claim to protect the brains of sports participants from concussions are extremely important.

I also want to be upfront about what I expected this study to show, because scientists almost always approach a topic with some preconceived notions; we need to acknowledge and embrace that. I expected headgear to have virtually no effect on concussions.

And lo and behold, the randomized trial showed just that! So why am I writing this critical post?

Well, it’s dumb luck that I looked at this study more closely even though it confirmed what I expected to see. Someone tweeted out the main results table from the study and something didn’t look right – in fact, something was mathematically impossible (SEE Irregularity #2 below). That led me to read the whole paper, and there are some substantial irregularities in the analysis I want to call everyone’s attention to.

The below points aren’t my only concerns with the study, but they’re the biggest ones to which I want to call everyone’s attention.

Continue reading “Irregularities in a Study on Soccer Headgear And Concussions”

Diagnosing CTE in the Living: Proper Control Groups and the Importance of Your Research Question

A very interesting if preliminary paper looking at the possibility of using positron emission tomography (PET) scans (similar to a CT or MRI) to measure accumulations of a protein called tau in people’s brains was published this week in the New England Journal of Medicine.

Tau is important because its accumulation in the brain is the main way that chronic traumatic encephalopathy (CTE), a degenerative brain disease resulting from repetitive head trauma such as is experienced in contact sports like football and ice hockey, is diagnosed. CTE currently can only be diagnosed at autopsy, severely limiting research on the disease. A test for tau in the brains of the living would be a game-changer for CTE research.

A number of people, however, have been criticizing the paper for not including the right kinds of people. Specifically, while the study looked at ex-football players with cognitive and neuropsychiatric symptoms and healthy non-players, some seem confused or even suspicious about why it didn’t also include healthy ex-players? The short answer is because it shouldn’t have. Here’s why:

Continue reading “Diagnosing CTE in the Living: Proper Control Groups and the Importance of Your Research Question”

“Why Attack Football? Girls’ Soccer Has More Concussions!”: A Study in Bad Reasoning and Worse Stats

When I started in football injury epidemiology I focused on muscle, ligament, and bone issues. I shied away from brain injuries.

There were two reasons for this. First, I thought enough smart people were already working on brain injuries that my time was better spent elsewhere. Second, as a new entrant to the field I didn’t want to touch such a heated area. But since my article with Dr. Kathleen Bachynski estimating the prevalence of chronic traumatic encephalopathy (CTE) in NFL retirees appeared in Neurology last November I’m off the bench and into the game.

Seeing the backlash to those working to measure and quantify the short- and long-term impacts of brain trauma in football has been enlightening. Some of the points football’s “defenders” make have merit – we don’t know about the prevalence of CTE in high school or college players, for example, nor do we have a good handle on how many ex-players will show actual symptoms of CTE and other brain diseases rather than just the brain damage associated with them (although we do know that ex-NFL players die of neurodegenerative diseases such as Lou Gehrig’s disease at 3-4 times the rate of the general population).

But as the push to ban tackle football for kids 14 and younger gains steam, there’s one talking point from football’s “defenders” that is so callous in its logic and fallacious in its statistics that I feel compelled to call it out directly here.

Continue reading ““Why Attack Football? Girls’ Soccer Has More Concussions!”: A Study in Bad Reasoning and Worse Stats”

Article Review: Pitching a Complete Game and DL Risk

Despite the name of this blog being NFL injury analytics, since I’m now spending at least a third of my time working in baseball I’m going to start writing about that, too.

I wanted to start by discussing an article that came out a couple months ago in the Orthopedic Journal of Sports Medicine entitled “Relationship Between Pitching a Complete Game and Spending Time on the Disabled List for Major League Baseball Pitchers.

tl;dr: This article makes a claim that is completely unsupported by its analysis. Please don’t use it. Read on below for a deeper dive into (some of) the big reasons why.

Before I get into that, though, I just want to say that I’m only writing about this article in particular because it happened to slide across my Twitter timeline on a morning when I had some spare time and motivation to write about it. I do think the errors here are particularly egregious, but the same types of mistakes are extremely common in a lot of sports injury papers. I hate to feel like I’m singling out just one research group, so please understand that my criticisms will likely apply to a lot of what you read, not just these folks.

OK, onward!

Continue reading “Article Review: Pitching a Complete Game and DL Risk”

New Sports Performance Technologies: The 4 Key Questions Teams Need to Ask

I’ve had some thoughts on this topic for awhile, and then this article from a couple scientists with, I believe, the San Antonio Spurs came out in the Strength and Conditioning Journal. It does a great job at summarizing the key issues, but I wanted to try and distill it down to a 4-question framework that I use when evaluating a new sports performance technology for teams that I consult for.

So here are the four simple but broad questions that I ask with any new technology, in the order in which I ask them. I’ll use a single example of a hypothetical portable tool that measures forces acting on the shoulder joint of baseball pitchers to illustrate my four questions.

Continue reading “New Sports Performance Technologies: The 4 Key Questions Teams Need to Ask”

Just How Dangerous IS the NFL vs. Other Sports?

Turns out, yeah, pretty dangerous, actually.

As I’m working on my dissertation literature review I figured I’d put a small piece of it to good use for someone other than my committee. 4 people would normally read this, so I’m really hoping we can double that.

There is a wide-ranging belief that the NFL has the highest injury rates of the major North American sports, but by how much? Because game injury rates (rather than injury rates in practices) tend to be easier to calculate (and more available across studies) than those incorporating practices, we’ll focus exclusively on those.

We’ll need to introduce one concept to make the analyses below make sense: the idea of an “athlete-exposure” (AE). This is simply defined as 1 athlete participating in a practice or, in the case of this article, a game. Thus a single NFL game where every available player plays in the game would count for 92 athlete exposures – the 46 guys on the active game day roster on each of the two teams. All the injury rates below are presented per 1,000 AEs.

The studies compared below all have differences in how exactly injuries are defined (namely how much practice/competition time a player has to miss for something to count as an “injury”) and the years they covered, but in an effort to ensure high-quality data I’ve limited the studies to official league injury surveillance systems wherever possible.

Injury Rates Across Sports

Among the “Big 4” North American sports, the NFL has by far the highest game injury rate at 75.4 per 1,000 AEs. This means for every 1,000 players playing in a single game, 75.4 will suffer some sort of injury. So in a single game with 92 AEs, you might expect about 75.4 x 92/1,000 = ~7 injuries.

Compared to the other Big 4 North American sports, the NFL thus has roughly 4-5 times the in-game injury rate of the NBA, MLB, and NHL.

What about more international sports (MLS statistics weren’t readily available, so soccer goes here, stop complaining)? The NFL also has roughly 4 times the in-game injury rate of soccer and, interestingly, Australian rules football.

The only sport that comes close to touching the in-game injury rate of the NFL is rugby. The data are variable, but in general rugby seems to have either a similar or higher injury rate to the NFL. I wanted to look into concussions specifically in the NFL vs. rugby, but I’ve found some conflicting data and I want to do more research on this before saying anything.

I’ll spare you the gory details of the data sources unless you want to click below the jump, where I’ve included everything you should need to evaluate these on your own if you want!

UPDATE ~6 hours after original post:

I purposefully simplified or ignored quite a few issues in writing this post to try to communicate my main point, but some good questions from friends have come up and I want to briefly address a couple shortcomings of this analysis:

  1. This post only considers one possible way to measure injury rates: per player-game. The results would likely look substantially different if we used another, more precise denominator such as player-hours. The NHL numbers, in particular, would look higher relative to the NBA and MLB in such an analysis since hockey players on average play a lower proportion of each game than baseball or basketball players. So this is not the be-all end-all answer for which sport is the “most dangerous.”
  2. Along those same lines, although the risk of being hurt in any given NFL game is way higher athletes play way more games in the NBA, NHL, and MLB. So the differences across sports in an individual athlete’s risk of being injured in a season are going to be much less than the differences in injury rates I’ve shown here. Maybe I’ll take a stab at calculating risks in another post!

In a nutshell: this is just one way of measuring which sport is more “dangerous,” and it’s probably the way that casts the NFL in the most negative light. Injury rates per player-hour or 1-season risks would likely make football look less dramatically bad.

Continue reading “Just How Dangerous IS the NFL vs. Other Sports?”

Checking a Reported Drop in Starting QB Injuries in 2016: Definitions and Counts

I saw a remarkable stat reported in Pro Football Talk (PFT) this morning: that starting QBs missed only 35 games in 2016, vs. 76/77/59 in 2013-15, respectively. The claim is sourced to Peter King’s MMQB column this morning, but unfortunately I haven’t been able to find a source document for the actual numbers. Per King, the NFL’s Competition Committee views these data as evidence the new QB protection rules are having their intended effect.

Now that seemed like a huge drop to me. I did some quick calculations – Teddy Bridgewater (16 games) plus Tony Romo (10 games) plus Jay Cutler (11 games) alone gets us to 37 games missed by starting QBs in 2016. That’s already more than the number reported by King.

Epidemiologists spend a large chunk of our time just counting things. As it turns out, that’s not grade school math. It’s really, really hard, and a correct count relies on counting a.) the right things and b.) doing so in a consistent manner. So I wanted to go into my injury data, sourced from Pro-Football-Reference (, and see if I could replicate the numbers reported by King (and, ostensibly, the NFL). Spoiler alert: I couldn’t really.

Continue reading “Checking a Reported Drop in Starting QB Injuries in 2016: Definitions and Counts”