Diagnosing CTE in the Living: Proper Control Groups and the Importance of Your Research Question

A very interesting if preliminary paper looking at the possibility of using positron emission tomography (PET) scans (similar to a CT or MRI) to measure accumulations of a protein called tau in people’s brains was published this week in the New England Journal of Medicine.

Tau is important because its accumulation in the brain is the main way that chronic traumatic encephalopathy (CTE), a degenerative brain disease resulting from repetitive head trauma such as is experienced in contact sports like football and ice hockey, is diagnosed. CTE currently can only be diagnosed at autopsy, severely limiting research on the disease. A test for tau in the brains of the living would be a game-changer for CTE research.

A number of people, however, have been criticizing the paper for not including the right kinds of people. Specifically, while the study looked at ex-football players with cognitive and neuropsychiatric symptoms and healthy non-players, some seem confused or even suspicious about why it didn’t also include healthy ex-players? The short answer is because it shouldn’t have. Here’s why:

 

The Study Included the Right People for Its Research Question/Objective

The study’s goal was to see if a new test for tau works. That’s it. It did not, among other things, set out to:

-Prove a link between playing football and tau accumulation
-Prove a link between playing football and cognitive decline
-Prove a link between tau accumulation and cognitive decline

Oversimplifying, the study’s goal was to see if a new type of PET scan (a new radiotracer, actually) could be used to accurately measure the amount of tau accumulation in living brains. The thing is, before doing this study we have no idea if the scan works!

Thus we need to do the scan in brains where we expect high levels of tau and where we don’t expect high levels of tau and see if the scans’ results line up with what we expect. If they do, we can have some confidence the test might work. It’s not a guarantee – we won’t know unless we run the PET scan and then soon after do autopsies on the same brains – but it’s encouraging. If the results don’t line up with what we expect, it’s back to the drawing board.

So did the researchers do this correctly? Yes. They chose two groups: 1. Former football players experiencing cognitive or neuropsychiatric symptoms; 2. Similarly-aged men without any cognitive or neuropsychiatric symptoms or history of traumatic brain injuries.

Based on prior research we would expect the first group to have some of the highest tau levels, and the second group some of the lowest. I would argue maybe they shouldn’t have even age-matched, but honestly it’s probably fine.

They ran the tests in these two groups and found, indeed, that the new PET scan did show higher levels of tau in those expected to have higher levels of tau. Far from conclusive, but definitely encouraging.

 

Why Didn’t the Study Include Football Players WITHOUT Cognitive Impairment, or Cognitively-Impaired Non-Players?

This has been a surprisingly common criticism of this study. And the answer is really quite simple: because this wouldn’t have helped the study achieve its goal (or, in more technical terms, address its research question). In fact, it would’ve hurt the study. Consider the following grid:

2x2_Tau

The study wanted to compare people with expected high and low tau values. They chose the right groups! Including the other groups would have muddied the waters about whether the test worked or not.

Including the other groups would have made more sense for other research questions that this study didn’t address. For example:

Group I vs. Group II: Is the same amount of tau present in the brains of anybody exhibiting cognitive symptoms, or just those with a history of repetitive head trauma?

Group I vs. Group III: What are the differences between ex-football players who do and do not show later-life cognitive symptoms? For example, is there a difference in their tau levels?

Group II vs. Group IV: Among people without a history of repetitive head trauma, are tau amounts similar in those who do and do not exhibit later-life cognitive symptoms?

Groups I+III vs. Groups II+IV: Are later-life cognitive symptoms more common among those with a history of playing football than those with no history of repetitive head trauma?

Critically, again, this study was not designed to answer any of these questions, so these groups (II and III) would not have been appropriate.

 

Conclusion

A study needs to choose the right control group to accomplish its goal. This paper did so.

People clamoring for other control groups seem to be expecting this study to answer every question about the connections between football, tau, and CTE. Unfortunately that’s just not how science works. Ever.

If you want to say this study didn’t prove any links between football, tau, CTE, and cognitive symptoms, that’s fine! But that wasn’t its goal. And there’s plenty of other literature that does make the case for these various links (though the science is far from fully explored).

This study is only a small piece of the puzzle, but it used the right people to accomplish what it set out to do. Before criticizing studies I encourage you to first think carefully about what its goal was (this is typically found in the last sentence of its introduction, and hopefully in the abstract, too!). Then ask yourself, “Did it answer the question it set out to?”

Sorry if it didn’t answer the question you wanted it to, but that’s a different criticism entirely.

“Why Attack Football? Girls’ Soccer Has More Concussions!”: A Study in Bad Reasoning and Worse Stats

When I started in football injury epidemiology I focused on muscle, ligament, and bone issues. I shied away from brain injuries.

There were two reasons for this. First, I thought enough smart people were already working on brain injuries that my time was better spent elsewhere. Second, as a new entrant to the field I didn’t want to touch such a heated area. But since my article with Dr. Kathleen Bachynski estimating the prevalence of chronic traumatic encephalopathy (CTE) in NFL retirees appeared in Neurology last November I’m off the bench and into the game.

Seeing the backlash to those working to measure and quantify the short- and long-term impacts of brain trauma in football has been enlightening. Some of the points football’s “defenders” make have merit – we don’t know about the prevalence of CTE in high school or college players, for example, nor do we have a good handle on how many ex-players will show actual symptoms of CTE and other brain diseases rather than just the brain damage associated with them (although we do know that ex-NFL players die of neurodegenerative diseases such as Lou Gehrig’s disease at 3-4 times the rate of the general population).

But as the push to ban tackle football for kids 14 and younger gains steam, there’s one talking point from football’s “defenders” that is so callous in its logic and fallacious in its statistics that I feel compelled to call it out directly here.

 

Girls’ Soccer Has More Concussions, Why is No One Talking About Banning It!?

This argument is so bad we have to take it down in parts. First I’ll discuss its flawed logic and then its flawed statistics.

Logic

This is whataboutism at its finest. “Sure this thing is bad, but look at this other bad thing? What about that, huh? What do you have to say about that? HYPOCRITE!”

This is a weak, lazy, bad faith argument deployed strategically, often by powerful people, with the goal of stopping you from doing anything if you can’t fix everything. It is a common tool used by lobbyists and politicians to distract from their clients or donors who do bad things. You have to be on the lookout for it constantly as it can seem very persuasive, especially to good people who just want to help.

It turns out that in public health and many other fields we can walk and chew gum at the same time! Regulating full-contact football for children does not mean we can’t also recognize and address the fact that concussions are an issue in other sports, too. If anything, doing the former should help the latter, not hinder it.

Imagine another more obviously absurd example from environmental health: “Why is everybody so focused on lead, huh? What about arsenic? That’s poisonous, too.” The people making this argument are hoping that will lead you not to regulate either rather than both.

But maybe the people making the argument are really just invested in the brain health of young girls. How can we know they’re making this argument in bad faith? It’s simple: ask them whether they would support banning girls’ soccer for those younger than high school.

Statistics

Like all good lies, this one has a kernel of truth at its center that is then warped to the point of being unrecognizable.

The kernel: In sports played by both boys and girls, reported concussion rates are roughly 50-100% higher for girls. We’re not sure why – is it some sort of physical difference, or are girls just more culturally willing to report brain injuries? But it’s pretty clearly true. Football, however, is played (almost) exclusively by boys so this argument, even if true, is irrelevant.

Concussions also account for a larger percentage of injuries in girls’ soccer than in football. That does not, however, mean that there are more concussions in girls’ soccer than football – more on that below.

The warping: The above points are twisted in some motivated people’s minds to become that concussion rates are higher in some girls’ sports than football, and thus they are the real public health issue.

The truth: Below is a table of the annual number of concussions as well as the rate per 10,000 athlete-exposures (1 athlete participating in a game or practice is 1 athlete-exposure) for various men’s and women’s sports. The data come from two large nationwide sources for youth and college sports: High School Reporting Information Online (2017-18 report) and the NCAA’s Injury Surveillance System (two separate analyses from the 2009-10 to 2013-14 seasons and 2011-12 to 2014-15 seasons). I included data from two NCAA reports because the newer report did not include nationwide estimated concussion counts.

This is what happens when you actually dig into the data rather than parroting cheap talking points.

In high school, football has by far highest concussion rate at 9.70 per 10,000 AEs; second is girls’ soccer at 6.91 per 10,000 AEs, nearly 30% lower. Because of the number of children who play each sport, football is also responsible for nearly twice as many concussions each year as girls’ soccer (103,830 vs. 59,447).

In college, football (6.71-7.50 per 10,000 AEs) is essentially tied for the second-highest concussion rate. Wrestling leads the way (8.90-10.92 per 10,000 AEs). Football’s rates are in line with or a bit below those for men’s (7.40-7.91 per 10,000 AEs) and women’s ice hockey (7.50-7.80 per 10,000 AEs).

However, because more students play football than wrestle, football is responsible for by far the most collegiate sport concussions each year (3,417). Women’s soccer comes in second and is responsible for less than a third of football’s concussions (1,113).

Summary:

Football is either at (high school) or near (college) the top when it comes to concussion rates. Because of the number of people who play it, football causes far more concussions than any other sport among both high school and college athletes.

Stop using bad stats about girls’ soccer to distract from the concussion problem in football.

Article Review: Pitching a Complete Game and DL Risk

Despite the name of this blog being NFL injury analytics, since I’m now spending at least a third of my time working in baseball I’m going to start writing about that, too.

I wanted to start by discussing an article that came out a couple months ago in the Orthopedic Journal of Sports Medicine entitled “Relationship Between Pitching a Complete Game and Spending Time on the Disabled List for Major League Baseball Pitchers.

tl;dr: This article makes a claim that is completely unsupported by its analysis. Please don’t use it. Read on below for a deeper dive into (some of) the big reasons why.

Before I get into that, though, I just want to say that I’m only writing about this article in particular because it happened to slide across my Twitter timeline on a morning when I had some spare time and motivation to write about it. I do think the errors here are particularly egregious, but the same types of mistakes are extremely common in a lot of sports injury papers. I hate to feel like I’m singling out just one research group, so please understand that my criticisms will likely apply to a lot of what you read, not just these folks.

OK, onward!

Continue reading “Article Review: Pitching a Complete Game and DL Risk”

New Sports Performance Technologies: The 4 Key Questions Teams Need to Ask

I’ve had some thoughts on this topic for awhile, and then this article from a couple scientists with, I believe, the San Antonio Spurs came out in the Strength and Conditioning Journal. It does a great job at summarizing the key issues, but I wanted to try and distill it down to a 4-question framework that I use when evaluating a new sports performance technology for teams that I consult for.

So here are the four simple but broad questions that I ask with any new technology, in the order in which I ask them. I’ll use a single example of a hypothetical portable tool that measures forces acting on the shoulder joint of baseball pitchers to illustrate my four questions.

Continue reading “New Sports Performance Technologies: The 4 Key Questions Teams Need to Ask”

Just How Dangerous IS the NFL vs. Other Sports?

Turns out, yeah, pretty dangerous, actually.

As I’m working on my dissertation literature review I figured I’d put a small piece of it to good use for someone other than my committee. 4 people would normally read this, so I’m really hoping we can double that.

There is a wide-ranging belief that the NFL has the highest injury rates of the major North American sports, but by how much? Because game injury rates (rather than injury rates in practices) tend to be easier to calculate (and more available across studies) than those incorporating practices, we’ll focus exclusively on those.

We’ll need to introduce one concept to make the analyses below make sense: the idea of an “athlete-exposure” (AE). This is simply defined as 1 athlete participating in a practice or, in the case of this article, a game. Thus a single NFL game where every available player plays in the game would count for 92 athlete exposures – the 46 guys on the active game day roster on each of the two teams. All the injury rates below are presented per 1,000 AEs.

The studies compared below all have differences in how exactly injuries are defined (namely how much practice/competition time a player has to miss for something to count as an “injury”) and the years they covered, but in an effort to ensure high-quality data I’ve limited the studies to official league injury surveillance systems wherever possible.

Injury Rates Across Sports

Among the “Big 4” North American sports, the NFL has by far the highest game injury rate at 75.4 per 1,000 AEs. This means for every 1,000 players playing in a single game, 75.4 will suffer some sort of injury. So in a single game with 92 AEs, you might expect about 75.4 x 92/1,000 = ~7 injuries.

Compared to the other Big 4 North American sports, the NFL thus has roughly 4-5 times the in-game injury rate of the NBA, MLB, and NHL.

What about more international sports (MLS statistics weren’t readily available, so soccer goes here, stop complaining)? The NFL also has roughly 4 times the in-game injury rate of soccer and, interestingly, Australian rules football.

The only sport that comes close to touching the in-game injury rate of the NFL is rugby. The data are variable, but in general rugby seems to have either a similar or higher injury rate to the NFL. I wanted to look into concussions specifically in the NFL vs. rugby, but I’ve found some conflicting data and I want to do more research on this before saying anything.

I’ll spare you the gory details of the data sources unless you want to click below the jump, where I’ve included everything you should need to evaluate these on your own if you want!

UPDATE ~6 hours after original post:

I purposefully simplified or ignored quite a few issues in writing this post to try to communicate my main point, but some good questions from friends have come up and I want to briefly address a couple shortcomings of this analysis:

  1. This post only considers one possible way to measure injury rates: per player-game. The results would likely look substantially different if we used another, more precise denominator such as player-hours. The NHL numbers, in particular, would look higher relative to the NBA and MLB in such an analysis since hockey players on average play a lower proportion of each game than baseball or basketball players. So this is not the be-all end-all answer for which sport is the “most dangerous.”
  2. Along those same lines, although the risk of being hurt in any given NFL game is way higher athletes play way more games in the NBA, NHL, and MLB. So the differences across sports in an individual athlete’s risk of being injured in a season are going to be much less than the differences in injury rates I’ve shown here. Maybe I’ll take a stab at calculating risks in another post!

In a nutshell: this is just one way of measuring which sport is more “dangerous,” and it’s probably the way that casts the NFL in the most negative light. Injury rates per player-hour or 1-season risks would likely make football look less dramatically bad.

Continue reading “Just How Dangerous IS the NFL vs. Other Sports?”

Checking a Reported Drop in Starting QB Injuries in 2016: Definitions and Counts

I saw a remarkable stat reported in Pro Football Talk (PFT) this morning: that starting QBs missed only 35 games in 2016, vs. 76/77/59 in 2013-15, respectively. The claim is sourced to Peter King’s MMQB column this morning, but unfortunately I haven’t been able to find a source document for the actual numbers. Per King, the NFL’s Competition Committee views these data as evidence the new QB protection rules are having their intended effect.

Now that seemed like a huge drop to me. I did some quick calculations – Teddy Bridgewater (16 games) plus Tony Romo (10 games) plus Jay Cutler (11 games) alone gets us to 37 games missed by starting QBs in 2016. That’s already more than the number reported by King.

Epidemiologists spend a large chunk of our time just counting things. As it turns out, that’s not grade school math. It’s really, really hard, and a correct count relies on counting a.) the right things and b.) doing so in a consistent manner. So I wanted to go into my injury data, sourced from Pro-Football-Reference (pro-football-reference.com), and see if I could replicate the numbers reported by King (and, ostensibly, the NFL). Spoiler alert: I couldn’t really.

Continue reading “Checking a Reported Drop in Starting QB Injuries in 2016: Definitions and Counts”

The “Post-Probable” Injury Report Era: Full-Season Update

Hey, long time. Been awhile. How are the kids? Childish? That’s good.

I’ve been a bit distracted with side projects lately – buying a house, co-teaching a high-level statistics course, my dissertation…you know, little things – so sorry for not updating this blog that no one reads for a few months.

BUT! I’m back with a very exciting post: I’m updating my prior investigation into the effects of the NFL’s decision to remove “Probable” from its injury report this past season, now that we have a full season to see how teams adapted (the original analysis had only weeks 1-8). Let me tell you, it’s been miserable for NFL injury analysts and honestly…probably pretty much fine for everyone else.

Since my previous two posts lay out all the relevant background, methods, and data sources in detail, we’re gonna skip right to the results update!

Continue reading “The “Post-Probable” Injury Report Era: Full-Season Update”