Q: I don’t understand why doctors can’t just tell patients the statistical conclusions of research studies to help decide what treatment is best for them.
A: Mark Twain famously said “there are three kinds of lies: lies, damned lies and statistics.” He was trying to point out that statistics can be taken out of context or misinterpreted, and if they are they can seem to support something which they actually do not.
There are many ways that medical studies and statistics may be misinterpreted.
The details of a clinical study must be understood in order to correctly interpret and apply the results. My favorite study to illustrate this point is one published in the digital version of the British Medical Journal evaluating the efficacy of parachutes. In this study 92 volunteers were randomized to having a functioning parachute (23 subjects) or a backpack with no parachute in it (69 subjects). All of the volunteer subjects jumped out of either an airplane or a helicopter. The results showed no survival benefit for having a functional parachute. If you were to stop reading at this point you would be very confused and surprised by the conclusion that “parachutes don’t save people who fall out of planes.” However, as it would be unethical to have people jump out of an airplane or helicopter high up in the air using a placebo parachute (this would not end well for the placebo study subjects), all the volunteers jumped out of the helicopter or plane while it was on the ground (so had about a three foot fall when they jumped).
The final example I will include today has to do with the ‘power’ of large numbers. If the sample size is very large, small(ish) random chance may end up looking like a ‘significant’ result (significant is defined as “the probability of the finding not being the result of chance”). When we analyze data sets we often use statistics to determine a ‘p-value’, which is a mathematically calculated number that is meant to test whether something is due to chance or not. If the sample sizes are very large, the calculated p-value can be misleading. Consider tossing a coin and seeing whether it ends up heads or tails. If we toss the coin 10,000 times, I do not think anyone would be surprised if we got 5070 heads and 4930 tails (basically 70 extra heads results out of the 10,000 tosses, so less than one percent extra heads). If a second coin had 4930 heads and 5070 tails, and we calculated the ‘p-value’ (using a 2 by 2 contingency table and calculating using the two tailed chi-square method with a Yate’s correction) we would find p equals 0.049, a value that a clinician usually interprets as ‘significant.’ Now I know the statistics experts out there are shaking their heads (pun intended) at this foolish approach, however, this is the kind of misapplication of statistics that too often occurs (bringing us full circle to Mark Twain’s quote above).
The take home message from all of this is that medicine is not an exact science; interpreting study results and statistics and applying them to a specific patient must be done with great care. I hope that together with last week’s column readers can see that there is a lot more to medical decision making than just quoting statistics from clinical studies!
Jeff Hersh, Ph.D., M.D., can be reached at DrHersh@juno.com