EVIDENCE

Evidence: Interpreting the Science

‘The fact that an opinion has been widely held is no evidence whatever that it is not utterly absurd.’ Bertrand Russell

The heart of this blog is the effect of exercise (or lack of it) on health and disease, and the extent to which exercise can both prevent a number of diseases and also contribute to their treatment.  Although I do not trouble you with endless references, I try to stick to “evidence based” information. So I thought that you might be interested in evidence as applied to facts and beliefs in the medical setting. In the case of exercise and health, we will be talking about cause and effect.

Epidemiology

When it comes to exercise as a factor in health and disease, an importance source of evidence is found in epidemiology – sorry about the long word.

‘Epidemiology is the study and analysis of the patterns, causes, and effects of health and disease conditions in defined populations. It is the cornerstone of public health, and shapes policy decisions and evidence-based practice by identifying risk factors for disease and targets for preventive healthcare’.

OK, that’s pretty indigestible. In simple terms, epidemiology is the observation of associations between disease and certain behaviours, environmental conditions and physical states. For example, when Professors Richard Doll and Tim Peto from Oxford identified cigarettes as the main cause of lung cancer, they used epidemiological evidence – the observation that people who smoked cigarettes had a far higher incidence of lung cancer than those who did not. A lot of the evidence related to the effects of exercise are epidemiologically based; for instance the finding that people who take a lot of exercise are less likely to develop type 2 diabetes.

Both these examples demonstrate only associations, not proof of cause. Association is not the same as causation – though it may be! Take this example: during the period in the 20th century when the incidence of coronary disease was increasing rapidly, so was the use of the radio – but few people believe that coronary disease is caused by radio waves. In the case of smoking and lung cancer, however, there is no other reasonable explanation and there are good biological reasons to believe that smoking might cause lung disease.

In the case of exercise and diabetes, we cannot be quite so sure. There may be other ‘confounders’ – that is to say, other factors with which both exercise and diabetes are associated, such as sugar consumption. If exercisers consume less sugar than non-exercisers, that might be the explanation for the association. There are many differences between exercisers and non-exercisers and it would be impossible to find groups of each whose other characteristics were identical, so many of the associations that I will describe below are just that – associations with a very strong presumption of a causal link.

Clinical trials

Stronger evidence of causation is provided by the clinical trial. The purest form is the randomised, double-blind, controlled trial (RCT). This is best understood in the example of a drug trial. A group of individuals with a particular condition is selected to clarify the effect of the drug and they are divided randomly into two groups. One group receives the treatment and the other (the control group) receives a placebo – that is, an inert pill that looks the same as the real treatment. Neither the person giving the treatment and assessing its effect nor the patient receiving it knows who is getting the real treatment and who is getting the placebo. At the end of a predefined period the code is broken and the difference between the two groups analysed. If a trial of a cancer treatment is being tested, this might be the cure rate, and if the cure rate is significantly higher in the treatment group than in the control group it can be inferred that the drug is effective.

There are lots of possible pitfalls even in this very straightforward scenario. The patients might not have complied properly with the treatment – this is most likely for the treatment group if the drug has unpleasant side effects. The randomisation process might not be perfect and, despite the randomisation, there may be unexpected differences between the two groups. The statistical analysis has its own problems. In clinical trials statistical significance is reached if the chance of the difference found between the two groups is less than 1 in 20 (in statistical shorthand, p = <0.05). With a clearly effective treatment this can be derived from a trial with small numbers of patients. The less obvious the effect, the more patients will be needed in the trial to show it. The more patients needed in the trial to show an effective outcome, the less effective the drug must be. If the number needed is very great, a statistically significant effect might be clinically insignificant – not worth the candle. The counter-side to this is that small trials are more likely to produce erroneous results – the bigger the trial, the more reliable a test it is. Even apparently well-conducted RCTs can yield results with problems of interpretation. When such trials are carried out by pharmaceutical firms on their own products, they are far more likely to have a positive outcome than independently funded and conducted trials.

If you think that all that is bad enough, just consider the difficulties of carrying out randomised, controlled trials of exercise!  For a start, they cannot be double-blind – treatment group members know they are in the treatment group and it is quite hard to prevent the observers from knowing too. Compliance can be a nightmare and there may be cross-contamination – that is, members of the control group may start exercising, even if they have been asked not to. When I carried out a randomised, controlled, but not blinded, trial of exercise for patients following heart attacks, one of the non-exercise controls bought himself a static bicycle on his way home from his initial exercise test!

Meta-analysis

There is one more level of evidence which in some instances can provide the strongest evidence – the systematic review and meta-analysis. These are most useful when the numbers required to show an effect in an RCT are larger than can be reasonably recruited in a single trial. It is also an essential technique for unravelling the facts when different studies of the same topic come up with different answers. A systematic review answers a defined research question by collecting and summarising all the evidence collected through observation that fits pre-specified eligibility criteria. A meta-analysis uses statistical methods to summarise the results of these studies. In other words, a meta-analysis is a summation of the results of all the trials carried out using the particular intervention under investigation. The total number of subjects is much larger than in single trials and the results should therefore be that much more convincing.

Again, there are problems. Different trials are different in many aspects and combining them sometimes involves mixing oranges and pears. The choice of trials for inclusion is crucial if a particular question is to be answered – there must be as many similarities as possible. And there is the ever-present bias of the tendency to publish trials with a positive outcome but not those with a negative outcome.

Medicine is very far from being a perfect science – it is said that half of what doctors believe to be true today will, in time, be shown to be wrong. The trouble is that we do not know which half! Interpreting evidence is a huge problem, but it is at the heart of ‘evidence-based medicine’.

Next week I will tackle the presentation of evidence.