COVID-19 and the problems of statistical comparison
One of the biggest growth industries of the past few weeks has been the publication of comparison graphs tracking the numbers of infections or deaths in various counties.
The ones which have some modest value are those which look at how the curves are shaped, particularly in countries where the infection is further along the track with a view to seeing where we are in the country where the analysis is being done and what might happen next.
The ones which are totally worthless are the graphs which attempt to make a political point that someone is doing well or badly and in my humble opinion this applies equally to those I have seen which attempt to prove that, say, the UK government is doing well and those which attempt to prove that Britain is doing badly.
It is way to early to prove in anything remotely resembling a scientific manner which countries have done well or badly and, frankly, we would be far better employed discussing what should be doing next than getting into a blame game.
There is certainly room for a debate about how we get to the point when we can relax the lockdown and what precautious we take in advance and during the relaxation to make sure that infections and deaths don't immediately start going through the roof again.
What we certainly don't have is sufficient hard information even about what is going on in our own country, never mind consistent information between countries, to do an analysis of relative performance which is worth the paper it would be printed on.
Many of the graphs I have seen shared on social media quote total number of deaths without even bothering to divide by the number of citizens in the country concerned. These are clearly worthless because it should be obvious that any given number of deaths represents a higher mortality rate in a country of ten million people than one of sixty million.
The UK government, and a number of other governments, have been publishing figures for the number of people who died in hospital after testing positive for COVID-19. This has the advantage that it is a stable benchmark, but the disadvantage that it excludes people who died in nursing homes or at home, especially if they have not been tested. This will distort the figures, and it will distort figures for different countries to a different degree depending on how effectively those countries are admitting people to hospital and what the testing regime is.
It doesn't end there. As the Economist wrote in an article on the subject,
"Critics have made comparisons between Britain and Ireland, which is seeing roughly half as many deaths per person and went into lockdown earlier. But deaths lag behind infections by several weeks, and the virus has been spreading for longer in Britain.
Travel may also play a part. As Keith Neal, an epidemiologist at the University of Nottingham, notes, international hubs are particularly vulnerable; London is home to a quarter of Britain’s cases.
Northern Ireland, where the lockdown followed the same path as in the rest of the United Kingdom, but which shares an island with the Republic, has seen a similar number of deaths per person as in the south.
Careful accounting will require adjustment for a huge range of things like the prevalence of comorbidities (Britain has high rates of obesity, for instance), geography (it is denser than most European countries) and demography (it is relatively young, and should thus fare better in this regard), and indeed for factors which affect the disease’s transmission but of which we are not yet aware.
It will also require more data on the number of additional deaths, not just those that have occurred directly as a result of the virus, as well as on the economic and social costs of the lockdown."
By "the number of additional deaths" The Economist is referring to the fact that total death rates in every country for which they can reliably be measured, not just the United Kingdom, are up by significantly more than the highest possible estimate of the number of people who have died directly from Coronavirus.
In countries (of which Britain is not one) where the health service has been completely overwhelmed, people may have been unable to get treatment for conditions other than COVID-19 where in the absence of the pandemic doctors and nurses would have been able to save them.
We in Britain appear to have the opposite problem: people are not going into hospital for other conditions because they are afraid of catching COVID-19 and consequently dying from those conditions when a hospital would have been able to prevent this.
It is worth mentioning that, although some "elective" operations and treatments have been postponed, Britain's NHS hospitals are still largely open to anyone who has an urgent medical need for treatment. Furthermore, to minimise the risk of people attending because of COVID-19 infecting patients presenting with other issues, NHS hospitals like the West Cumberland Hospital have separate entrances for Coronavirus and non-Coronavirus cases.
Instead of looking an death rates in other countries in an attempt to allocate praise or blame, it would be far more useful in terms of saving lives to look at other countries with the following two questions in mind:
1) What have other countries done which we are in a position to copy and which appears to have worked, which we might be able to do here and save lives?
2) What have other countries done which we are in a position to avoid and which appears to have gone wrong, so that we can try not to let the same things happen here?
The ones which have some modest value are those which look at how the curves are shaped, particularly in countries where the infection is further along the track with a view to seeing where we are in the country where the analysis is being done and what might happen next.
The ones which are totally worthless are the graphs which attempt to make a political point that someone is doing well or badly and in my humble opinion this applies equally to those I have seen which attempt to prove that, say, the UK government is doing well and those which attempt to prove that Britain is doing badly.
It is way to early to prove in anything remotely resembling a scientific manner which countries have done well or badly and, frankly, we would be far better employed discussing what should be doing next than getting into a blame game.
There is certainly room for a debate about how we get to the point when we can relax the lockdown and what precautious we take in advance and during the relaxation to make sure that infections and deaths don't immediately start going through the roof again.
What we certainly don't have is sufficient hard information even about what is going on in our own country, never mind consistent information between countries, to do an analysis of relative performance which is worth the paper it would be printed on.
Many of the graphs I have seen shared on social media quote total number of deaths without even bothering to divide by the number of citizens in the country concerned. These are clearly worthless because it should be obvious that any given number of deaths represents a higher mortality rate in a country of ten million people than one of sixty million.
The UK government, and a number of other governments, have been publishing figures for the number of people who died in hospital after testing positive for COVID-19. This has the advantage that it is a stable benchmark, but the disadvantage that it excludes people who died in nursing homes or at home, especially if they have not been tested. This will distort the figures, and it will distort figures for different countries to a different degree depending on how effectively those countries are admitting people to hospital and what the testing regime is.
It doesn't end there. As the Economist wrote in an article on the subject,
"Critics have made comparisons between Britain and Ireland, which is seeing roughly half as many deaths per person and went into lockdown earlier. But deaths lag behind infections by several weeks, and the virus has been spreading for longer in Britain.
Travel may also play a part. As Keith Neal, an epidemiologist at the University of Nottingham, notes, international hubs are particularly vulnerable; London is home to a quarter of Britain’s cases.
Northern Ireland, where the lockdown followed the same path as in the rest of the United Kingdom, but which shares an island with the Republic, has seen a similar number of deaths per person as in the south.
Careful accounting will require adjustment for a huge range of things like the prevalence of comorbidities (Britain has high rates of obesity, for instance), geography (it is denser than most European countries) and demography (it is relatively young, and should thus fare better in this regard), and indeed for factors which affect the disease’s transmission but of which we are not yet aware.
It will also require more data on the number of additional deaths, not just those that have occurred directly as a result of the virus, as well as on the economic and social costs of the lockdown."
By "the number of additional deaths" The Economist is referring to the fact that total death rates in every country for which they can reliably be measured, not just the United Kingdom, are up by significantly more than the highest possible estimate of the number of people who have died directly from Coronavirus.
In countries (of which Britain is not one) where the health service has been completely overwhelmed, people may have been unable to get treatment for conditions other than COVID-19 where in the absence of the pandemic doctors and nurses would have been able to save them.
We in Britain appear to have the opposite problem: people are not going into hospital for other conditions because they are afraid of catching COVID-19 and consequently dying from those conditions when a hospital would have been able to prevent this.
It is worth mentioning that, although some "elective" operations and treatments have been postponed, Britain's NHS hospitals are still largely open to anyone who has an urgent medical need for treatment. Furthermore, to minimise the risk of people attending because of COVID-19 infecting patients presenting with other issues, NHS hospitals like the West Cumberland Hospital have separate entrances for Coronavirus and non-Coronavirus cases.
Instead of looking an death rates in other countries in an attempt to allocate praise or blame, it would be far more useful in terms of saving lives to look at other countries with the following two questions in mind:
1) What have other countries done which we are in a position to copy and which appears to have worked, which we might be able to do here and save lives?
2) What have other countries done which we are in a position to avoid and which appears to have gone wrong, so that we can try not to let the same things happen here?
Comments