When clever people do stupid things - reprise
Prior to the recent post on devolution which touched off an absolute flurry of comment, the highest level of response I had to a post on this blog was to an item called "When clever people do stupid things" which pointed out that mistakes by clever people can do a lot more damage then less eminent individuals are ever given a chance to do.
This week the case which inspired that article was in the news again, and demonstrated the way that we are far too ready to tolerate an inability to understand numbers where we would never find acceptable equivalent consequences caused by, say, the inability to read.
My original article was inspired by Professor Sir Roy Meadow, who was one of the most eminent paediatricians in the country, but whose knowledge of statistics, and particularly of conditional probability, was so poor that it would have been disappointing in a V former reading Stats O level and grounds for disciplinary action in a VI former studying the statistics part of Maths A level.
Sir Roy gave evidence as an expert witness in the trial of a number of women whose children had died and who were accused of murdering them. Unfortunately, because of his enormous expertise in the area of child health, at least two juries accepted at face value statements which he made about the probability of a family losing two or more children to cot death which were provably nonsense. In consequence of Sir Roy's incompetence with statistics, at least two women who were almost certainly innocent were wrongly convicted of murder.
Sir Roy was the main culprit, but he was not the only one. The defence lawyers should have challenged his statistics. The jurors should have realised that his expertise in medicine did not guarantee expertise in maths. But above all, our society is too ready to both to tolerate bad statistics unless we have good reason to want to disbelieve them, and to reject good statistical data which does not fit our preconceptions. This particular case, where innocent women were sent to jail because of bad statistics, is an extreme one but it is far from being the only case.
Another example where medical experts were not able to give accurate advice about statistics, this time from the United States, is one of the many excellent examples given by Professor Joel Best in his superb book "More Damned Lies and Statistics." He quotes the figures for breast cancer screening among American women in their forties who had no other symptoms. About 0.8% of such women do in fact have breast cancer. If a woman has breast cancer, the probability that a mammogram result will be positive is 90%. If she does not, the probability is 93% that the mammagram will come out negative.
In a study 24 doctors were given these figures and asked, if a woman has a positive mammagram, what is the probability that she actually has breast cancer. Only 2 out of the 24 doctors got it right: two thirds of them were not just wrong, but way out.
(The correct answer, by the way, is 9%. If an American woman has a negative test result, it is very likely indeed that this result is correct. If the tests says that a woman does not have breast cancer, the chance that she really does have it is less than one in a thousand. However, if the test is positive the chances are almost ten to one that it is a false alarm. This does not mean that screening is not worth doing: it means that the correct response to a positive test is more tests, not the assumption that the test is correct.)
When it became clear that basic errors in statistics were resulting in innocent women being sent to jail for murders which never happened, the General Medical Council took disciplinary action against Sir Roy Meadow, and he was struck off the medical register.
Sir Roy appealed to the High Court, who reversed the decision, ruling that expert witnesses should be immune from disciplinary action, that Sir Roy's evidence did not constitute serious professional misconduct. This decision in turn was appealed to the Appeal Court, who came to a far more balanced decision this week.
The Appeal Court found that Sir Roy was undoubtedly guilty of some professional misconduct, but they did not consider that it was sufficiently serious for striking him off the medical register to be a proportionate penalty. This is an interesting view, because I am sure it caused him a lot less suffering than he caused the two women who were wrongly convicted of murder and sent to prison on the basis of his testimony. But I can accept that everyone makes mistakes, and since he was not the only person to blame for what happened he should not be the only scapegoat.
The Appeal court quashed, however, the ruling that expert witnesses had immunity from disciplinary action. I think the Appeal court was wise to do so. Everyone makes honest mistakes, but if someone is so careless of the truth as to be negligent, and that negligence causes a miscarriage of justice, they should be accountable for their actions.
And regardless of the question of whether Sir Roy should have been struck off, we really should learn lessons from this about the importance of training people to understand statistics. In law, for instance I am coming to the conclusion that passing a good stats course should be a mandatory requirement for barristers to qualify. And every jury room should have copies available to the jury of the following three books:
"How to lie with statistics" by Mel Calman
"Damned Lies and Statistics" by Joel Best
"More Damned Lies and Statistics" by Joel Best.
This week the case which inspired that article was in the news again, and demonstrated the way that we are far too ready to tolerate an inability to understand numbers where we would never find acceptable equivalent consequences caused by, say, the inability to read.
My original article was inspired by Professor Sir Roy Meadow, who was one of the most eminent paediatricians in the country, but whose knowledge of statistics, and particularly of conditional probability, was so poor that it would have been disappointing in a V former reading Stats O level and grounds for disciplinary action in a VI former studying the statistics part of Maths A level.
Sir Roy gave evidence as an expert witness in the trial of a number of women whose children had died and who were accused of murdering them. Unfortunately, because of his enormous expertise in the area of child health, at least two juries accepted at face value statements which he made about the probability of a family losing two or more children to cot death which were provably nonsense. In consequence of Sir Roy's incompetence with statistics, at least two women who were almost certainly innocent were wrongly convicted of murder.
Sir Roy was the main culprit, but he was not the only one. The defence lawyers should have challenged his statistics. The jurors should have realised that his expertise in medicine did not guarantee expertise in maths. But above all, our society is too ready to both to tolerate bad statistics unless we have good reason to want to disbelieve them, and to reject good statistical data which does not fit our preconceptions. This particular case, where innocent women were sent to jail because of bad statistics, is an extreme one but it is far from being the only case.
Another example where medical experts were not able to give accurate advice about statistics, this time from the United States, is one of the many excellent examples given by Professor Joel Best in his superb book "More Damned Lies and Statistics." He quotes the figures for breast cancer screening among American women in their forties who had no other symptoms. About 0.8% of such women do in fact have breast cancer. If a woman has breast cancer, the probability that a mammogram result will be positive is 90%. If she does not, the probability is 93% that the mammagram will come out negative.
In a study 24 doctors were given these figures and asked, if a woman has a positive mammagram, what is the probability that she actually has breast cancer. Only 2 out of the 24 doctors got it right: two thirds of them were not just wrong, but way out.
(The correct answer, by the way, is 9%. If an American woman has a negative test result, it is very likely indeed that this result is correct. If the tests says that a woman does not have breast cancer, the chance that she really does have it is less than one in a thousand. However, if the test is positive the chances are almost ten to one that it is a false alarm. This does not mean that screening is not worth doing: it means that the correct response to a positive test is more tests, not the assumption that the test is correct.)
When it became clear that basic errors in statistics were resulting in innocent women being sent to jail for murders which never happened, the General Medical Council took disciplinary action against Sir Roy Meadow, and he was struck off the medical register.
Sir Roy appealed to the High Court, who reversed the decision, ruling that expert witnesses should be immune from disciplinary action, that Sir Roy's evidence did not constitute serious professional misconduct. This decision in turn was appealed to the Appeal Court, who came to a far more balanced decision this week.
The Appeal Court found that Sir Roy was undoubtedly guilty of some professional misconduct, but they did not consider that it was sufficiently serious for striking him off the medical register to be a proportionate penalty. This is an interesting view, because I am sure it caused him a lot less suffering than he caused the two women who were wrongly convicted of murder and sent to prison on the basis of his testimony. But I can accept that everyone makes mistakes, and since he was not the only person to blame for what happened he should not be the only scapegoat.
The Appeal court quashed, however, the ruling that expert witnesses had immunity from disciplinary action. I think the Appeal court was wise to do so. Everyone makes honest mistakes, but if someone is so careless of the truth as to be negligent, and that negligence causes a miscarriage of justice, they should be accountable for their actions.
And regardless of the question of whether Sir Roy should have been struck off, we really should learn lessons from this about the importance of training people to understand statistics. In law, for instance I am coming to the conclusion that passing a good stats course should be a mandatory requirement for barristers to qualify. And every jury room should have copies available to the jury of the following three books:
"How to lie with statistics" by Mel Calman
"Damned Lies and Statistics" by Joel Best
"More Damned Lies and Statistics" by Joel Best.
Comments