 |
| U.S Olympic Diving Team trials |
Statisticians who have delivered their own judgments on the state of Olympic judging have a troubling message for fans of such sports as gymnastics, diving, boxing and dressage: There is no surefire way to remove national bias from the results.
In gymnastics, the highest and the lowest score for execution are tossed out in attempt to weed out possibly biased appraisals. But if more than one score is biased upward, or downward, that won't do the trick. In diving, officials from competitors' countries if possible aren't assigned to the semi-final and final—but they can be in earlier rounds. In the Winter Olympics, figure skating tried obscuring which judge awarded which score, on the theory that this would thwart collusion by judges. One study found that increased bias.
Eric Zitzewitz, the Dartmouth College economist who wrote that paper, says it is difficult to design one system to prevent all forms of bad judging, be it corruption—suspected but never proven in figure-skating a decade ago—intentional bias or unconscious bias. "It's very hard to design a perfect system in a vacuum," he says. "It depends on the specific form of activity you're trying to prevent."
Some sports, such as gymnastics, use a "trimmed mean," in which the highest and lowest scores are discarded and the remaining ones averaged. The assumption is that the biased scores are likely to be extreme ones.
Warren Smith, a retired mathematician from Temple University, advocates the trimmed mean for judged sports, finding it superior to some other Olympic systems. But Yale University statistician John W. Emerson found a drawback to the approach when he studied the 2000 diving scores. One Chinese judge showed the least evidence of bias toward any particular country's competitors. If he rated one diver higher than another by a certain margin, that tended to be the consensus margin between those divers, regardless of uniform.
The problem is that he tended to award higher scores to everyone—either out of enthusiasm or a different sense of what constituted a perfect dive. That meant his scores were more likely to be tossed than those of judges who did appear to systematically favor one country's divers over another
"If you do an incorrect analysis, you think he's guilty as sin," Prof. Emerson says.
There is another reason not to merely toss outlying scores: Recent news from London suggests it doesn't suffice to prevent deliberate impropriety. To calculate the London Interbank Offered Rate—a benchmark short-term interest rate for about $10 trillion in loans—the British Bankers' Association asks banks for their rates then discards the 25% highest and 25% lowest entries before averaging the rest. Yet some banks are under investigation for allegedly trying to tamper with Libor.
In systems that involve tossing outliers, "throwing out the top, say, six will work if you only have six people trying to manipulate" the results, says Prof. Zitzewitz. "But the minute you have seven or eight or nine, it doesn't work."
While skulduggery is possible, the more realistic problem with Olympic judging is conscious or unconscious national bias. Prof. Emerson, in his 2009 study of diving results, found 10 of 17 judges displayed favoritism to their own country's divers at a level that had less than a 1% chance of arising by statistical fluke.
What this all adds up to in terms of gold, silver and bronze depends on the sport and the extent of bias. In many cases, says Prof. Emerson, the effect is "subtle." He found that in 2000 it mostly didn't affect diving medals, though in the women's 10-meter platform competition, the winner, American Laura Wilkinson, might have gotten a silver instead, and Chinese diver Li Na the gold, if judges' national preferences were removed.
Jungmin Lee, an economist at Sogang University in Seoul who has studied Olympic judging, suggests a fix beyond the scoring system: increasing the number of judges. That, he says, would remove the effect of any one judge and make results reflective of a broader base of opinion. The downside, Prof. Zitzewitz points out, is that it would increase cost and might reduce the average quality of judges.
Some sports take other approaches to ensure fairness. Dressage uses a blind draw of judges and allows only one per country for the Olympics, says a spokeswoman for the Fédération Equestre Internationale, the international governing body for Olympic equestrian events. Dressage and boxing also show all spectators all judges' scores.
Indeed, the researchers say the best way to avoid judging problems is transparency—releasing results that include which judges gave what scores to which athletes. Whether this happens varies by sport, says International Olympic Committee spokeswoman Sandrine Tonge. Even when full scores are released, they sometimes are available only to accredited media.
"It's naive to think there's a perfect system out there that is robust to manipulation," Prof. Zitzewitz says. "The great thing about transparency is, it helps you detect any form of manipulation."
Edited By Cen Fox Post Team
