Why four judges give four scores, and how to measure it
The pattern ends. The four umpires raise their scores: 9.6, 9.2, 9.4 and 8.8. All four watched the same performance, from the same table, under the same rules. Nobody cheated. And yet the result depends on which umpires that competitor happened to get.
This is not an anecdote from your local tournament. It happens in every sport judged by a panel, and in martial arts it has been measured.
What the data say
In 2023 a group of researchers from TU Wien and from universities in Turkey published a study with 16 kickboxing referees. They showed them bouts on video in a training tool called JudgED and compared every decision against the correct one. They then set those results against the real decisions of the 2021 WAKO World Championships.
- Average accuracy was 43 %: fewer than half of the decisions matched the correct one.
- Agreement between referees was low, with a Fleiss' kappa between 0.37 and 0.40.
- The average reaction time was one second. A whole combination fits in one second.
- The authors' conclusion: more training is needed to improve decision accuracy.
Fleiss' kappa measures how much several raters agree beyond what they would agree on by chance. Zero is pure chance, one is perfect agreement. A 0.4 is usually read as barely moderate agreement. This is not a panel that fails now and then: it is a panel that, in a good share of decisions, is not applying the same criterion.
Where the disagreement comes from
Hardly ever from not knowing the rules. The causes are more uncomfortable, because reading does not fix them:
The angle
Each umpire watches from a different place. In sparring, a kick that lands clean on the trunk seen from one corner can land on the arm seen from the other. In patterns, the foot returning to the starting spot looks different from the centre of the table than from one end.
Speed and attention
A pattern umpire has to watch the stance, the height, the wave, the breathing, the hand that is not striking and the foot that pivots, all at once and in every movement. Nobody sees all of it. Each umpire sees a sample, and four umpires' samples are not the same.
The threshold
This is the most important one. Two umpires see the same hesitation. One deducts it and the other does not, because for them it did not last "more than a split second". Both know the rule. What differs is where they draw the line. That personal threshold is shaped by each umpire's experience, and nobody sees it, not even the umpire, until they compare.
Fatigue and drift
The umpire of the first bracket in the morning is not the umpire of the last one in the afternoon. Thresholds move over the day without anyone deciding to move them.
The rule itself
"More than a split second", "lack of power", "dynamic", "controlled on the target". Every rulebook in a judged sport has areas that depend on judgement. They are there on purpose: you cannot write a rule for every case. But each one is a place where the panel can split.
Bias and noise: two different problems
When an umpire drifts from the panel there can be two reasons, and it pays to separate them:
- Bias: they always drift in the same direction. The umpire who systematically gives 0.2 less than everyone else. Easy to correct once seen, because it is stable.
- Noise: they drift in different directions each time. Stricter today, more lenient tomorrow, with no pattern. More serious, because no adjustment fixes it: the judgement itself has to be trained.
An umpire with bias and no noise is almost as useful as a perfect one. An umpire with no bias and a lot of noise is right "on average" and wrong in every bout.
The rules already count on disagreement
Look at how a pattern or a bout is decided in ITF Taekwon-Do: the highest average does not win. Each umpire votes, and the winner is the one with the majority and at least two votes. The system is designed so that one umpire's disagreement does not decide alone. It is a sensible defence, but it is a defence: it does not reduce disagreement, it dilutes it.
How to measure a panel's consistency
Measuring is simple to explain and hard to do without tools:
- Everyone scores the same thing. The same videos, in one pass, no pause or rewind, without talking to each other.
- There is a reference. It can be an expert judge's scoring, recorded in sync with the video, or the median of many judges. The median and not the mean: one judge who goes far off moves the mean and barely moves the median.
- Each judge's difference from the reference is worked out, on every video, and turned into a precision.
- It is repeated over time. One measurement is a snapshot. Several are a trend, and they separate bias from noise.
- Look at where, not only how much. If the clicks are recorded against the video's time, you know at which second each judge split off. That is where the useful conversation is.
What changes when it is measured
An umpire who gets their difference after every video learns something no course tells them: which way they go wrong. If they always give more than the panel, they know what to look for. If sometimes more and sometimes less, they know their problem is attention or threshold, not the rules.
For a panel, measuring means being able to say something almost no federation can say today: how much its umpires agree, and whether that figure improves from one year to the next.
Frequently asked questions
Why do martial arts judges give different scores?
Because of the angle they watch from, the number of things to see at once, the personal threshold they use on borderline cases, and fatigue over the day. It is hardly ever because they do not know the rules.
What is Fleiss' kappa?
A statistical measure of how much several raters agree beyond chance. Zero is pure chance and one is perfect agreement. In the study with kickboxing referees it was 0.37 to 0.40.
Is it better to compare against the median or the mean?
Against the median. One judge who drifts far moves the whole panel's mean and barely moves the median, so the median is a steadier reference.
How do I know whether I am a strict or a lenient judge?
By scoring the same videos as other judges and looking at your difference from the reference many times. If you always drift the same way, that is bias. If it changes sides, that is noise.
Sources
- Hoelbling, D. et al. (2023). JudgED: Comparison between Kickboxing Referee Performance at a Novel Serious Game for Judging Improvement and at World Championships. Applied Sciences, 13(17), 9549.
- ITF, Official ITF Rules of Competition, version 2026-1: decision by majority of the umpires' votes.