What judge calibration is and why your federation needs it

A scale is calibrated by placing a known weight on it and seeing what it reads. If it reads high, it is adjusted. Nobody trusts a scale that was never compared with anything.

Calibrating judges is the same. You put in front of them a performance whose correct score is known, measure how far each one is from it, and correct it with practice. You repeat until the panel reads the same, and measure again to check it still does.

What it is and what it is not

Judge calibration is measuring the distance between what each judge scores and a reference, on the same performances, and using that measurement to train.

It is not the same as the things it gets confused with:

• It is not a rules exam. A judge can answer a quiz perfectly and score 0.6 below the panel. The exam measures what they know. Calibration measures what they do.
• It is not a course. A course teaches. Calibration checks whether what was taught reached the table, and it happens many times, not once every two years.
• It is not a tournament performance review. At a tournament there is no reference: nobody knows what the correct score was. In a calibration there is, because the material was chosen in advance.

Why a federation needs it

Because disagreement exists and has been measured

In a 2023 study with kickboxing referees, average accuracy against the correct decision was 43 % and agreement between referees was low. The authors concluded that more training is needed. We cover it in Why four judges give four scores.

Because training today ends at the course

A judge's usual path is a course every one or two years, then tournaments. Between one course and the next, nobody measures anything. If a judge has grown more lenient, it is discovered on a podium.

Because competitors' trust depends on it

A competitor accepts losing. What they do not accept is losing because they drew the wrong corner. A federation that can show how much its judges agree has an answer to that complaint. One that cannot only has its word.

Because someone has to choose who to send

When a federation puts judges forward for an international event, it chooses. Today that choice is made by seniority, degree and reputation. With calibration, it can also be made by measured precision.

How to build a calibration programme

1. A library of references

Videos of real performances, each with its reference score. The reference can be:

• An expert judge's scoring, recorded in sync with the video, tap by tap. The most demanding, and the most useful for training.
• The median of a senior panel, when you want a reference that does not depend on one person.

The library should have a bit of everything: clear performances, borderline ones, and videos chosen for one specific mistake, like the return to the starting spot or a kick that may or may not have landed.

2. The same conditions as at the table

One pass, no pause, no rewind and no talking. If judges practise with pauses, the measurement says nothing about the tournament.

3. One measurement per person

For each judge and each video: the difference from the reference and a precision. In patterns, also a pass tolerance, for example landing within 0.5 of the reference. In sparring, whether they picked the same winner as the reference.

4. The detail, not only the number

If taps are recorded against the video's time, you know at which second each judge split off. That shows whether it was a lapse of attention or of judgement.

5. Regular sessions, and one before every event

One calibration a year is a snapshot. One a month is a trend. And one the day before a tournament aligns the panel that is about to sit together. See how to align a panel of judges before a tournament.

6. History

The proof that a programme works is that each judge's deviation falls from one quarter to the next. Without history there is no proof.

What to ask of a calibration tool

• That it scores with the tournament's own rules, not an approximation.
• That it records every tap against time, not only the final score.
• That it does not allow pausing or rewinding while scoring.
• That it works on the phone the judge already has, ideally with nothing to install.
• That it allows a simultaneous session for the whole panel, in the same room.
• That it keeps a history per person and exports the results.
• That it covers more than one discipline, if the federation has more than one.

How Anchor does it

On Anchor's web platform an expert judge records their scoring in sync with the video, tap by tap, and each attempt by each judge is compared against that reference. The report gives precision, the winner picked against the reference's, pass or fail, and the expert's comments pinned to the exact second of the video.

Sessions are grouped into courses. A creator can restrict a course or a live class to a list of email addresses, so only the judges called up can join. In a live class the whole room scores at once from their phones, joining by QR code. Measured against the production server, 100 judges at two taps per second saw a p95 latency of 252 ms. Attendance, activity and results export to CSV.

ITF Taekwon-Do patterns and sparring come configured. Each discipline's rules are configuration, so karate, poomsae, kickboxing or boxing are added without programming.

Frequently asked questions

What is judge calibration?

Measuring how far each judge's score is from a known reference, on the same performances, and using that measurement to train until the panel scores alike.

How is it different from a judges' course?

A course teaches the rules and happens every one or two years. Calibration measures what each judge actually does and is repeated many times, including between one course and the next.

Who sets the reference score?

An expert judge who scores the video in sync, tap by tap, or the median of a panel of senior judges.

How often should a panel be calibrated?

Regularly, to see each judge's trend, and always the day before an important event, with the judges who will work together.

Does it work for martial arts other than Taekwon-Do?

Yes. Any discipline judged by a human panel can be calibrated: forms or sparring, karate, poomsae, kickboxing, boxing.