How to Check Inter-Rater Reliability in Football Scouting Before Averaging the Grades

Inter-rater reliability concerns the consistency of assessments made by different observers. In football scouting, it asks whether scouts evaluating the same evidence apply a rating task consistently. A player record on RubiScore can establish match context, but agreement about a player's potential requires a separate examination of how the judgements were formed.

Public information at https://rubiscore.com can help align the player's identity, team and appearances where those records are available. It does not supply a scouting consensus. The workflow below is a practical proposal for comparing human assessments before compressing them into one apparently precise recruitment grade.

Define the decision before defining the rating

Begin with the question the scouting exercise is intended to answer. Suitability for an immediate first-team role, readiness for a development programme and long-term potential are different decisions. A scout answering one cannot be meaningfully compared with a colleague answering another.

Write the intended role and playing environment in ordinary language. A midfielder expected to receive under pressure faces a different assessment from one recruited mainly to protect space behind attacking teammates. Broad labels such as good midfielder leave too much room for private interpretation.

The time horizon matters as well. Present performance and projected development should be recorded separately. Otherwise, one scout may award a high grade for what the player does now while another gives the same grade for what the player might eventually learn.

Before collecting any ratings, ask each observer to restate the task. A disagreement at this stage is useful: it exposes an unclear brief before the player becomes the apparent source of the inconsistency.

Make the observation conditions comparable

Scouts may watch different matches, different minutes or different camera views. Those observations can all be valuable, but differences between their ratings do not necessarily measure inconsistency. The player may simply have faced different tasks in each observation.

For a small calibration exercise, give observers the same footage and the same assessment brief. Preserve the full relevant sequence rather than selecting only highlights. The exercise then asks how similarly the scouts interpret shared evidence, which is narrower than asking whether they agree after separate scouting trips.

Record what the footage does not show. A broadcast angle may conceal off-ball movement or the player's preparation before receiving. If an attribute cannot be observed, the report should allow that absence rather than force a grade.

This distinction separates an evidence problem from a judgement problem. Asking everyone to score an invisible behaviour will not produce useful reliability simply because the form contains a required field.

Replace broad adjectives with observable questions

Words such as intelligent, dynamic or composed can mean different things to different observers. Translate them into behaviours that can be located in a sequence. For receiving under pressure, examples might concern scanning, body orientation, first-touch direction and the availability of the next action.

The aim is not to reduce a player to a mechanical checklist. It is to make the reasoning behind a judgement inspectable. A scout can still offer an overall interpretation, but the reader should be able to see which observations support it.

FIFA Training Centre's research brief by Dennis Lüdin highlights the difficulty of achieving agreement between scouts and discusses shared selection criteria and independent assessments. That supports examining the assessment process rather than assuming several expert opinions will automatically converge. FIFA Training Centre: agreement between scouts.

The specific worksheet proposed here is an editorial application of that principle, not a validated FIFA assessment instrument. A club would need to test whether its chosen questions help with its own selection decisions.

Collect independent notes before the discussion

If scouts discuss a player before writing their assessments, the resulting agreement may partly reflect the conversation. A senior colleague's enthusiasm can become a reference point for everyone else. Independent first notes preserve the differences that the review is meant to examine.

Ask each observer to record the behaviour, the clip reference, the interpretation and their confidence. These fields separate what was seen from what was inferred. A confident conclusion based on limited evidence then looks different from a cautious conclusion supported by several examples.

Allow observers to mark an attribute as unobserved or uncertain. Treating those entries as poor performance would punish the player for a lack of opportunity in the selected footage. Treating them as average performance would create evidence that does not exist.

Keep the initial notes after the group meets. A revised assessment may be better informed, but replacing the original version removes the ability to understand how the discussion changed the judgement.

Separate disagreement about evidence from disagreement about value

When two scouts differ, begin by comparing their examples. Did both see the same action? Did one notice an earlier scan that the other missed? Did the camera angle make the situation ambiguous? These are questions about evidence.

Next compare the interpretation. Both scouts may agree that the player passed backwards under pressure, yet disagree about whether it was sensible. One may value security in that role; the other may believe a forward option was available. The disagreement now concerns the decision and the brief.

Finally, compare the scale. One scout may reserve the highest grade for exceptional evidence while another uses it for any clearly successful action. Averaging those grades creates a number without resolving what each grade means.

RubiScore can support the factual context of the observed appearance, but these differences belong to the scouting process. A match statistic cannot decide which of two privately defined grading scales is correct. The team must make the scale and role expectations explicit.

Inspect the pattern before calculating a summary

A simple review can start by listing where observers agree, where they differ and where evidence is missing. This often reveals whether one observer grades consistently higher, whether one attribute causes confusion or whether disagreement is concentrated around a particular role.

Do not assume that identical rankings mean identical ratings. Two scouts can place players in the same order while assigning very different grades. Conversely, similar average grades can conceal different ordering among the players who matter for a selection decision.

Formal reliability statistics may be appropriate, but the choice depends on the data and purpose: categories, ordered grades and numerical measurements pose different questions. The calculation should follow that design. A familiar coefficient is not a universal answer to every scouting form.

For a small internal exercise, transparent disagreement notes may be more useful than a headline reliability score presented without context. The practical issue is whether the differences would change a decision and what additional observation could resolve them.

Do not confuse agreement with accuracy

Scouts can agree and still be wrong. A shared assumption about body size, playing style or development can produce consistent assessments without producing sound forecasts. Reliability concerns consistency; whether the judgement is useful for the intended decision is another question.

This distinction is especially important for potential. A later career reflects coaching, opportunity, health, personal circumstances and many other influences. A selection outcome cannot be attributed solely to the original scout's insight or error.

Retain both the assessment and the subsequent opportunities when reviewing past decisions. A player who received little development support presents a different evaluation problem from one given sustained opportunities in the intended role. Neither case should be reduced to a simple hit-or-miss anecdote.

Likewise, disagreement can be informative. An observer who notices a less obvious strength may initially differ from colleagues. The purpose of review is to examine that evidence, not force consensus because a unanimous sheet looks orderly.

Close the review with a new observation task

The meeting should produce a decision about evidence, not merely a compromise grade. A useful closing note records:

• The assessment question and intended role.
• The footage or matches examined independently.
• The attributes with meaningful agreement.
• The specific observations behind unresolved differences.
• Any changes needed to the rating definitions.
• The next match situation or behaviour that needs to be observed.

For example, scouts may agree that a player receives comfortably without pressure but lack shared evidence about tighter situations. The next task is then to observe that demand, rather than debate a general label such as composure indefinitely.

Inter-rater reliability becomes useful when it improves the questions asked of the player. Preserve independent judgement, make the evidence visible and investigate disagreement before averaging it away. The final recruitment decision remains a judgement, but it rests on a process that others can understand and review.