A long read: The quality of umpiring has never been better...

The narrative of poor performance by professional umpires is unfair: they are more accurate and consistent than ever. Why has this narrative come about?

To follow me on Twitter, click here

Recent controversies and an unfair narrative

In recent weeks, there have been some high-profile criticisms of umpires in Test matches (such as the South Africa vs. Bangladesh series), in the IPL (culminating in the now infamous ‘no-ball’ that wasn’t, for which umpire Nitin Menon was unjustifiably vilified), and even in the County Championship, with the LV County Insurance Twitter feed running a few ‘would you give this out’? tweets to encourage some ‘debate’.

Click here to see the IPL 'no ball'

(N.B. this is a link to Youtube; I have no control over any adverts).

For what it’s worth, I think the Nitin Menon in the IPL example, was completely correct. As a reminder, the Law that deals with this is 41.7.1, which states:

Any delivery, which passes or would have passed, without pitching, above waist height of the striker standing upright at the popping crease, is unfair. Whenever such a delivery is bowled, the umpire shall call and signal No ball.

As I said when tweeting on this earlier:

• The batter is not standing upright

• The ball has still got some distance to travel to get to the batter

• Waist does not mean 'hip'

This sort of example is being used to create an unfair narrative of poor quality umpiring both because first, the original decisions are not (or not necessarily) incorrect and second, the wider lessons being drawn from a small pool of examples (even if the decisions were incorrect) are the wrong ones.

In fact, the quality of umpiring has arguably never been better. This long read will explore two broad areas of evidence for this (accuracy and consistency related to lbws) and then discuss why a narrative of poor umpiring has developed.

Accuracy: umpires are highly accurate decision makers

There is considerable evidence that professional cricket umpires make incredibly accurate decisions. Adie et al (2020) looked at 5578 leg before wicket (lbw) decisions made in elite level cricket between 2009 and 2016. They discovered that 98.08% of these lbw decisions made by umpires were correct. In addition, a further study by Adie et al (2022) using the same set of lbw decisions to determine whether umpires had expectation bias against lower order batters found that they did not. Although lower-order batters were more likely to be given out lbw, this was because they were more likely to actually be ‘out’. Whilst these are just two studies around one set of data (albeit large) pertaining to one mode of dismissal, it is not unreasonable to come to a conclusion that there is likely to be a similar degree of accuracy amongst professional umpires in other countries.

One interesting thing to come out of the first study (Adie et al, 2020) and, indeed the main focus of the investigation (and this could be something to explore in a further long-read) is that decisions were less accurate in T20 matches than one-day or first-class matches. Adie et al (2020) also found that umpires were more conservative in their decision-making in T20 cricket (i.e. inclined to give more ‘not out’ decisions for lbws) which is linked - interestingly - to the lower level of accuracy in this format of the game.

Nonetheless, Adie et al (2020) conclude that umpires are highly accurate lbw decision-makers.

Consistency: umpires are more consistent than ever

A strong case could be made that consistency of decision making also points to ‘better’ umpiring. We know, from DRS, that more balls go on to hit the wicket than would ever have been imagined 25 years ago. We therefore want today’s umpires to be significantly more likely to give a batter out lbw. And, in order for high-quality umpiring, we want umpires to be consistent in their (correct) approach to giving batters out lbw. A fascinating study by Douglas Miller, this time focusing on English first-class cricket (‘Analysis of lbw in the County Championship 1919-2010’) originally published in the March 2011 version of Cricket Statistician and now available on the Association of Cricket Statisticians and Historians’ excellent website, demonstrates that umpires are both significantly more likely to give a batter out lbw and that umpires in the English first-class game are much more consistent in their approach to decision making than they were even twenty years earlier (Miller, 2011).

Firstly, then, Miller’s research demonstrates that umpires in English professional cricket are more likely to give batters out lbw than before. The figures are fairly conclusive: between 1919 - 1930, lbws accounted for 11.14% of all dismissals, rising to 14.90% by 1981-1990. In the following two decades, there was an explosion of lbws, so that by 2001-2010 18.94% of dismissals were lbws (and, in fact, in the 2010 season as DRS was beginning to be more widely used, the figure was 22.08%) (Miller, 2011). (A very statistically unsound check of the latest round of County Championships indicates that 16.4% of dismissals were lbw, slightly down on those averages, although it should be said that there were an unusually number of high-scoring draws).

Secondly, and as importantly, umpires are considerably more consistent in their approach to lbws. Douglas Miller has worked out a methodology to demonstrate which umpires are more likely to give batters out lbw. The methodology is adjusted for the decade(s) in which umpires stood (that is to say that an umpire from 2010 cannot be compared in absolute terms to an umpire from 1919, because all umpires in 2010 are more likely to give a batsman out lbw than even the most ‘trigger happy’ umpire of 1919). Miller describes his method thus:

To define more clearly each umpire’s proclivity to give lbw decisions in favour of the bowler, two other factors should be brought to bear: each umpire’s statistics must be adjusted to reflect the overall average over the precise years in which he stood; and a crude but realistic assumption must be made that, over time, colleagues standing at the other end conformed to the average of the period in question. Taking note of these two factors it becomes possible to create an index for each umpire revealing the extent to which his inclination to give batsmen out lbw deviated from the norm of his time. (Miller, 2011: 10).

With 100 representing the norm for the period 1919-2010, the most ‘bowler friendly’ umpire as regards lbws was Harold Elliott, a first-class umpire from 1939 to 1956 with a score of 144, with nearly 15% of wickets falling lbw in the matches in which he umpired (very high for the time) and the most ‘batsman friendly’ umpire was Kevin Lyons, umpiring from 1985 to 2002, with a score of 51 and 12.66% of batters in his matches falling lbw. (Miller, 2011).

The article has many fascinating insights (and I will write a couple of shorter reads on it at a later date) but for our purposes here, Miller’s findings show that umpires have become much more consistent in their approach to lbws. He writes:

It is also noteworthy that there is greater uniformity between the umpires on the list today than would have been the case in earlier decades. The range of indices for those standing in 2010 stretches from [Neil] Mallender on 117 to Richard Illingworth on 86. By contrast, [Sam] Cook and [Dickie] Bird overlapped on the panel for 16 years, the one with an index of 139, the other on 52 [sic: the actual figure was 54]. The greater uniformity must be applauded: it speaks well for the ECB’s feedback and mentoring procedures (Miller, 2011: 15).

Miller is right: umpires are following the technology in giving more batters out lbw and are being more consistent in doing so, which can only point to better umpiring. Anecdotally too, of the current first-class umpires, it would be difficult to identify a Sam Cook and a Dickie Bird!

So… why the narrative of poor performance?

A large part of the narrative of poor performance must lie in the impact of technology. Ironically, the very thing that has played a central role in improving the accuracy and consistency of professional umpiring has also helped to create a narrative of poor performance. Now, every decision in Test matches is under scrutiny; every small umpiring mistake is magnified. For example, in the recent Third Test in the West Indies vs. England series, Joshua Da Silva was given out caught behind by umpire Joel Wilson, shortly after reaching his hundred. He reviewed the decision, simply because the review was available. He thought he had hit the ball, and began to walk off, very nearly reaching the boundary rope before DRS revealed that he had not, in fact, hit it. Twenty years ago, nobody (batter, fielder, umpire) would have known that an error had occurred. Even in, for example, the English County Championship where matches are now livestreamed (a recent innovation) fans are able to sit in the comfort of their own home and pass judgment on Twitter (especially when the controversial decisions are ‘clipped’ to encourage this).

Technology has also played a more subtle part in developing the narrative of poor performance: it has created the impression that all decisions are black and white with no room for grey areas; it is arguably this that has led to some of the furore involving some of those ‘second tier’ decisions around, for example, the waist-high no-ball in the recent IPL match. It is true that there have always been some modes of dismissal in cricket which are absolute: bowled, caught, run out, stumped and so-on are absolute. And so are some of the ‘second tier’ decisions (for example, the front-foot ‘no ball’). However, DRS has made other modes of dismissal which were previously grey areas (predominantly lbw) into absolutes: ball-tracking demonstrates that the ball is either going to hit the wicket, or it is not. Arguably, then, this has changed the way that the cricketer and the fan thinks about the game and they view umpires’ decision-making through the lens of certainty, where there is often none (as with the ‘no ball incident in the recent IPL game). Ironically, so ingrained is the lens of certainty, that even technology itself has fallen prey to the mode of thinking it has helped to create such as in the controversy over the Virat Kohli lbw, early on in this year’s IPL. Kohli was given out lbw and reviewed the decision, believing he had hit the ball before the ball hit his pad. The video evidence was inconclusive and, to much consternation from some fans, the original out decision was upheld.

Conclusion

This article has used lbws to make the general point that professional umpires are both highly accurate decision-makers, and more consistent than ever before and argued that, ironically, technology has played a large part in both the accuracy and consistency of those umpires but has also been a key player in shaping the narrative of unfair decision-making because it leads to greater scrutiny, and gives the false impression of certainty surrounding every dismissal. And so Nitin Menon, who (correctly in my opinion, albeit marginally) did not call a no-ball, has been a recent umpiring ‘victim’ of a need for certainty.

It may be that, in future DRS will be used in all games, thus largely removing any certainty. It may be that Law 41.7.1 can be reformed to reduce doubt (for example, replacing the word ‘waist’ with ‘hip’, a more obvious and consistent measure). Or it may be that we should celebrate the way in which umpires have responded to technology, changed their practice and are more accurate and consistent than ever. And, of course, in the process embrace the glorious uncertainty which makes sport the joy which it is.

References

(For the first two references, the abstract only was used).

Adie, Joshua & Renshaw, Ian & Polman, Remco & Thompson, Matthew & Mann, David. (2020). When in doubt, it’s not out: Match format is associated with differences in elite-level cricket umpires’ leg-before-wicket decisions. Psychology of Sport and Exercise. 51. 101760. 10.1016/j.psychsport.2020.101760.

Adie, Joshua & Renshaw, Ian & Polman, Remco & Mann, David. (2022). No expectation bias in elite-level cricket umpires’ leg-before-wicket (LBW) decisions according to batting order position. Psychology of Sport and Exercise. 60. 102145. 10.1016/j.psychsport.2022.102145.

Miller, Douglas. (2011). Leg before wicket. The Cricket Statistician 153: 6-11 Available from: https://archive.acscricket.com/journals/153/8/index.html (Accessed 30 April 2022).