The BAINISTEOIR system

The rating system has been a fixture since the beginning of this website. However, it had several limitations; for example, the database it used only went back to 2012, it didn’t factor in score margins, and it failed to adjust for the extra uncertainty following a long gap between matches. In short, changes needed to be made, from this simple, slightly modified Elo system, to something better. After lots of work, the new Take Your Datapoints rating system is finally ready to be revealed: BAINISTEOIR (BAINISTEOIR’s Acronym Is Needlessly Intricate; Statistically Trained Estimator Of Iománaíocht Ratings)

What’s new?

BAINISTEOIR is the new rating system for the site, and it involves a total overhaul of how each team is rated. The previous iteration of the rating system only checked whether a team had won, lost or drawn, the rating of its opponent, and whether they were playing at home, away or at a neutral venue. How much weight it gave to both these variables into its equation for a new rating was based, for the most part, on what ‘felt right’.

Like the previous version, BAINISTEOIR factors in whether a team won, lost or drew, how good its opponent was, and where they were playing. However, it also factors in the expected scoring margin versus the actual result, and how long it’s been since the teams played. Additionally, it will revert a team’s score towards the mean for its tier at the end of each year, in order to prevent teams from maintaining excessively high ratings from past performances, thereby getting more accurate ratings for the current moment.All these factors were weighted through the use of a genetic algorithm, to ensure that it produces the most accurate results possible.

The most important change, however, was to the database.

The new database

The old database was fairly comprehensive for the time period it covered, but that time period was extremely limited. The new database contains records of matches going all the way back to 1887, the year of the first All-Ireland championship. This includes games from every division of the league, Munster, Leinster, Ulster (including the Ulster shield), and the All-Ireland, as well as the now defunct All-Ireland B Championship, Connacht championship and the Oireachtas cup. It includes not only every currently active team, but also the records for teams that competed in the past, such as New York or Glasgow. While the old database covered roughly 800 matches, this new version contains, at the time of writing, 5,705 games, taken from a variety of sources. This includes the date of the match, the teams involved, whether the match was at one team’s home grounds or at a neutral venue (if known), and, of course, the score. As well as improving the rating system, it is hoped that this new database will make a variety of future articles possible, as there is now much better access to historical data.

Unfortunately, one drawback is that the database is not complete for all of these years: While it’s easy to find comprehensive records for recent games, you only need to go back a few years before it becomes difficult to find the results for lower tier games, such as the lower divisions of the league. Before long, even more significant games begin to go missing. The chart below indicates how many records there are for each year, and it’s easy to see how difficult it becomes to find details of older matches, especially once you get any further back than the 70s.

This, combined with the smaller number of inter-county games per year, means that ratings are often closer in the early decades than in more recent years. This means that a team who wins an All-Ireland in recent years is more likely to have a better rating than one who did the same thing in the 50's. At the very least, attempts were made to include the results of finals in a given year. However, many early games are extremely difficult to pin down: even if a score can be found, sometimes the exact date, or venue, can’t be. As such, I’d like to remind any readers who may have corrections, additions or suggestions for the database that they can get in touch at mail@takeyourdatapoints.com

Home advantage

As with the old version of the rating system, teams have additional ratings for how they perform at home, away or in neutral venues. Though this improved the ratings, some teams ended up with certain ratings getting very extreme values. Galway, for example, played very few home games, due to playing in Leinster. As such, their neutral rating got much larger than it perhaps should have. Though they performed well on neutral venues, especially this year, the system was often overconfident in their abilities. You also ended up with teams, typically those playing against either much stronger or weaker opponents, who would end up with positive or negative ratings in each category. This went against the aim of the home advantage system, as it was only meant to account for their relative improvement or disimprovement based on where they play. As such, the new system has modified this, so the home/away/neutral ratings are relative to one another. An improvement in one will result in a disimprovement to the others, and the sum of all three should add up to zero. Therefore, a team that wins or loses all the time, regardless of venue, will have all three ratings close to zero. A team with a runaway home rating now actually means what it’s meant to: this team wins more at home than would be expected, and loses other games more than would be expected.

Scoring margins

A previous article on this website regarding scoring margins finished with inconclusive results. It was found at the time that there was no definite correlation between a team’s winning margin and how much better they were than their opponents. However, with the massively expanded match database, a more definite trend could be found. This investigation also took into account how scoring itself has changed, as the value of a goal varied in the early years of the sport.

On average, when the system predicted the expected odds for a team, for roughly every 3-4 percentage points above 50%, a team would increase their winning margin by a point. Obviously, there is a lot of variation in this, but overall the correlation was found to be strong enough that it was included in BAINISTEOIR. If a team had a better winning margin than expected, they were given a slight boost in their rating, relative to how much they exceeded expectation (or, for a losing team that kept the game surprisingly close, their rating wasn’t reduced by as much as normal). Similarly, a team that wins by less than expected will have a smaller than normal rating increase, and a team than loses by more than expected will see a bigger hit to its rating.

As well as improving the predictive ability of the ratings, this change has other, interesting implications: rather than just giving odds of how much a team will win by, BAINISTEOIR will also be able to provide a projected point spread for a given match, as well as a better idea of the likelihood of a draw. There may also be the potential for live updates on a team’s odds of winning as a match progresses.

Rating tiers

An issue faced by the old rating system was how slowly it would adjust to a team improving or disimproving, especially if their previous rating was very high or low. Kilkenny, for example, performed very poorly this year, exiting the league at the quarter final stage and the All-Ireland in round 2, and netted a losing record over the course of the year. However, the old rating system still had them as the second best team around. A solution to this, adopted by BAINISTEOIR, and which is already used by many other Elo-based systems, is to slightly revert scores back to the mean at the end of the year. This system also reduces the inflation or deflation of ratings suffered by the system as teams are added or removed. Each team has its rating slightly increased or decreased towards a mean value before its first match each year. This reflects the increased uncertainty created by how a team has changed in the off-season. Though teams should still have the same ranking, their odds of winning or losing against other teams will decrease. This has the added benefit of kick-starting the system in adapting to a team’s change in rating.

Hurling presented an extra issue for this, however: there is a huge gap between those at the top and the bottom. Having every team revert to the same value resulted in the rating system giving odds which were too high for very poor teams, or too low for very good ones. There is less of a steady gradient in quality in hurling than many other sports: currently, any of the top 8-10 teams could beat one another on a given day, but would almost never lose to anybody below that range. Similarly, the next 8-10 teams would be competitive within themselves, but almost never lose against those below them. In order to reflect this, it was decided that each team would revert to a score within its own tier.

Tier one teams are those who most recently played in division 1A or 1B of the league, the All-Ireland, Leinster, Munster or, when it was played, the Oireachtas cup. Their scores trend towards a rating of 1750. Examples include Galway, Wexford, Limerick and Offaly.

Tier two teams are those who most recently played in division 2A or 2B of the league, the Christy Ring championship, the Ulster championship or, when they was played, the Connacht championship and All-Ireland B championship. Their scores trend towards a rating of 1500. Examples include Antrim and Wicklow (though Antrim will begin playing in tier one in 2018 with their promotion to division 1B of the league).

Tier three teams are those who most recently played in division 3A or 3B of the league, the Nicky Rackard championship or the Ulster Shield. Examples include Tyrone and Louth.

Tier four teams are those who most recently played in division 4 of the league (when it existed) or the Lory Meagher championship. Their scores trend towards a rating of 1000. Examples include Cavan and Lancashire.

The average team from one tier would be expected to beat the average team from the tier below more than 80% of the time. They would be expected to beat a team two tiers below roughly 95% of the time, and a tier one team should beat a tier four team almost 99% of the time.

This tiered system is an attempt to strike a balance between preventing teams’ scores getting too extreme or having them all trend towards unrealistically similar values. It also sets up the system to work more on a system of ‘how good are these teams relative to their peers?’, rather than ‘how good are these teams relative to all-time?’. It was felt that this was important, as hurling has become such a different sport over the decades, that trying to compare present teams to those of the past would be futile.

One further question is what to do about teams who haven’t played in a long time. The database has gaps, especially in the early years, and even recently teams like Cavan have re-entered senior hurling after several years away. If a team like Glasgow decided to compete again, should they have a tier one rating because their last recorded match was in the All-Ireland, even though it was more than 100 years ago? This would result in very inflated ratings for any them, or team who beat them, while their rating readjusted. To combat this, any team whose last fixture in the database was from more than a year ago has their rating reset. A team enters at the value for the tier of the competition it plays in, with no home, away or neutral advantage. Cavan, for example started playing again this year for the first time since 2012. When they stopped, they had a rating of 832. This year, upon entering the Lory Meagher competition, they had their rating reset back to 1000 before their opening match.

Rust

One final change was made to reflect a team’s rustiness. The GAA calendar is highly irregular, and teams can vary between playing week on week at the height of the league, to having month-long gaps in the later stages of the All-Ireland. These long gaps introduce uncertainty about how a team will perform, and so the predicted odds are narrowed the longer the teams have gone without playing. This adjustment is slight, and has upper and lower limits (in other words, a team that hasn’t played in two months has the same adjustment as one who hasn’t played for three), but it helps to prevent the system getting overly confident, or overly pessimistic, about teams who haven’t played in a long time.

New presentation

Finally, the important business: how will the website change? The Ratings section of the site has undergone an overhaul. As before, you can see a table of how each team is rated, and their odds for the current league or championship, once those get underway. Another feature which will be available when the new season starts is a page on upcoming match predictions, giving odds for each team to win, and by how much. In order to make use of the expanded database, historical ratings can also now be viewed for each team by clicking on them in the rating table. You can see the highs and lows of every team, when they did or didn’t win their leagues or All-Ireland championships, and see which teams under- or over-performed in terms of silverware. Hopefully you’ll enjoy exploring these new features, but here are some interesting points related to the rating histories to get you started:

Who is the best of all time? This is obviously subjective, but for BAINISTEOIR’s money, it’s going to stick with the 2009 Kilkenny team, with their rating of 2271 on 6th-September. It’s important to remember that BAINISTEOIR works better as a relative rating against other active teams, and so better teams may have existed in more competitive eras. Teams in previous eras often didn’t have as many matches to allow their ratings to increase the way that Kilkenny did in this span. Still, following their fourth All-Ireland in a row, and their seventh of the decade, in the same year that they won the league, there’s definitely a strong argument to be made here for Kilkenny in 2009 being the best ever. Cork supporters will, of course, want to highlight their own four in a row…

Though not as highly rated as Kilkenny’s, this is still Cork’s second highest rating of all time, with a total score of 2095 after their win in 1944. This rating should possibly be higher, but suffers from the hurling league not being played in any of those four seasons, giving them fewer games to show off their abilities. Their highest rating ever, a slightly higher value of 2099 in 1954, occurred following their third All-Ireland in a row, but they also had a league win in 1953 to help pad out their score.

You may be curious about whether or not we’re witnessing a golden age for any teams right now: According to the system, this year was Galway’s best ever, with their rating of 2132. Following the All-Ireland win, they narrowly beat out their former peak of 2113, achieved during their league win in 1989, following their two All-Ireland championships in a row. Whether this team goes on to create a dynasty, or this year goes down as an just a fleeting flash of brilliance, remains to be seen.

In contrast: which current top tier team set their peak score the longest time ago? Out of the teams who played in division one of the league, and also qualified for at least the first round of this year’s All-Ireland, Cork have gone the longest since their peak, the aforementioned rating of 2099 in 1954. This gap appears to be a result of just how good that team was, rather than how they’ve played since then, since Cork have won a massive 8 leagues and 11 All-Ireland championships in the meantime. Their greatest peak since the Christy Ring era occurred in 2005, with a score of 2084.

While many team’s peaks line up with their All-Ireland successes, it’s always interesting to see those great teams that never quite made it. A perfect example of this are Clare: Though they finally went on to end their championship drought in 1995, soon followed by their 1997 win, the rating system still considers their best team of all time to be their panel from the late 70's which won two leagues. After qualifying for the 1978 Munster final, Clare's rating reached an all-time high of 2069. However, they would go on to lose that final by a mere two points to the eventual champions Cork. This team is often brought up as an example when discussing how the backdoor might have changed things if it was around earlier, and with good reason. Even in recent years, Clare’s 2013 All-Ireland winning team or 2016 league winning team were not considered as good as their 2006 team, who reached league and championship semi-finals but got no further. Based on the rating system, there is an unusual trend for Clare where their most successful teams are almost never their best ones, but perhaps the ones with the best timing.

Many consider Waterford to be the most overdue team, going without an All-Ireland since 1959. Admittedly, a lot of Waterford teams in the meantime have simply been bad, but for the last 10 years or so, they’ve been consistently competitive. Since 2007, they’ve won 2 leagues, reached two All-Ireland finals, set several new peak ratings, and maintained an average rating of 1965, more than enough to make them an All-Ireland contender. In fact, their average rating over the course of the decade from 2007-2017 is better than the best rating they had prior to that period! This even includes their two All-Ireland wins.

How good does a team have to be to win an All-Ireland? Every team’s rating coming into their winning All-Ireland final match was gathered, and compared. The average All-Ireland winner has a score of 1963 prior to the match, and the median rating of an All-Ireland winner is only slightly higher, at 1968. The highest ever was Kilkenny in 2009, with a pre-match rating of 2264, and the lowest rating was Galway in 1923, with a rating of only 1730; 20 below the tier one average! For reference, of the 2017 teams, only four finished the year with ratings above the median: Galway, Waterford, Tipperary and Kilkenny. Cork and Clare also had ratings roughly equal to the average.

Over time, however, the average rating of the All-Ireland winner has increased, and so it’s likely that modern teams should be held to a different, and higher, standard. Within the last 20 years, the average rating of an All-Ireland winner has been 2091, and the median has been even higher, at 2102. Of this year’s teams, only Galway, the actual winners, finished the season with a rating above the median. Tipperary, Waterford and Kilkenny all fell within a standard deviation of the average, but it’s clear that the bar has been raised in order to be a true All-Ireland contender. Offaly have the distinction of possessing the lowest rating to win an All-Ireland during this 20 year period, with a rating of only 1849 coming into the 1998 final. For reference, of the teams in 2017, this rating is closest to, but somewhat worse than, Limerick’s final rating for the year, who were knocked out in the first round of the All-Ireland. This was still a big improvement over their previous All-Ireland, however, as they entered the 1994 final with a rating of only 1790.

I’m sure that you’ll be able to find plenty more notable pieces of information browsing through the website’s new rating system. No new entries will be added to its database from now until the beginning of next year’s hurling league, though once the 2018 fixtures have been announced, then work will begin on adding predictions for that year’s competitions. Following the 2018 All-Ireland, it’s hoped that further updates to the algorithm, and possibly to the database, will be made, and the BAINISTEOIR system will continue to improve.

Link to the main rating system page