Goalie Performance
14 min read
This article consolidates five posts published between 2023 and 2024.
Starting with empirical Bayes
Recall from the introductory paragraph of the series on goalie consistency: “Goaltenders make up the least predictable position in hockey. Their behavior confounds analysts and casual fans alike. It isn’t uncommon for a good goalie to have a below replacement level year, or for an unknown goalie to come in and dominate the league for a stretch of time. This may partly explain the relative dearth of analysis on goalies - they’re voodoo, it’s often said.”
This analysis focuses on goalie performance. It aims to enhance the estimation of goalie performance using an empirical Bayes framework. Similar applications were outlined in previous papers. The strategy has several advantages over frequentist methods, chiefly the ability to measure uncertainty, which is crucial in describing goalie performance. The idea is first sketched using raw save percentage as the performance metric, then refined.
The empirical Bayes approach involves two steps. The first step is to use observed data to fit a prior distribution, and the second step is to update this prior using observed data.
Step One
Imagine a scenario where a new and unknown goalie emerges. What probability can we assign to their career Fenwick* save percentage being .930, .940, or .950?
Fenwick save percentage refers to the total percentage of unblocked shots saved, including shots that miss the net. It has been established that goalie skill correlates with the ability to make players miss the net.
To represent the possible career Fenwick save percentage for a new goalie, we can utilize the provided histogram of career 5v5 Fenwick save percentages for goalies who have faced 200+ shots (including those facing less adds noise - don’t worry, it’ll be addressed in a future post).
While the histogram provides a rough distribution, it contains random bumps throughout. To obtain a more structured representation, we fit a distribution to it.
The beta distribution is a commonly used prior when the variable of interest is a percentage, as in the case of the raw save percentage. Fitting a beta distribution to the career Fenwick save percentage distribution yields the following result.
The fit is… not really good, and alternative distributions such as gamma or Weibull will be explored in future posts to find the best fit, but it serves its purpose as an introduction to the framework. The beta distribution has two hyperparameters, alpha and beta, which can be interpreted as successes (saves) and failures (goals). The fitted beta distribution in this case has hyperparameters 852 and 55.6, indicating that we attribute 852 saves and 55.6 goals to a goalie before knowing anything about them. This corresponds to a .9388 Fenwick save percentage, or, a little below the median save percentage for goalies facing 200+ shots (.9406).
Step Two
The second step involves updating the prior distribution with each goalie’s career results. When a goalie has faced only a few shots, their estimated save percentage shrinks towards the mean of the prior distribution, while the uncertainty (represented by variance in the distribution) remains high. As a goalie faces more shots, the uncertainty of their estimated save percentage decreases.
Updating the prior with observed goalie results is relatively straightforward. It requires adding the observed successes and failures (saves and shots) to the hyperparameters of the beta distribution.
As an example, let’s plot the posterior distributions for two 24-year-old goalies: Jake Oettinger and Jeremy Swayman.
These posterior distributions offer interesting insights, like:
- There’s a 60.28% chance that Swayman’s save percentage is higher than Oettinger’s.
- There’s a 94.12% chance that Oettinger’s save percentage is higher than average.
- There’s a 93.81% chance that Swayman’s save percentage is higher than average.
- Oettinger’s distribution is tighter than Swayman’s because he’s faced more shots.
It’s important to list all the assumptions with this method:
- All shots are assumed to be equal.
- Age is assumed to be irrelevant.
- The prior distribution is assumed to be beta with hyperparameters 852 and 55.6.
- Goalies facing less than 200 shots are ignored.
- Goalie careers are equal.
- Scoring rates are assumed to be constant.
- Team systems are assumed to be identical.
These will be challenged and addressed below.
Code available here: https://github.com/spazznolo/goalie-performance/blob/main/posts/post-1.R
Accounting for shot quality
All shots are not equal, in that they do not have the same probability of becoming a goal. This is established. Many Expected Goals (xG) models have been developed to account for this. Fortunately, Peter Tanner’s website MoneyPuck provides detailed data on each unblocked shot in the NHL, including the probability of the shot being a goal according to his logistic regression model.
To adjust a goalie’s save percentage for shot quality, we can incorporate these expected goal probabilities from MoneyPuck. Usually, after accounting for shot quality, goalie performance is measured by the number of goals saved above expected (GSAx). This changes the metric from a rate (percentage of shots saved) which is bounded by 0 and 1 to one that can include any real number. Unfortunately, the beta distribution only works with rates. One approach to retain the metric as a rate, and thus the beta distribution as a prior, is as follows:
- Fenwick Save Percentage (FSV%) = 1 - (Goals Against / Fenwick Shots Against)
- Expected Fenwick Save Percentage (xFSV%) = 1 - (Expected Goals Against / Fenwick Shots Against)
- Median Save Percentage (MSV%) = Median of Goalie (20+ xG faced) Career Save Percentage
- Adjusted Save Percentage (AdjSV%) = MSV% + (FSV% - xFSV%)
Let’s plot the distribution of career AdjSV% for goalies who have faced 200+ shots. We will also include a fitted beta distribution in white, and a weibull in red.
At first glance, goalies’ career AdjSV% seems to follow a weibull distribution! Cool, but we’re going to sidestep this finding for now, because (hint, hint) there maybe be more than one distribution here. So we’re fitting another beta, which means the remaining steps stay the same. The prior is similar, except this time we add 933 saves and 60 goals (up from 852 and 55.6).
Let’s revisit the Jake Oettinger and Jeremy Swayman comparison.
The posteriors change as follows:
- There’s a 78.77% (previously 60.28%) chance that Swayman’s AdjSV% is better than Oettinger’s.
- There’s a 88.28% (previously 94.12%) chance that Oettinger’s AdjSV% is better than the MSV%.
- There’s a 97.30% (previously 93.81%) chance that Swayman’s AdjSV% is better than the MSV%.
These changes occur because Swayman faces more difficult shots on average, with an xFSV% of 94.07 compared to Oettinger’s 94.39.
Comparing the estimates
Below is a collection of plots which compare the save percentage metrics discussed above.
A couple of key points:
- Goalies who have a poor start to their career tend to play fewer games (surprise, surprise).
- The relationship between a goalie’s SV% and their AdjSV% seems to strengthen as they face more shots.
- A goalie’s AdjSV% converges with their posterior AdjSV% as they face more shots (indicated by the yellow diagonal line).
- Due to the previous points, there is heteroskedasticity in the relationship between a goalie’s SV% and their posterior AdjSV%.
- There is likely survivorship bias present.
Code available here: https://github.com/spazznolo/goalie-performance/blob/main/posts/post-2.R
Adjusting for age
Like shot quality, the effect of age on performance is a well-researched concept. It has been shown that goalies tend to improve as they age, peak, and then recede for, well, forever. This concept is typically called an age curve. Though goalie age curves are already available elsewhere, they differ slightly, so I’m going to define a custom age curve based on the Moneypuck dataset.
- Obtain goalie birth dates.
- Derive goalie age for each game.
- Define new analysis population (we won’t find every goalie’s date of birth).
- Explore age.
- Adjust for age.
Finding goalie birth dates
I scraped hockey-reference for each goalie’s date of birth (code available here). By combining the goalies’ birth dates with the dates of each of their games, I determined their exact age for every game played.
The analysis population changes as follows (due to incomplete linkage):
- Goalie population drops from 315 to 308.
- Harmonic mean of shots against rises from 12,690 to 12,721 (mean rises, 4,198 to 4,286).
- Harmonic mean of AdjSV% stays at .939 (mean rises, .932 to .933).
They barely differ.
Exploring age
Let’s start by simply grouping shots into bins by goalie age (rounded to the first decimal, ex: 26.0, 26.1, etc.), and then calculating the group-wide save percentage. Here’s what that looks like, with points becoming paler for smaller groups sizes:
Some thoughts:
- Most shots are taken on goalies aged 23-35.
- If you squint, you can see a little bit of an age curve here.
- This plot is riddled with bias (particularly for goalies with short or long careers).
Let’s fix some of the bias above with a few changes. We’ll follow a well-worn strategy seemingly developed by Tango Tiger here called the delta method.
Here are the steps:
- Take change in save percentage (dSV%) from each goalie’s age to the next.
- Take the harmonic mean of dSV% for each age as the the observed change in SV%.
- Clip off underrepresented ages (-21, 39+).
- Clip off the final year of each goalie’s career.
- Take the cumulative sum of dSV% throughout the retained age range.
Altogether, this gives us the curve below:
Some thoughts:
Applying the adjustment
The cleanest way that I can think of adjusting for age is to bake it into the current framework which already adjusts shots by their probability of becoming a goal. This can easily by done by first setting the peak (age 25) as the standard and then adjusting for all other ages, so that, for example, an xFSV% of 0.940 at age 25 is equivalent to an xFSV% of 0.93976 at age 22 (0.94000 - 0.000237) and an xFSV% of 0.93586 at age 38 (0.94000 - 0.00414).
- Adjusted (xG) Save Percentage (AdjSV%) = MSV% + (FSV% - xFSV%)
- Age Curve Adjustement (acAdj) = f(age), where f is the smoothed curve in the plot above.
- Adjusted (Age + xG) Save Percentage (AdjSV%) = MSV% + (FSV% - xFSV% + acAdj)
With this adjustment, Lundqvist’s career save percentages work out like this:
- SV%: 0.948638
- SQ AdjSV%: 0.947320
- Age SQ AdjSV%: 0.948619
- posterior SQ AdjSV%: 0.947082
- posterior Age SQ AdjSV%: 0.948338
Code available here: https://github.com/spazznolo/goalie-performance/blob/main/posts/post-3.R
Career length and the prior
The 200-shot cutoff
The 200 shot cut-off surprisingly filters out 90 out of 314 goalies (or, 28.7%). To illustrate this, here’s the cumulative distribution function of career shots faced for goalies:
This is a problem. One possible solution to this was proposed by David Robinson in his Baysian series on baseball. Instead of fitting a beta distribution to batting averages, he fit a negative binomial distribution using 1) at bats and 2) hits. This can easily be applied to goalies using 1) shots faced and 2) adjusted saves. It has its problems, though (which, of course, David addresses in a string of fantastic blog posts, eventually turning into a book). The main problem is that it introduces bias - goalies who perform well are likely to get more opportunities to play compared to those who perform poorly. As proof, here’s a plot showing the average AdjSV% by career seasons played:
Because of this bias, we’re faced with a compromise: Do we weigh the observations by shots faced, which will lean the analysis towards better performing goalies, or do we treat all goalie careers equally, losing nearly 30% of our population (albeit less than 1% of shots faced) at the same time? Or… is there an alternative?
David’s alternative was to fit a beta-binomial regression of batting averages on at-bats. Essentially, each at-bat total has its own prior. Though this is a good idea for his use case, it isn’t great for mine. The problem is that I want to build a decision making tool for goalies who are still playing, and we don’t know how many shots they will face in the future! This is the point of departure.
Combining Priors
To address these issues, the Bayesian framework can be expanded to include two priors, glued together by a probability. The goalie career AdjSV% density plots are revisited, this time splitting goalies into two groups: those facing -1,500 shots and those facing 1,500+. From these group distributions, two separate priors can be built, and they can be merged by including a probability of a goalie belonging to each group, which can change as they face more shots.
Here’s a simple idea for the class probability described above: run a logistic regression using cumulative shots faced and AdjSV% onto the outcome of whether a goalie ended up facing 1,500+ shots. This model would be easy to train and interpret.
And, it turns out, such a model is well-calibrated:
The new equation to derive the posterior save percentage then becomes:
P(1500+)*(alphaO + AdjSV%)/(alphaO + betaO + S) + P(-1500)*(alphaU + AdjSV%)/(alphaU + betaU + S)
where:
- P(1500+) = Predicted probability that goalie faces 1500+ shots in career.
- alphaO = alpha from fitted beta distribution on goalies facing 1500+ shots.
- betaO = beta from fitted beta distribution on goalies facing 1500+ shots.
- P(-1500) = Predicted probability that goalie faces -1500 shots in career.
- alphaU = alpha from fitted beta distribution on goalies facing -1500 shots.
- betaU = beta from fitted beta distribution on goalies facing -1500 shots.
Code available here: https://github.com/spazznolo/goalie-performance/blob/main/posts/post-4.R
Contextualizing experience
Goalies in the analysis
Whenever we decide on a range of years for an analysis (in this case, 2007-2022), we are cutting of goalies mid-career. Some, like Carter Hart, are beginning their careers. Others, like Lundqvist, ended them during in the middle of the analysis period. Since we are not yet adjusting for age, we can’t include these goalies in the analysis. Therefore, only goalies who started their careers post-2007 and ended their careers pre-2022 can be included. This is a 15 year range.
The analysis population changes as follows:
- Goalie population drops from 315 to 140.
- Harmonic mean of shots against rises from 12,690 to 14,568 (mean drops, 4,198 to 3,225).
- Harmonic mean of AdjSV% stays at .939 (mean drops, .932 to .927).
Career length
Let’s start simple: How long is the average goalie career? How many goalies face over 30 shots? How many make it past 200?… All of these questions can be answered by plotting the cumulative distribution function (cdf) of career shots faced for goalies.
Some thoughts:
- 25% of goalies faced 33 shots or less (!).
- 50% of goalies faced 202 shots or less (that’s about 7 games).
- 75% of goalies faced 2,606 shots or less.
We can gain another perspective on goalie experience by repeating the plot above for goalie career seasons played.
Some thoughts:
- 42% of goalies played only one season.
- 74% of goalies played five seasons or less.
- 90% of goalies played twelve seasons or less.
It should now be obvious that most goalies don’t really have a typical NHL career as we imagine them. Let’s confirm the assumption that goalies with more experience perform better than those with less.
It is difficult to understand a goalie’s path by looking at their save percentage or shots faced in isolation. What’s nice about the empirical Bayesian method is that it considers these measures at the same time. Moreover, we can repeatedly re-evaluate a goalie’s pAdjSV% after each shot they face. We can then plot this posterior over each shot of a goalie’s career to get a sense of their path. In order to extract more insight from this, let’s section goalies by their career shots faced, like this:
Goalies facing:
- less than 300 shots -> -0300.
- more than 300 shots, but less than 1,500 -> -1500.
- more than 1,500 shots, but less than 6,000 -> -6000.
- more than 6000 -> 6000+.
As an example, Braden Holtby’s pAdjSV% after facing various shot totals:
- 0 shots: 0.9417 (as is every goalie’s)
- 1,000 shots: 0.9421
- 5,000 shots: 0.9425
- 10,000 shots: 0.9452
- 15,000 shots: 0.9450
- 19,555 shots: 0.9433
Some thoughts:
- Nearly every goalie (77.1%) who faces -6000 shots ends his career with a pAdjSV% below expected.
- Goalies facing 1500+ but -6000 seem to fade as their career progresses.
- These goalies tend to be backups, facing ~600-1000 shots a season.
- This fade could partly be due to aging effects.
To get a clearer sense of the dynamics described above, let’s take the group average pAdjSV% through each shot faced.
Some thoughts:
- The effects are much clearer here.
- There isn’t much to glean from goalies facing -300 shots.
- These short NHL careers are almost certainly due to reasons outside their play in the NHL.
- Goalies facing -1500 shots fade quickly. They are given a decent look and fail acutely.
- Goalies facing -6000 shots start as the best group through the first 1,000 shots, then fade.
- This seemingly unintuitive result is likely due to randomness, and, more interestingly, age.
Let’s revisit the unintuive plot comparing pAdjSV% over goalie careers, grouped by career length. This time, we’ll plot the average age of goalies in each group as they face shots over their career.
Some thoughts:
- Goalies facing -6000 shots are ~1.5 years older than 6000+ goalies throughout their career.
- This difference obviously includes the span from age 23 to roughly 27.
- This is precisely the age range in which goalies seem to be improving in AdjSV%.
- We can adjust for this.