NHL Pick Probability
20 min read
This article consolidates six posts published between 2021 and 2024.
From rankings to pick probabilities
If you’re picking first at the next NHL draft, you want Lafreniere. If you’re picking second or third, you want Byfield or Stutzle. If you’re picking fourth, or fifth, or sixth, or seventh, you’re picking Rossi or Perffeti, or Raymond, or Drysdale… Notice how the list lengthens as you make your way through the draft? That’s because the uncertainty of a player being better than all other available players increases the deeper you get into the draft. So maybe you really like Rossi, but you’re picking sixth, and you want to be reasonably certain he will still be available. Well, what are the odds Rossi is still available at six? It’s hard to say.
Let’s say we only have our own rankings to go on. Instead of only predicting each player’s draft position, we could create probabilities of each prospect being drafted at each pick. Let’s use Rossi again as an example. Here’s how we might place probabilities on Rossi’s draft result:
This is a probability distribution of Rossi’s predicted draft result. Though this is a step in the right direction, these are only our predictions of where Rossi might go. What if the teams drafting ahead of us don’t rank him as highly? Their probability distribution for Rossi might look like this:
If the teams ahead of us view Rossi closer to the plot above, then he’ll likely slide lower than we predicted he would, and our chances of drafting him are higher than we previously thought. In fact, if we knew what other teams thought of him, we could pretty accurately predict where he’ll still be available in the draft, which allows us to either a) be comfortable we have a strong chance of picking him without trading up, or b) slide down a couple spots, pick up a mid-round pick and still get him. An important thing to remember is that Rossi’s draft position is much less affected by how we think of him than it is by how everyone else thinks of him.
In practice, no team will ever know exactly how every other team has ranked each prospect. Instead, player-pick probability distributions need to be approximated by other means. Dawson Sprigings outlined one way of doing this for Hockey-Graphs which used bayesian inference and pro rankings publishers. I’m going to outline another possible way, which uses mock drafts generated by users on Draft Site to derive probability density functions for each player.
An Example
Draft Site, gets hundreds of user mock drafts each year. These mock drafts naturally create probability distributions for each player’s potential pick placement. For example, here was Mikko Rantanen’s in 2015 where each user’s draft was weighted equally.
This distribution should be smoothed, but first, I’d like to address the fact that some mock drafts are more informative than others. A user who can more correctly predict a draft’s order is more valuable than one who cannot. Therefore, larger weights should be given to users who are likely more accurate in their mock draft. Thankfully, there are a couple quality indicators available: a user’s difference to the average user draft, and the number of days before the draft date that a user last updated their mock draft.
Mock Draft Quality Indicators
Difference to the Average Draft
Mikko Rantanen’s median pick from the raw user data was 9. If a user selected him 25th, they would be 16 spots off. A user’s absolute error rate can be computed for each pick in their mock draft. Below is the relationship of users’ mean absolute pick difference and their mean absolute pick error to the actual draft from 2015-2019. There’s a strong relationship between the two.
Days to the Draft
Usually, ranking publications will release a preliminary rankings list about a year before the draft. Then, as the draft approaches and player’s develop – or don’t - the rankings are updated. The plot below demonstrates that user data becomes more accurate as draft day approaches.
The variables discussed above were used to (1) filter what are likely low quality drafts and (2) create weights for each user mock draft. Players’ adjusted pick probabilities were then fit and dampened (more information on these decisions is available in the Notes section).
Comparing Probability Distributions
The Effect of Treatments on Player-Pick Probability Distributions
Here’s a visualization of the effects various treatments have on the user rankings.
The downside to user mock drafts is that prospect ranking is likely a hobby for most users. They may mostly rely on second hand information provided by hockey sites, experts, and prospect ranking models. Before going further, it’s important to measure its capacity for prediction. One way of doing this is to build a draft ranking from the data, and then measure its accuracy against professional ranking publishers.
Derived User Rankings vs the Pros
Draft rankings are derived from player-pick probability distributions by iterating through each pick of each draft, and drafting the player with the highest probability of being taken. After each pick, the player distributions are re-approximated (more information is available in the Analysis Notes section).
Here is the mean absolute error of derived user draft rankings from 2015-2019 compared to various pro projections. All experts but Bobby Mackenzie, the gold standard of draft projections, have been greyed out. It’s worth noting that the goal of some professional draft analysts is to predict which players will have the best careers, and not necessarily the order they might be picked. Purple is the group average.
User data, when properly treated, is competitive with pro ranking publications at predicting draft order. The advantage is that user data has built-in player-pick distributions which can be used to answer important questions about the draft.
Another Perspective on Probability Distributions
Here’s the probability Mikko Rantanen had of being selected at specific picks: 1, 0.1%; 2, 1.0%; 3, 2.6%; 4, 4.6%; 5, 7.9%. Another way to look at this is to say the probability Mikko Rantanen would be selected in the first five picks was 16.3% (the addition of each pick probability for picks 1-5). These cumulative probabilities can be calculated for each pick. Here’s what this looks like on a plot.
This is called a cumulative distribution (each pick takes the cumulative sum of all previous pick-probabilities). A pick-probability curve like the one visualized above can be derived for each player. Given that these cumulative pick-probabilities are the cornerstone of the analysis, it’s important to measure their accuracy. The fit of these curves can be evaluated by going through each pick and asking the questions: what was the probability of this player being drafted by this pick and was he drafted by this pick?
Evaluating the Fit
Either a player was drafted by a certain pick, or they weren’t. This is called a binary event, with values 0 (he wasn’t) and 1 (he was). Mikko Rantanen had a 55.5% probability of being drafted by the eighth pick, and the result was that he wasn’t yet picked (0). The error for the probability attributed to this event can be seen as 0.555-0 = 0.555. Rantanen also had an 86.4% probability of being drafted by the twelfth pick, and the result was that he was picked (1). The error for the probability attributed to this event can be seen as 0.864-1 = -0.136.
Each player has probabilities attached for the first thirty picks of the draft. This is roughly 7,000 events. Errors can be attributed for these events the same way as they were outlined in the previous paragraph. Since the aim is to build well calibrated probability distributions, the brier score will be used to evaluate the fit. The calibration plot is plotted below along with Brier scores.
The perfect fit is the grey line, where outcomes occur the predicted percentage of time. Whenever a curve slides away underneath the line of perfect fit, like it does with the raw user data (and to a lesser extent the adjusted user data), it means the method tends to be overconfident in its assignment of probabilities. The plot above, along with the Brier scores, suggest the adjusted user data (score ~ 0.0604) is a better fit than raw user data (score ~ 0.198).
Cumulative player-pick probabilities for the 2020 draft are available here.
Notes
Filtering Users
Adjusted to User Average:
- RMSE to user average < 15
- Days to draft < 150
Adjusted to Pro Consensus:
- RMSE to pro consensus < 15
- Days to draft < 150
Attributing Weights to Users
A linear regression is fit using: RMSE to user average, days to draft, (and RMSE to pro consensus for data weighted to pro consensus) as predictors and the RMSE to actual draft order as target. User weights are the inverse of the linear model predicted RMSE of user ranking to the actual draft order.
Fitting Player Distributions
A gamma distribution is fit to adjusted data.
Dampening Player Distributions
Player distributions are dampened by the variance observed in prior years. This is superior to the variance in the raw data because, in this case, it is caused by actual deviations as opposed to what are likely bad user predicitons.
I wrote a similar article for the NBA Draft at Nylon Calculus.
A rank-ordered model
The prospect pick probabilities in the “Draft Pick Probabilities” tab of the 2023 Draft Stock tool are generated through a process which primarily involves the application of a rank-ordered logit model to draft rankings released throughout the year. The methodology is a simplified version of Predicting the NHL Draft with Rank-Ordered Logit Models.
There are three main components - first, partial draft rankings are made complete, then a rank-ordered logit model is fit, finally 100,000 drafts are simulated from model outputs.
Imputation of partial rankings
Technically, there are thousands of draft prospects. Consequently, draft rankings cannot include every prospect, and so, by definition, they are partial rankings. They need to be made complete to fit into our framework. We do this by first, restricting the population to prospects ranked in the top 100 by at least one publication. Then, through the PLMIX package in R, rankings are made complete using the frequency of their appearance in rankings as weights.
Create full rankings
The following historical code excerpt shows the ranking-imputation step:
Show historical code
# Get skater appearance counts
top_skater_freq <-
rank_summaries(
data=ranking_matrix,
format_input="ordering",
mean_rank=TRUE,
pc=FALSE) %>%
.$marginals %>%
colSums()
# Impute partial rankings to create full ranking matrix
full_ranking_matrix <-
make_complete(
data=ranking_matrix,
format_input="ordering",
probitems=top_skater_freq) %>%
.$completedataFitting of Plackett-Luce models
As for the rank-ordered logit models, we’re currently operating two. The first, a time-weighted frequentist method from the PlackettLuce package in R; the other, a Bayesian, tier-weighted implementation written in Stan by Tyrel Stokes.
The time-weighted frequentist implementation is a standard application of the Plackett-Luce, except that ranking lists are weighed based on their distance to the draft in days. The ranking weights were determined using my previous work on user mock drafts. The weights are available here for those interested. Using the draft day as the index, rankings published a month out are weighted at roughly 90%, two months at 77%, four months at 50%, and a year at 17%.
Build Plackett-Luce model
The following historical code excerpt shows the model-fitting step:
Show historical code
# Fit the Plackett-Luce model
pl_model <- PlackettLuce(full_ranking_matrix, weights = weights, npseudo = 0.1, maxit = c(5000, 100))
# Obtain maximum likelihood estimates from the Plackett-Luce model
mle_estimates <- coef(pl_model, log = FALSE)The tier-weighted Bayesian implementation is taken wholesale from Tyrel Stokes’ work on track racing. His implementation contains weights, however they are determined by the Bayesian framework, which was not written with time, but tier importance + noise in mind.
Simulation of drafts
These rank-ordered logit models attribute a “strength” score to each player. Drafts are simulated (100k times) by randomly drawing (without replacement) players using their strength score as weights.
The following historical code excerpt shows the simulation step:
Show historical code
# Simulate draft rankings
draft_simulations <- replicate(100000, sample(1:skaters, skaters, replace = FALSE, prob = mle_estimates))Assumptions
There are three main assumptions which don’t quite fit in this methodology. The first is that the rankings are truly full rankings (they are not). The second is that draft rankings aren’t related over time (they are). The third is that ranking publications are representative of NHL organizations (unsure, could be verified with historical data). We explain each below.
On “Full” Rankings
In an ideal scenario, full rankings would encompass every eligible prospect in each ranking, providing a comprehensive view. However, this is impractical in reality. The challenge lies in incorporating partial rankings into a complete ranking system. One approach is to establish a cut-off point for the number of players included in the rankings. In my opinion, a suitable cut-off would be to limit the rankings to the top 100 prospects. If a publication ranks more than 100 prospects (e.g., 150 or 250), we disregard prospects ranked 101 and beyond. This approach minimizes information loss since only a few publications extend their rankings beyond 100. By focusing on the players within this new set, we are left with approximately 200+ prospects and growing. However, it’s important to note that the current model does not acknowledge the existence of other prospects outside this set. Consequently, the model assigns a 0% probability to a prospect ranked 250 being drafted within the top 100. As a result, the estimated probability of the 200+ prospects being drafted becomes overestimated. This issue becomes increasingly problematic as the draft progresses, and the model’s effectiveness is limited after the first two rounds. One potential way to address this is by adjusting for historical undercoverage, but unfortunately, we lack the necessary data for such adjustments.
On Time
To simplify the model, time was included through weights. Essentially, we are flattening time by saying “a ranking published right before the draft is worth about twice as much as a ranking published four months ago”. Even though this works for prediction, it is not actually how the dynamic works. The reason rankings change is not because of time, but because of what prospects/scouts do during this time. This assumption is addressed in Predicting the NHL Draft with Rank-Ordered Logit Models, linked above.
On Selection Bias
The method employed in this model assumes that prospect ranking publications represent the perspectives of NHL organizations. By “representative,” we mean that these publications assign similar value to players and exhibit comparable variance in their evaluations. For instance, it is plausible that ranking publications lack the same level of resources as professional hockey teams, leading them to rely predominantly on data rather than in-game observations. Consequently, they may place greater emphasis on prospects who excel in ways which are represented in game logs. Any inherent “bias” present in these publication rankings would consequently influence the published pick probabilities in this model.
The code used to generate these probabilities is available on GitHub here.
Turning probabilities into pick value
The prospect pick probabilities available on the “Draft Pick Probabilities” tab of the 2023 Draft Tool, created by Pranshanth Iyer and myself, can be used by NHL organizations to inform their decision-making process. Let’s take the Montreal Canadiens as an example.
The Canadiens pick fifth, which, according to consensus, is just outside the top tiers of prospects consisting of Connor Bedard, Adam Fantilli, Matvei Mitchkov, and Leo Carlsson. The next “tier”, contains a handful of players which ranking publications seem to mostly disagree on. One common question is, if you’re the Canadiens, do you trade up to guarantee the drafting of a potential superstar, or do you take the risk and hope one of them slides in the draft? However a more interesting question, in my opinion, is: “How are you making this decision?”
The truth is that either way you’re leaning, you are making a complex calculation with instinct. You’re considering 1) the value of each prospect, 2) the probability each prospect is available at a given pick, and 3) the appropriate cost in assets to move up in the draft. This is an exceedingly difficult calculation to make in the moment, especially in the heat of the draft.
Thankfully, the probabilities in this tool can help.
Prospect Values
First, we need the value of the top 5 prospects according to the Canadiens. For this example, let’s assume they evaluate prospects based on their predicted Wins Above Replacement (WAR) while their rights are held by the team (7 years), and the predicted WAR of their top five ranked prospects is as follows (feel free to use the metric and values of your choosing):
- Connor Bedard: 24.0
- Adam Fantilli: 19.0
- Matvei Mitchkov: 17.5
- Leo Carlsson: 14.0
- William Smith: 10.0
Prospect Pick Probabilities
Next, we can use our tool to get the probability that each player is still available at each pick. Here they are:
Pick Values
As the Canadiens, we derive our very own pick values by multiplying the probability a prospect is available at a certain pick by their predicted WAR. Obviously, we’re going to take the highest-value player available, so the pick values are calculated like so:
1st pick - (1.000*24) = 24.000
2nd pick - (0.002*24) + (0.998*19) = 19.010
3rd pick - (0.000*24) + (0.210*19) + (0.790*17.5) = 17.815
4th pick - (0.000*24) + (0.028*19) + (0.420*17.5) + (0.552*14.0) = 15.610
5th pick - (0.000*24) + (0.002*19) + (0.111*17.5) + (0.269*14.0) + (0.618*10.0) = 11.927Using the third pick as an example:
- There is a 0% chance Bedard, the Canadiens’ highest ranked prospect, is available, so his value is multiplied by 0.
- There’s a 21.0% chance Fantilli, the Canadiens’ second-ranked prospect, is available, so his value is multiplied by 0.210.
- If neither Bedard nor Fantilli is available (79.0% chance), the Canadiens would select Mitchkov, whose value is multiplied by 0.790.
- In total, the pick is valued at 17.815 WAR.
Previous Work
Note that this is a departure from the usual draft pick value chart which has long been established in the hockey analytics community (below is an example of the Athletic’s) which uses average pick values:
The reason for this proposed departure is that, in practice, there are weak drafts (like last year), and strong drafts (like this year). Even within a draft class, there can be pockets of heterogeneity where talented players are clustered together, as well as instances of significant drops in value. Consequently, for our purposes, it is more appropriate to derive pick values based on the prospects eligible for this year’s draft. Moreover, since the historical value chart is well-established, the Canadiens should be trying to leverage it to find value gains.
Decisions, Decisions
With pick probabilities and prospect values, the Canadiens now have a quantitive framework to determine the value of a pick in the upcoming draft. This allows them to assess the value difference between picks and consider trade scenarios. Which brings us back to the initial question: should the Canadiens consider trading up?
Let’s explore this with a hypothetical scenario. Imagine they have the opportunity to trade up to the fourth pick. To gain organizational value, they would need to give up less than the value of the fourth pick (15.610) minus the value of their fifth pick (11.927), which equals 3.683 WAR. If they can trade their fifth plus a piece which is worth less than 3.564 WAR, they’ve gained value. With this, a complex decision becomes relatively simple, at least in theory.
By combining prospect pick probabilities with internal evaluations, the Canadiens can make more informed decisions about their draft strategies and potential trades.
For more information, visit the 2023 Draft Tool. You can reach Prashanth and I on twitter (@iyer_prashanth, @spazznolo).
Adding uncertainty
The framework can also include prospect value uncertainty.
The Michkov problem
One of the main considerations for teams drafting in the 2-7 spot this year was whether they should risk their extremely valuable lottery pick on Matvei Michkov. He signed a 3-year contract in Russia. He only met with a few teams. He might be the most talented player in the draft. He may never come to North America… etc. Essentially, the question these teams were asking is this: is it worth drafting Michkov’s uncertainty when more reliable, though perhaps less skilled, players are available? Once again, teams have to make a complex calculation. Once again, they’re left with mostly instinct.
Unless!
Unless, prospect value uncertainty is quantified and included in the decision process.
Assigning Uncertainty
Adapting the framework to include uncertainty is actually easy. Instead of assigning prospect values through point estimates (Bedard was at 24 WAR, Michkov at 17.5), prospects are assigned probability distributions which reflect the uncertainty of their value.
As an example: instead of Bedard’s value being 24 WAR, it will be normally distributed with mean 24 and standard deviation 3. Here’s what that looks like:
In this example:
- Bedard has a 50% chance of providing 22-26 WAR.
- Bedard has a 25% chance of providing 26+ WAR.
- Bedard has a 25% chance of providing -22 WAR.
Like the point estimates above, the actual distribution doesn’t matter, it’s just an example to illustrate the framework I’m introducing.
Let’s assign these value distributions to each player, making sure Michkov’s value has higher uncertainty by increasing the standard deviation of his distribution. Prospect values now look like this:
Then, these probability distributions are multiplied by the probability a prospect is available at each pick. Pick values now look like this:
With these pick value distributions you get the same comparisons as above, plus:
- the probability that a given pick will be more valuable than another.
- WAR for whichever “outcome” you’re interested in (like top 10% scenario, bottom 10%, etc.)
Let’s re-visit the Montreal example where they should only trade up if the value of the fourth pick (15.87) exceeded that of their package (fifth pick (13.15) plus a piece). After including uncertainty, these point estimates become probability distributions and the difference between the two picks looks like this:
Code available here: https://github.com/spazznolo/draft-rankings/blob/main/scripts/post_4.R
A drafting strategy
The pick value framework captures the unique dynamics of a given draft and the uncertainty of prospect values. It also suggests a drafting strategy which aims to maximize draft value. But first, some background.
Background on Optimization
The optimization of draft value has been explored specifically for the NHL by now Kraken analyst @nnstats in this 2017 paper. It was a counter to the prevalent discourse at the time, which assumed the optimal draft was achieved by simply picking the best players as they were available. Instead, she argued that a team should only draft the best player available (BPA) if he won’t be available for their next pick. For instance, Jamie Benn, a sixth round pick, should have been drafted in the fifth round to maximize total draft value.
The key outstanding question from this work was: “How can we determine if the risk of deferring on the BPA and potentially losing them is worth the reward of drafting another quality prospect before landing the BPA later?” The prospect pick probabilities from our draft tool, when combined with prospect values, allow for a probablistic framework to make such a decision.
The strategy
The core objective of this strategy is to determine a team’s maximum conditional draft value at each pick. The maximum value is selected from a list of conditional draft values associated with each available prospect. Using a team’s first two picks as an example, the strategy can be written (somewhat) formally like this:
for n remaining prospects, ranked 1 to n
max draft value = max(v1, v2, ..., vn)
where,
vi = estimated draft value when choosing prospect i with the next pick, specifically for i ≠ 1
vi = v(pi) + (Pj(p1) x v(p1)) + ... + ((1 - Pj(p1) - ... - Pj(p(x-1))) x v(px))
where,
v(pi) = value of prospect i,
Pj(pi) = probability that propsect i is available at pick j, and
Pj(p1) + ... + Pj(p(x-1)) < 1 and Pj(p1) + ... + Pj(p(x-1)) + Pj(p(x))) >= 1To further illustrate, let’s consider the first round of this year’s draft up to pick 17.
An example using the Red Wings’ first round picks
The Red Wings held the 9th and 17th picks in this year’s draft. I’m going to calculate the optimal drafting strategy for the Red Wings using a set of well-performing prospect rankings from Bob McKenzie, Cam Robinson, and Chris Peters. Prospect values are assigned using their respective rankings in the historical pick value chart.
At the 9th pick of the draft, the remaining prospects were ranked as follows:
Let’s determine who each analyst should draft at the 9th pick to maximize their expected draft value for picks 9 and 17. To answer, we use the equation shown above, which requires: 1) the value of the prospect taken at 9, 2) the conditional probabilities of the remaining prospects being available at 17, and 3) the values of these remaining prospects.
Here, McKenzie and Robinson should take the BPA to maximize conditional draft value, but Peters should defer and take his second BPA, Zach Benson.
After the draft
In general the probabilities were pretty well calibrated. However, the model was too certain in the top 10 picks. As a result, the sportsbooks cooked us.