The Error of Our Ways

I was going to post this at Tom Benjamin’s Weblog, but felt it was going on too long and it fits better on this website than anywhere else. It would appear Tom is getting all the good discussions lately.

True. The question is whether the NHL season is long enough for us to get an idea of an individuals' true talent level from those results. I tend to think that it is - I give a lot more credence to the teammate line than the sample size line. If the sample sizes were too small to give any indication, you'd see drastically different lists of scoring leaders each season. You don't. – Tyler (mc79hockey)

Goals/Second

Error is pervasive in hockey, most specifically because goals are so low. You can easily approximate error for plus and minus statistics by using a simple binomial formula sqrt(n*p*(1-p)) where n is seconds on ice, and p are goals/second. For the average player n = 50000s and p=0.00071003G/s and their minus or plus statistic for that matter is ± 12. One can approximate the accuracy of the plus-minus statistic by using the simple sqrt(12²+12²) = 17, suggesting the plus minus statistic has a range of around ± 17. Of course this isn't perfect, as it's the error for a binomial distribution (hockey is arguably Poisson), but it's the best I can quickly do. What it really comes down to is that 35 scoring events don’t constitute very good information. Think about flipping a coin 70 times: you expect 35 heads, but you’re going to get 35 ± 8 95% of the time, would this make heads more likely if you get 40 heads vs. 30 tails, no it would just appear that way.

Shots

One cannot easily fix this error by looking at shots as opposed to goals (as shots occur at a relatively constant rate for each player), the results are similar: ± 12, playing less reduces the error slightly, but reduces the goals by even more increasing thee percent error from 12/35 = 35% error, to 8/17 = 46% error. Shots per second are more useful as there are 10 times as many shots as goals, this works out to 350 ± 18 or an excellent 5% error. You might actually get some reproducible results with this information. Problem of course is not all shots are equal, not all shots are bad. We would likely be in agreement if I said a shot from greater than 60’ out has almost no chance (~2%) of going in the net and 30’ constitutes a reasonably bad distance to shoot from (~5%). Of course this isn’t the same for all players. This general information generated interesting analysis at Hockey Analytics, which basically leads to a binary logistic regression. This regression provides information about the likelihood of each shot going in. As I showed in this study these shots against are good measures of defense (they ignore quality of goaltending as well). Problem is they appear to be a poor measure of offense (although after going through the offense error today I might re-analyze how poor). However the regression and the analysis (as most people have already discovered) leave much to be desired, a 14’ shot could be a break away or just a soft throw at the net. There’s no way to tell on a score sheet. The distances recorded by the NHL are significantly flaws (distance from backboards mean nothing to me), they’re good approximations, but unless you get real x and y coordinates you’re data has more error as a result of these problems. In order to understand this error I need to understand logistic regressions better.

A few examples

Weinrich played 16 games in Vancouver, has was a -17 and plus 5 in 13000 seconds, his -17 score is ± 8, so 8/17 suggests 47% error, his plus 5 was ± 2 or 45% error, Weinrich had a expected goals against of 7 (note this is outside of the expected range) and an expected goals for of 7, so he should have been a 0. In St. Louis, Weinrich, was also -51 ± 14 in St. Louis, but had an expected goals against of 43 (this is inside the expected range), but both are lower and does create some suspicion to the accuracy of shot measure of skill. It’s conceivable though that when Weinrich was on a team that was doing well he would do better then expected (and worse on bad teams).

In one game a player might go -2 in 15 minutes of even strength time, however the error is over 100% of the data, meaning one cannot really learn something (by looking at goals) about a player in one game. Shots against statistics in one game are 90% error.

Forsberg, for example has played 10 seasons in the NHL if you take statistics on season scoring numbers you get a standard error for the mean of 3.5 (± 7), which is lower than the expected (± 14), of course there’s error in the standard errors was well. Jagr has a standard error of the mean (for points) of 6.1 (± 12), which is certainly well within what’s expected. There are a number of factors involved that go beyond a player’s skill (how much power-play time, how much ice time, line mates etc.), but you can see there is enough error in aggregate points to make the above error conclusions.

Generally a statistic that goes out of the bounds of “normal” is probably unstable, for example Anson Carter shooting percentage was 23% (twice the average). This is unsustainable statistic, while he may hope to score the same number of goals next season he should be happy getting 25 goals next season.

Cooke this year was chastised for his poor play (even though he never played all that much), but he could’ve had 8 more goals (or 8 less for that matter) and it would be in the range of expected error

Ups and Downs

This information helps explain why a player can have up and downs among fans, their skill level doesn’t actually change, but the results do. A player may do poorly for a while as a result of nature of the sport and be chased out a city very quickly. He can move on and do very well.

Top Scorers

A quick note on top scorers: they have much less error due to their increases in goals. The error does not increase all that much, but the goals do, so the error as a percentage of goals is drastically lower. So the best players in the NHL don't change as much as the average players do.

Pre-Season

The Battle of Alberta has an interesting article on the significance of the pre-season and its ability to predict the regular season. Using a nifty tool the Spearman’s correlation rank test Colby Cosh concludes “that success in exhibition games does predict success in the regular season”. I will argue that points and thus winning percentage satisfy normality, homoscedasticity and linearity assumptions and so I can use a standard regression. This isn’t too much of a stretch.

Colby Cosh used Yahoo! sports to compare the pre-season data of 2005-2006. I decided to include 2003-2004, so my data set has 60 data points.

Regression

So the easiest tool is a regression on winning percentage pre-season vs. winning percentage regular season as one can see in the graph below. If you remove the constant (removing constant creates a “bad” regression) you get a near “perfect” coefficient of 0.939 ± 0.04 (note: doesn't reject 1) [graph 1]. In fact this regression has better error properties than the error with a variable intercept (reg = 0.445 + 0.182 * pre [graph 2] ). I don’t have the patience to compare these two regressions, but I will say the relationship “looks good”. Interestingly goals for have similar regression: Goals For Reg-Season = 0.945* Goals For Pre-Season, similarly for Goals against: Goals Against Reg-Season = 0.962 Goals Against Pre-Season.

There’s not much to say here, but to look at the actual graph of this regression, it’s not pretty, but you can see how the regression was fit and how it “makes sense”. You can also see it’s not chasing the outliers like a standard regression would. Another way of looking at these problems can been seen with this example: a team that performs at 70% in the pre-season has a confidence interval of 60% to 70% (confidence interval is of the “mean”), and you can only predict with ±30% (worthless in the NHL), and this is the real question here. With the regression that includes the intercept the prediction intervals are cut by 50%, but still cover most teams’ general success level for a season (not much better). So while there’s an obvious relationship between preseason and regular season (as one would expect) this is not useful as it’s an average and there are a lot of below average and above average teams that comprise this score.

Conclusion

So while there’s a general tendency for teams to do well in the regular season if they’ve done well in the pre-season it’s no guarantee or even something to predict the regular season with. As a Vancouver fan I don’t exactly like these results, but I understand they’re a poor predictor, so I’ll wait for the real thing to come to make any conclusions.

As an aside: Vancouver has to stay out of the box, so far by counting they’re a -8 in terms of power play opportunities. Also, I hope Luongo improves his 0.839 save percentage.

Added Note:

I decided to include another variable in the model: that is the winning percent of last season (it's a good control variable). If you do a regression with two varibles: pre-season and last season, you get a regression that looks like: winning% = - 0.011 + 0.565 * pre + 0.941 * last - 0.861 * pre * last. However, the only term that is significant in this model is last season's performance [The others should be removed, but I wont]. So the preseason is a worse "predictor" than the previous season. For example a team that is 100% in pre-seaon play and 55% [playoff team] is predicted to perform anywhere from 42% to 78%, however a team that does 55% in per-season and 55% in last season is expected to perform anywhere from 40% to 73%, so not a huge difference for the preseason.

In other words, if you want a prediction look at last season it's much more informative than the pre-season...

Diving Preferences

I was going to throw this on Tom Benjamin's NHL Weblog, but it turns out I’m a spammer, so I decided to expand and present it a little better and put this longer post on my webpage.

The first diving infraction would result in a warning letter being sent to the player. A second infraction would be accompanied by a $1,000 fine. A third infraction would result in a telephone hearing with the league and a possible one-game suspension. The length of the suspension would double for any subsequent violation. ~ NHL new Diving Rules

Last I checked $1,000 is 0.2% of the league min (should be at least 2% of salary - yah $135,000 for Luongo!). Also a one game suspension is much more expensive to Naslund than let's say Goren (although he may never come back...).

However this is a math website, what would it be without some sort of equations and numbers?

If you'd like to control diving in the current rules you need to first consider the value of a dive:

Value of dive = p_power_play * 0.2 Goals/power_play * Value/Goal

If p_power_play is close to 0.5 you're looking at 0.1 Goals per dive... To prevent dives one needs to have costs in excess of these benefits. A marginal goal is worth around $200,000¹ so a dive would be worth about $10,000. There are likely some "opinion" costs (other players hate you). Of course this isn't a fixed cost: a marginal goal is worth more when the game is tied in the end of third period. It's doubtful you could prevent diving with any tools; due to the value can approach $1,000,000 in the end of the game in an average game and almost unlimited to make the playoffs (bounded by $40,000,000?)

The other side is the cost:

Actual costs = p_caught * costs

Currently p_caught sits around 0.05, (IMHO) as this keeps the false positives down (only penalize the obvious). At this rate the cost to the player and team would have to be around $200,000 (a marginal goal), raising p_caught to 0.25 would lower this to $40,000 (penalty shot works here...). Since Actual costs need to exceed benefits in order to prevent them the equation we want here is:

Costs > Value of a dive/p_caught.

This really makes the job quite simple: decrease the value of a dive, increase probability of being caught or increase costs associated with diving.

Of course diving isn’t quite that simple the other has to do with the subject quality of diving. That is of course is false positives (player doesn't dive and gets called) and false negatives (player dives doesn't get caught). Even if you could figure out a fool proof set of rules you're going to have these problems - note: referees are afraid of these (if management disagrees with you then you could be in the dog house, mostly for the false positives). So when considering rules you want to make sure they’re not too severe so that referees are comfortable making the false positive periodically.

For example when a referee sees a player fall there’s a potential dive, he considers the situation in order to determine if it’s a dive. In general statistics a false positive is much more serious than a false negative, it is much more serious for the NHL to hand out serious punishment (one game suspension) to a player who didn’t dive and as such a false positives becomes much more embarrassing. “There is no [way] that will simultaneously make both [false positives] and [false negatives] [lower]” (Devore - Quote simplified for non-math people). Reducing the false positive rate means increasing the false negatives. I suspect the NHL uses something 99-99.5% confidence to determine dives. This means a significant number of false negatives (we all know this already). However in terms of entertainment and respect for the game false negatives are much more costly, as they are making the sport a joke.

In order to fix the above problem the NHL has to make false positives much less costly. As such a referee who calls a dive that isn’t a dive doesn’t get criticized and lambasted for it. A called dive results in fines and other punishment outside of the sport that garners a lot of attention. A dive should be controlled within the confines of the given game. There are many ways to do this.

So what are the solutions?

1. Simplest: fines to team and player. This is as explained above and calculated as $10,000/p_caught, or likely around $200,000 (equivalent to 1 marginal goal). These rules could lower p_caught further making these rules something that’s never used and just a waste of the paper it’s written on.
2. Of course don’t call any penalties that include a dive (don’t call either player). This would reduce penalties called in total; p_caught would have to be very high. False positives wouldn’t be that bad (neither player was called, no one is actually penalized) as the NHL already misses a lot of calls.
3. The opposite of the above is to penalize both players (current standard) this also requires a high p_caught value. The NHL throws fines and other such penalties lowering p_caught and as such this has been ineffective.
4. Of course the best way to control diving is that the value of a dive is nothing. Most fans realize this and referees would likely like this as well, but in practical terms it’s very hard to see infractions in hockey, but a falling player is quite noticeable. Also the NHL seems to have a general understanding that the punishment should be a function of the results and not the action itself (take note of Bertuzzi and other suspensions and actions).
5. Penalty shot for each dive. As a penalty shot is worth ¼ of a goal this would be worth about $50,000, a p_caught rate of 20% would be needed. False positives would not be significantly costly in terms of the game ($10,000/$1,000,000 = 1%). But it would equal the benefits associated with diving.

There are many other things that can be done obviously within the above context; the goal is the same, control the costs associated with false positives. In general it’s better to give the costs associated with false positives to a team rather than a player as they will average out over the course of a season. The second goal is to make sure that players can see the costs and that these costs will exceed the benefits. However, the NHL has made it more costly for false positives and they still don’t have rules that exceed the benefits.

So will the new rules make a difference, absolutely not $1,000 is nothing to a hockey player, and missing a game isn’t even worth as much as dive except to the team's best players (who the rule “doesn’t apply to”). And as such the benefits still exceed the costs so players will likely continue to dive at the same rate and the NHL won’t know what to do about it.

So enjoy a season filled diving. There's nothing new here. Also, enjoy a season filled with a ton of garbage calls as a result of this diving and expect the NHL to do nothing about it. The NHL wants more scoring, to get that you need more power plays and diving produces more power plays. The NHL has no incentive to get rid of diving. They just want to make it appear like they’re trying to control them so the fans are happy.

Kesler

As most people know by now Kesler has signed with the Philadelphia Flyers for $1.9 million dollars. Of course the Canucks have until September 19, 2006 to make the same offer to Kesler to keep him, which has just happened. Kesler is 22, which, by chance is the same age as me I might add. He’ll be a free agent when he turns 27 according to the new CBA (I think), which means that he will have played 5 seasons with the Philadelphia Flyers, provided no one does the same thing to the Flyers. Someone else signing Kesler is unlikely considering his value is going up not down and as such one would need to give him more than $2.0 million meaning he would also cost a first and third “rounder”.

Kesler is a first round draft pick (23 overall in 2003), that’s only 2 years ago. In my opinion he’s been a disappointment offensively, but this is likely due to the limited support he’s had. He certainly was one of the best players on the team in terms of defense. He was one of the teams better (if not best) penalty killer. And no question played a significant role no matter how you look at it.

It is extraordinarily rare for a 21 year old centre to see that kind of responsibility and still end up at evens in EV+/-. ~ Vic Ferrari.

The first question would be: What are the Flyers getting: they’re getting an NHL ready restricted free agent for 5 years ($1.9 million for the first year). I’m not sure how next years pay for Kesler works out for next year (it doesn’t happen too often a 23 year old takes a pay cut). Most people agree they’re paying for two things here: a free agent and the player. Remember also this signing is in terms of the free agent market one has to compare to group II free agents not group III’s because he is chosen amongst alternative in the group II’s (they could’ve signed any group II instead, at least a few months ago). The price includes the “value” of a second round draft pick. (Each team needs 20 players, each player has an NHL life span of about 10 years => Teams should be built up with about 10 1^st “rounders” and 10 2^nd “rounders” (this is extremely rough and wrong, but it’s a reasonable estimate: age: 25 – 35), of course many teams have 3^rd “rounders” and 4^th “rounders” for reasons such as trading away their 2^nd “rounders” for playoff hopes (try to calculate those numbers). And well teams aren’t very good at drafting. Some get injured some die, some sign in Europe etc. It suffices to say that a 1^st “rounder” such as Kesler should be a 1^st or 2^nd line player, as such their value ranges anywhere from about $1.5 - $3 million at this present time (Kesler isn’t at this point yet). (3^rd liners are $0.8-1.5M and 4^th liners less than $800,000). What I’m trying to say is that a 2^nd round draft pick = a group III free agent for let’s just say 4 years for a 4^th or 3^rd line player (not all that much if you ask me ~$200,000 per year for 4 years or $800,000), although in terms of “error” one could argue there’s a good chance that 2^nd “rounders” become as good as first “rounders”.

Let’s now assume Kesler still can’t score like a top line for the next two years this is how his salary might be projected (this is extremely important to the Flyers). Please don’t jump on the numbers as they are nominal numbers not real and as such include inflation. I believe Kesler will expect a pay cut next year and will sign for lower.

Year – Age	Canucks	Flyers
2006 – 22	$1,000,000	$1,900,000
2007 – 23	$1,200,000	$1,300,000
2008 – 24	$1,500,000	$1,800,000
2009 – 25	$1,800,000	$2,000,000
2010 – 26	$2,250,000	$2,250,000
Total:	$7,750,000	$9,250,000

Of course the Canucks “option” is no longer available due to the fact that the signing of Kesler at $1.9M puts him on the Flyers path either way. I realize this number is arbitrary, and it could be almost anything it’s hard to argue that the cost of Kesler hasn’t gone up at least a bit due to this years contract and subsequent years should also be higher (no one takes a 50% pay cut).

This scenario shows the importance of the group III offer sheet rules, without them Kesler would be unable to sign simply because the Canucks were ripping him off (it’s hard to debate that because of the price the Flyers will pay), in fact the offers by the Canucks were likely 20% below market value (if they weren’t below market value the Flyers would have no incentive to sign him). To be honest the only thing this shows me is how poorly Nonis has dealt with Kesler. Nonis told Kesler this was the last offer (whatever it was) and that he should sign it or else... Well Kesler decided to do the "or else" and left for a more reasonable GM (and won I might add).

The interesting thing of course is that I approximate the worth of a 2^nd round draft pick to $800,000 and the difference between this year and last at around $900,000 (although the Flyers might argue this number is lower), this suggests that the Flyers expect to save around $1,700,000 over the next 5 years in savings resulting from having Kesler, which suggests that a 1^st “rounder” like Kesler is worth around $400,000 per year, or twice that of a second “rounder”.

Of course all of this is guess work, something I don’t do all that much. I'm not sure what this all means, but teams will try to continue to recoup their losses by paying less (even more less I suspect) to their restricted free agents as such GM's with a a good eye for value can quickly snatch them up (of course no team wont match). Since every team matches it would appear all this will do is drive up the cost of having group II free agents and the value of draft picks will fall (and development with that) leaving teams with little incentive to develop talent (as a number of people have already predicted). We'll see if this happens again in the next few years or if thi is just another anomaly, but even if it doesn't happen it should effect negotiations as players know they can use this to their advantage.

Brendan Morrison

The new Canucks will rely heavily on Brendan Morrison he will no longer be the small guy hidden behind Naslund and Bertuzzi, but are the articles such as “Morrison looks to Increase Scoring” and “‘Mo’ Production Expected this Year”

"When you play with those guys, the tendency is just to get them the puck," Morrison said after skating with some of his teammates at a local arena. "Now I might have the puck a little more, get the chance to create a little more and definitely shoot the puck more. That should lead to more production."

So is he right? That is hard to answer on its own, but Morrison has a few times when he played with out Bertuzzi simply because Bertuzzi has a habit of being suspended. The first time occurred early in Morrison’s Vancouver Career when Bertuzzi was suspended 10 games leaving the players’ bench to join a fight. During these 10 games Morrison got 4 goals and 6 assists, which put him on pace for a point a game compared to the 0.611 ppg¹ in the other 72 games.

Bertuzzi was suspended another 13 regular season games when he sucker punched Steve Moore. During these 13 games Morrison scored 5 times and assisted another 7 times for 12 points in 13 games for 0.912 ppg¹ (again very close to 1), but scored a mere 0.637 ppg¹ in the other 69 games. Note this is very similar to the results 2 years prior. (Now whether their line improved is a more complicated question).

In the playoffs after the 2003-2004 seasons the Canucks matched up with Calgary (defensive team). The top line struggled mightily and Morrison got 5 points in 7 games for 0.74 ppg¹.

I should state that these differences between Morrison with and without Bertuzzi are not statistically significant (at the 95% level). The differences are large, but the sample size is too small to make any real conclusions, but I will say this is promising, although it only really indicates Morrison will score around 30 goals (as apposed to his usual 20). Naslund should be able to get 40 goals and Cooke/Bulis/No-name should be able to get another 20-30, which results in a net goals for the top line of under 100, but then again that’s not different than what happened last year.

Maybe Morrison is driving this lines bus. If so Nonis made a good decision here. It would be hard to tell because Morrison is always healthy.

[1]. PPG = Points per game.

Ownership is 9/10ths of the Law

Most people will agree, at least at some level, that possession of the puck plays a major part in game outcomes. How long you can control the puck may be a lot more important than many people realize. The exact value of possession is hard to quantify (not due to lack of ability, but because of lack of data). Many commentators talk about the importance of battles along the boards, face-offs and give-aways and take-aways. So how important is it really?

First off using the limited data available, how do you research possession? That isn’t easy, but certain events say something about possession, for example you must have possession to take a shot, if you win a face-off you have possession after the face-off. If you hit a player you shouldn’t have the puck (otherwise it should be interference). A blocked shot indicates you didn’t have possession. The NHL provides a couple obvious possession pieces of information in give-aways and take-aways (they say something both before the event and after). Using this data one can be reasonably sure about certain amount of time of the game (I believe it worked out to 50-70%). The rest is still up in the air, but in terms of averages it works out quite well.

Now when I originally did the study I found there was little or even negative correlation between possession and winning individual games (which was a surprise). However, when I averaged the data over the season I got some interesting data. I have two graphs.

The first graph (winning percentage vs. predicted wins) shows the standard Pythagorean prediction of winning using goals for and against. [Has a R² of 86%]. The second one (winning percent vs. possession) compares the teams winning percent (or points) over the season to their percentage of possession [Has a R² of 92%]. Now just considering their R², possession is a better predictor than the Pythagorean prediction. Now this is a small data set, and I hope to test it on more data, but in general being able to control the puck when you need to will result in more wins.

Why might this be? If you think about hockey if you have the puck the other team can’t score and if you don’t have the puck you can’t score. If you’re winning by one goal in the third period and you control the puck you are more likely to win, and if you are down and control the puck you’re more likely to get that tying goal and as such win (or OTL). So puck possession shows the difference between good and bad teams. The question that should be answered next is how then can a bad team possession wise (8^th seed) do so well in the playoffs vs. better possession teams. I’ll save that for another day.