May 13, 2007

The Joy Luck Club

I often have people say that my statistical measures of shot quality don’t factor in whose stick they are coming from. That is to say it makes a huge difference if Tanguay takes a shot over say Josh Green given the same variables: distance, rebound, power play etc. However, I am going to make a short little explanation that this is probably not the case and that good players score because they’re in the right place at the right time and get their shots on net. Then I am going to list players who have significantly outperformed expectation over the year and those who have done a lot worse.

Sniper

How can I show that outperforming expected goals is the result of luck and not skill? Year over year correlations are the best way. That is to say, do a regression on how well compared to expected results a given player did in 2005-2006 and compare that to identical player in 2006-2007. If skill is a significant component of the data then we should see that good players in 2005-2006 are good in 2006-2007. If this is not the case then there are other reasons (most of which can be attributed to luck).

The graph below is a scatter plot of the Z scores for each year. Z scores simply allow me to normalize out the quantity of shots and create a unitless measure of skill. The amount of variability in the data is about 5-10% greater than I would expect, assuming binomially distributed data, which is likely the result of problems with the shot quality model.

There are small signs that there are players who can outperform expectations. There are a few regressions that are significant at the 5% level, but these regressions explain 1% of all the variability, so I will consider that other influences, such as luck, have a larger effect on the results. I’m not saying that players can’t outperform expected goals; I’m just saying that outperforming expected goals is more likely a sign of luck than a sign of skill.

Here’s the list

The team listed is one of the team’s the given player has played on. It just helps me figure out who the player is. P = F:forward, D:defense, G = goals (not empty nets), D = difference between actual goals and expected goals, Z = z-score of the difference using binomial standard deviation. [sqrt(npq)].

The Z-score represents a unitless value about how much better (or worse) a player did than expected. It just creates a nice value to sort the list by. The D value is the absolute number of "unexpected" goals.

Below I’m going through the best and the worst on the list to demonstrate that it makes sense with the extreme values and that skill doesn’t seem to be the reason players come to the top of the list or the bottom of the list.

Explanations of the top 10.

1. Semin: Scored 10 goals in the 52 games previous to this season and wasn’t a goal scorer in Europe. But managed almost 40 in 77 games. He’ll likely be a 20 goal scorer in the future. His goal:assist ratio is 1:0.9 (it should be 1:1.7), this is also a sign of luck. Semin is likely a 20 goal scorer, which is good, but not amazing.
2. Souray: This season he should’ve scored 13, but he scored 26 goals. Last season he scored 12 (in a few less games). I don’t need better evidence than that. He’s a “13 goals per season” goal scorer (excellent for a defenseman).
3. Visnovsky: 44 goals in the 348 games (0.13 g/g) prior to this season. 18 goals in 69 games (0.26 g/g) should’ve scored 8 goals (0.12 g/g).
4. Kuba: Never surpassed 10 goals his entire career, but was able to score 15 goals this season. I’d bet he’ll closer to 7 than 15 next season.
5. Huselius: Many people were most impressed with Huselius this season and for good reason he scored over double his previous season. But he exceeded expectations by about 12. Meaning he should’ve only had about 19 goals and he’s historically been a 20 goal scorer.
6. Staal (J): He’s going to be good, but 8 shorthanded goals and a goals:assist ratio > 2:1 is insanely uncommon. However since he’s young, he’ll be better next year so I wont say how well he should do, but I expect it be somewhere between 20 and 30. And I expect a lot more assists.
7. Horton: Two almost 30 goal season back to back, you’d think there’d be something going on. Horton is more likely a 20 goal scorer than a 30 goal scorer, but he’s still young so he’ll get better as well.
8. Lecavalier: You always need a little luck to win the big trophies and Lecavalier is no exception (still a great player). But scored 16 more goals than expected, which would bring him very close to his previous season’s scoring of 35 goals.
9. Roche: Who? 6 goals on 32 shots. I need not say more. I expect him to disappear next year.
10. Drury: Buffalo is covered with outperforming players and Drury is the leader of the pack. In 2005-2006 Drury scored 30 goals and 18 in 2003-2004 or 48 goals in 156 games for 0.31 goals per game. In 2006-2007 he scored 37 in 77 or 0.48 goals per game. He has maintained this scoring level in the playoffs (7/13), but I wouldn’t bet on it continuing forever.

Explanations of the bottom 10.

1. Kobasew: Calgary probably thought they were getting a deal trading away dead weight at the end of the season, but Kobasew should’ve had 9 extra goals, which would have basically tripled his nice total of 5. 14 goals in 50 games would’ve brough Kobasew quite close to his 0.26 goals per game record in 2005-2006.
2. Hoggan: Who? Was demoted after scoring 0 goals in 46 games.
3. Cullen: 25 goals in 78 games last season and 16 goals in 80 games this season. My numbers say he should’ve scored 28. Sounds reasonable to me.
4. Niedermayer (R): Lowest goals per game in his career and the closest he has come to this horrible scoring total was 1994-1995, when he got 4 goals in 48 games (21 year old). He should’ve had 10, which is close to his normal historical performance level.
5. Gomez: 33 goals to 13 goals, need I say more?
6. Hollweg: Who? Probably won’t ever be a goal scorer, but probably won’t see such a bad year as this twice.
7. Sedin (H): 18 goal season to 10 goal season. Won’t be a goal scorer, but he should’ve scored a lot more
8. Svatos: Amazing rookie season, but only scored 15 this year. He should’ve scored about 25, which is great number of goals in 66 games. Probably score 30 goals next season.
9. Marchant: Historically scored 0.18 goals per game and this season scored 0.11.
10. Robidas: 0 goals on 106 shots. He’s not a goal scorer, but that’s just doesn’t happen very often to forwards.

All this said, being lucky this year doesn’t lower your chances of being lucky next year, so any prediction could be wrong simply because they could be lucky two years in a row (or unlucky for that matter).

Future Uses

I am planning to go through the players and adjust assists based on how lucky the scorer was all season. So, an assist on a luck player will be worth less than an assist on an unlucky player. So Daniel Sedin’s assists on Henrik Sedin’s goals will be worth more than assists on Drury goals. I’ll then see if the results make sense.

Anonymous said...

There are a gazillion factors that go into how a goal is scored that doesn't account for in your shot quality measurement including who shot the puck, how hard was it shot, how accurate was the shot, who and how was the play set up, who the opponents were, was it deflected in any way, was the goalie screened, and even who the goalie was. All of these things factor into scoring goals but the majority of them are not accounted for in your shot quality measurement and thus you are likely attributing differences to luck when you aren't attributing it to these other factors.

Look at it this way. A 6 foot shot could come on a breakaway, could come on a cross crease pass, or it could come on a scramble in front of the goal with the goalie lying flat on the ice. The chances of each of these going in are likely vastly different and you I believe you would rank them all the same. Do you factor in the goalie or do you rank those shots the same regardless of whether Dan Cloutier or Roberto Luongo is in goal?

I just think it is rediculous to assume that Souray's ability to blast the puck on a one-timer has little or no effect on his ability to score goals. Same for McCabe, another guy with a good shot from the point.

JavaGeek said...

Assume: An axiom or statement, not necessarily true but put forward and taken to be true to enable further analysis of a hypothesis

I just think it is ridiculous to assume that Souray's ability to blast the puck on a one-timer has little or no effect on his ability to score goals. Same for McCabe, another guy with a good shot from the point.

First off I didn't make that assumption, I've been working off the opposite for a long time (expected goals != actual goals at the individual level). I provided some reasonable evidence that luck dominates skill and therefor outperforming expected is a better indication of luck than skill.

A better offensive defenseman will be more accurate and therefor get more shots on net (miss the net less), so accuracy is accounted for to a certain extent. However, this is irrelevant, the fact that there is no correlation between doing well this year and doing well the in the next means that it is 99% more likely that Souray will play at his expected goals next year than the alternative: playing at his actual goals.

That said, players can outperform expectation, I'm not debating that, but luck is by far a bigger chunk of the information.

Basically saying:
Shooting % = shot quality % + luck % + skill %
skill %/shooting % = 0.01 (not quantifiable)
luck %/shooting % = 0.40 (calculated by using assuming luck %/shot quality %, because skill %/shooting % is small)

You would need a lot of skill to make the opposite conclusion: skill dominates luck.

What is more crazy of a conclusion about next year's goal scoring by Souray?:
A. He will be the league leader two years in a row and score 25 goals
B. He will be a good offensive defenseman and score 13, his historical average.

The last player to score 26 goals in one season was Gonchar in 2001-2002, it happens every 5 years! He scored 18 the next year (-8) and 13 this year.

I don't think I can convince you though, that's fine.