The Problem With Stats

The annual Sloan Analytics Conference was last weekend, making this as good of a time as any to look at the state of the statistical revolution. Full disclosure: I'm a stats guy. I grew up reading Bill James and Pete Palmer, and I own a copy of almost every Baseball Prospectus. The sports-reference sites are one of my most frequently used bookmarks. I love this stuff.

However, I am sympathetic to those who feel stats have ruined the game. It's hard to discard an old paradigm and be told, essentially, everything you knew is wrong. It's easy to learn things, but it's tough to unlearn them. Those of us who grew up with these concepts don't have a problem accepting them, but I can see the resistance to new schools of thought.

I also think that the anti-stats crowd has a bit of a point: some of the alphabet soup is a waste of time. Stat guys can get lost in the rabbit hole and end up just talking to each other, which defeats the whole purpose of the endeavor. The goal should be to increase everyone's understanding of the game, which means engaging a larger audience. Too often the statheads spend countless hours refining the math instead of thinking about the concepts behind the math. And it is the concepts that are important.

Take Runs Created, for example. Bill James created the formula in the mid-80s, trying to answer a basic question: how many runs did a player create? And the basic conceptual formula is simple: (A x B)/C in which A is the on-base factor, B is the advancement factor, and C is the opportunity factor. After some math, the formula can be expressed as OBP x TB. That's it, on base percentage times total bases, and it could predict a team's actual runs scored within about 5%. Simple, elegant, accurate.

Over the years, we've refined RC and it is the building block of WAR. Know what the formula looks like today?

via upload.wikimedia.org

Just kidding. That's what it looked like in 2001. In 2002, the formula was re-calculated to look like this:

via upload.wikimedia.org

In which:

A = H + BB - CS + HBP - GIDP

B = (1.125 x singles) + (1.69 x doubles) + (3.02 X triples) + (3.73 X home runs) + .29 x (BB - IBB + HBP) + .429 x (SH + SF + SB)

C = AB + BB + HBP + SH + SF

Sure, the formula is slightly more accurate, but who honestly cares? Was that extra bit of accuracy worth all of that complexity? The formula went from something I could do on the back of an envelope (OBP x TB) to this unholy beast. And we've lost sight of the concept: value is determined by getting on base multiplied by power.

So what makes a good stat? I think that question misses the point slightly. It's what makes a good concept. It's not the answers that should be important to analytics, but the questions themselves. It's why Football Study Hall's article on the Five Factors was so great. Really, go read it if you haven't and if you have, read it again.

That article works even if you remove all of the numbers. He lays out the concepts, and then shows why they are important. Even better, all of those fancy stats are available to you for free. But you don't need to calculate yards per point to understand the concept of efficiency coupled with explosiveness. You could never look at that stats page, but still be a more knowledgeable fan just by understanding the concept.

Advanced stats, especially ones difficult to calculate on your own, certainly have their place. But I think the analytics crowd sometimes loses their audience. You don't convince people by calling them stupid, for starters. You are using the flashlight to show the path, not club them upside the head with it. But even if a "normal" fan never once looks at the advanced metrics, his or her enjoyment of the game can be increased simply by being introduced to the concepts underlying these stats. That's the important part. I can't give a universal "good stat" seal of approval for everyone, but this is what I look for:

What is it measuring?

This is the most basic test. We like to use ATVSQBPI around here, and if you ignore all of our calculations behind it (which aren't that complex), you can understand it by this simple explanation: how many yards is a QB worth each time he calls his own number? Pretty simple, eh? You look at the number and can understand that it is how many yards he is worth per play. Even the scale makes sense.

Is the data public?

If you're formula is proprietary or uses data that is private, I'm out. I need to be able to recreate your results. I need you to show your work. The guys at Baseball Prospectus have done some great work, but I started to lose interest in them as too much of their alphabet soup was behind a paywall and the formulas were not public. At that point, I'm taking your word for it, and honestly, I don't trust you. I don't even know you.

Does it make sense?

I get that sometimes the numbers reveal things our lying eyes keep from us. But observation matters, and when the data doesn't jibe with observation, there better be a good explanation as to why. You can't just waive it away with confirmation bias. You have to convince me your stat is showing me the "actual" truth and why my observations are inaccurate.

Does it apply?

One of the biggest concepts in baseball stathead circles is replacement value. It is the foundation of WAR, and a lot of current analysis. It is also completely worthless to a college baseball fan. The reason is pretty obvious: there is no such thing as a replacement player in college. While a MLB team can cut a player and pick up the mythical replacement player on the waiver wire, a college team is largely stuck with what they have got. They can get a transfer, but must wait a year. You can recruit players, but again, you have to wait. I guess you could hold open tryouts and get a walk-on, but that sets replacement level somewhere around zero.

Does it increase understanding?

This goes more to usage, because at the end of the day, a number is just a number. It's what you do with the number that counts. One of the dangers of a value stat is that it reduces discussions on players down to "well, this guy is worth 4.2 and the other guy is 3.5, so clearly the guy with the 4.2 is better." That's not analysis, that's just regurgitating a stat. A player's production is not a straight line. A facilitating point guard provides different value than a hulking power forward. A speedy slick-gloved shortstop has different value than a power hitting 1^st baseman. Reducing them to one scale and one number removes those shades and makes the game less interesting. A hammer is not better than a screwdriver, unless you need to hammer a nail. But to build something, you probably will need both tools. One is not more "valuable" than the other, the value they provide is simply different.

I love that we live in this era of nearly limitless data. It's a great time to be a sports fan, but it's important to note that there is no right way to watch the game. Also, we live in an era in which there are so many ways to analyze the game, we can actually use our preferred tools for analysis. However, let's remember they are just tools. You can give me a chisel, and I still couldn't craft the Rape of the Sabine Women.