Jayson Stark's "Reasoning"

There are two possibilities for what Stark did in his columns (and probably his book):

1) Ask a bunch of scouts who they think is overrated or underrated

2) Dream up something off the top of his head and stick to his guns

In essence, this was a combination of (1) and (2). We can tell this by Stark's use of arbitrary statistics, such as counting statistics up to a certain age, career statistics for two players of the same age who entered the league at completely different stages of their development, pitcher wins, and batting average. Sometimes, he'll use some genuinely useful statistics, but that's only when they tend to support his view. Comparing the major league career statistics of Jose Reyes and Hanley Ramirez, though, is quite dishonest.

Another of the systematic flaws in Stark's reasoning is the failure to understand aging in baseball. The old standard age for peaking in performance was 27 at one point, but this has gotten closer to 30 or so, thanks largely to more sensible training methods, longer pitching rotations, etc. Further, different clubs will have different reasons for playing players in the majors before a certain age; Santana didn't enter the Twins' rotation full-time until he was 25 while Sabathia jumped right in at 20. The rate statistics, namely strikeout rate, walk rate, and home run rate statistics are important to consider here, but they must be observed in the proper context. Stark, though, compares ERA and BA of players that are of different ages and started at different points in their careers in different situations.

Perhaps the most striking failure is to have an objective reference point for which players are good and which are not. The criteria for what makes each player particularly good appears to be all in Stark's mind. The same is the case for determining which players are overrated and underrated.

***

The problem for Stark is not that he is using subjective analysis - it is very difficult if not impossible to make a truly objective determination of who is over and under rated - but he is certainly far too biased in his procedure. The evidence he uses is shaky in far too many examples. What is most egregious, though, is that he lacks any sort of consistency, other than trusting the opinions of scouts and bending the evidence around his own preconceived notions.