Around this time last year, I wrote a series of posts introducing the Weighted WAR framework for evaluating position players. At the time, I specifically excluded pitching value from consideration. At the risk of being a bit self-indulgent, I’ll quote my own post from last year on this topic:
“The question still remains: which form of WAR to use? This
is a thorny question for pitchers, because unlike the batting and fielding
components of different WAR systems (which may differ in outcome but are
generally trying to accomplish the same thing), pitching WAR varies
philosophically from system to system. To hugely simplify a complicated topic,
Baseball Reference WAR starts from pitcher runs allowed and makes a blanket
adjustment for the general quality of the team’s fielders. Fangraphs WAR uses fielding
independent numbers (strikeouts, walks and home runs) to measure a pitcher’s
entire contribution and assumes any variance outside of that is attributable to
the team’s fielders. And RA9 WAR (also presented by Fangraphs these days, and
used in other sources as well, including Seamheads for NeL stats) takes the
pitcher’s park adjusted runs allowed and ignores any fielding effects entirely.
I don’t particularly like any of the freely available WAR
systems for pitchers; I think they all simplify the pitcher-fielder dynamic too
much and produce extreme outcomes as a result. As such, both because of the
extra work that would be involved and a lack of faith in a reasonable outcome,
I’m not considering pitching value as part of this project. Again, this is by
no means intended to imply that pitching value shouldn’t count; I may circle
back to this topic in the future.”
Welcome to “in the future.” In my next several posts, I’ll
be introducing a new method of evaluating starting pitchers, one which I think
provides a compelling alternative to any of the readily available WAR systems.
But before we get into that, it’s probably a good idea to further explore those WAR
systems and explain why I think an alternative is worth introducing to begin
with.
As mentioned above, there are three versions of pitching WAR
that can be easily found on major baseball websites. I’ll take them on in
reverse order of my own preference.
My least-favorite option is also the simplest: RA9 WAR. RA9 stands for runs allowed per 9 innings; it’s similar to the more-familiar
ERA, but with unearned runs included (a choice that we’ll discuss later in the series, but one that I agree with). RA9 WAR simply park adjusts the
pitcher’s rate of runs allowed (different sources may use different styles of park adjustment, either general to the team or specific to the pitcher), then compares it to the league’s replacement
level for the year. Outside of the park adjustment, no other modifications to the pitcher’s performance are made. In
particular, there is no attempt to separate the pitcher’s performance from
the contributions of the fielders behind him (whether helpful or harmful), or
from his bullpen’s success or failure in stranding runners who were on base when the pitcher was
removed.
To take an extreme example: Say the starting pitcher records
two outs in the seventh inning, then strikes out the next hitter, but the ball
gets away from the catcher and the hitter reaches base. The starter is pulled
from the game, and the new pitcher immediately allows a line drive into the gap. The center
fielder has a chance at the ball, but can’t quite reach it; the right fielder
picks it up and tries to throw out the runner at home, but his throw is
slightly up the line and the runner scores. At minimum, five players
contributed to allowing that run, and one could argue that the starter bears the smallest share of that responsibility. But the official stats charge him with
the run, and therefore so does RA9 WAR.
While most runs allowed are not this heavily skewed away from the pitcher’s own contributions, it’s still obvious that the efforts of the fielders and relievers are crucial pieces of context for the starter’s performance. If you compare Jim Palmer and Rick Reuschel’s 1974 seasons, but fail to account for the fact that Palmer was pitching in front of brilliant fielders such as Mark Belanger and Brooks Robinson and Paul Blair while Reuschel suffered through the butchery of Bill Madlock and Rick Monday, you won’t get the full picture of their relative effectiveness. Ignoring fielding and relief work entirely strikes me as eminently counterproductive.
Next up, we’ll go to the other extreme with FIP WAR, primarily presented on Fangraphs (and therefore usually abbreviated as fWAR). Per Fangraphs’ own explanation of their system, fWAR uses five components on a per-inning basis: home runs, walks, batters hit by pitch, strikeouts, and infield popups. (Walks and HBP are treated as equivalent, as are strikeouts and popouts.) The theory behind this is that these components are the best representation of the pitcher’s actual ability, as they are the events for which the pitcher is most directly responsible. As such, they also tend to have better year-to-year correlations than rates of hits or runs allowed.
As a predictive measure of pitching ability, FIP is almost
certainly preferable to RA or ERA; it does in fact correlate better
year-to-year, both with itself and with future ERA. But WAR is not generally
advertised as a forward-looking projection system; it’s presented as a measure
of backward-looking value. A pitcher who posts a solid FIP but also allows a
high number of line drives to remote areas of the park might still have a
bright future, but the actual hits and runs he allows still hurt the team.
FIP is also overly reductive in that it ignores less-measurable but still existent pitcher abilities, including things like wild pitch/passed
ball tendencies, ground ball rate, and performance with runners on base (some pitchers work better from the stretch than others). Just to pick one, let’s focus on the
pitcher’s own fielding, specifically with regard to four-time Gold Glove winner Mark Buehrle. Buehrle was one of the
best-fielding pitchers in the game on batted balls, and also a master of
controlling baserunners. He allowed 59 total stolen bases over his 16-year
career (less than four per season); by comparison, he picked off an even100 runners
(just over six per year). Given that FIP by definition ignores the pitcher’s
own fielding, it is not a coincidence that Buehrle’s career RA9 WAR (59.9)
exceeds his fWAR (52.3) by a substantial margin, and this is clearly a case in
which fWAR gets it wrong. There are enough similar errors (in both directions) to
reduce the utility of fWAR for purposes of historical evaluation, which is my
primary interest.
That brings us to Baseball Reference WAR (generally called
bWAR), which is my preference among these options. Like RA9 WAR, bWAR starts
with the pitcher’s park adjusted runs allowed. However, it then adjusts for the
quality of the fielders behind the pitcher based on his team’s performance in
the fielding metric used in position player WAR calculations (which changes
depending on the level of data available for the year in question). This effect is calibrated for the
number of balls in play the pitcher allows; it will typically be worth between
0.5 and -0.5 runs per 9 innings on all but the most extreme teams.
Sounds great, right? Unlike fWAR, it doesn’t ignore actual
runs allowed, but unlike RA9 WAR, it accounts for the team’s fielders. And in practice, this usually works reasonably well; there’s a reason it’s my preferred option
among the three WAR systems. But just as with the other two, there are edge
cases that cause trouble. The primary problem here arises from the assumption
that all of the pitchers on a team get the same level of support from the
team’s fielders – an assumption that is not ridiculous on its face, but can be
cast into doubt by a few examples.
In this case, let’s look at a quartet of Texan aces in 2019:
Justin Verlander and Gerrit Cole of the Astros, and Mike Minor and Lance Lynn
of the Rangers. All four put up very good seasons, and all four received Cy
Young votes. By conventional numbers, Cole (20-5, 2.50 ERA) and Verlander
(21-6, 2.58) were solidly better than Lynn (16-11, 3.67) or Minor (14-10,
3.59). Unsurprisingly, given the difference in their ERAs, RA9 WAR prefers the
Astros pair as well; the gap narrows somewhat thanks to Texas’s status as a
hitter’s park, but the AL’s top four still goes Verlander 9.0, Cole 8.0, Minor
6.5, Lynn 6.3.
Houston, however, was a much better team than Texas (or
almost anyone else; the Astros won 107 games), so it’s not a surprise to learn
that they had better fielders. You might think, then, that FIP would close the
gap for the Rangers. And it does… for one of them. Lynn’s FIP of 3.13 is a
marked improvement on his 3.67 ERA, while Verlander’s 2.58 ERA balloons into a
3.27 FIP. Cole, on the other hand, posted a 2.64 FIP (thanks to an absurd 13.8
strikeouts per 9 innings), very comparable to his ERA. And Minor? His 3.59 ERA
climbs to a 4.25 FIP. As such, fWAR has a top 3 of Cole-Lynn-Verlander, and
Minor doesn’t show up until the #8 position.
And what does bWAR make of this group of pitchers? Well, as
noted, bWAR evaluates Houton’s fielders as very good (bWAR’s fielding
adjustment for Cole is +0.46 runs per 9) and Texas’s as very bad (Lynn’s
adjustment is -0.30). So, starting from the baseline of RA9 WAR and accounting
for quality of fielding, bWAR gives a result of Minor 8.0, Lynn 7.7, Verlander
7.4, Cole 6.7.
This is a bit surprising, given the other results, so what’s
going on? As noted, B-R gives Minor substantial credit for pitching effectively
in front of a bad defense. As a sanity check for this, let’s look at batting
average on balls in play (BABIP for short). The AL average BABIP
in 2019 was .299; the Rangers allowed a mark of .314, 15 points higher than
average and consistent with a poor fielding team. (By contrast, Houston allowed a .272 BABIP.) Lance Lynn’s BABIP
allowed was slightly worse than his team’s, at .322; adjusting for this seems
totally fair in his case. As for Minor? His BABIP allowed was .288 – lower than
the AL average. There are three possible interpretations of this. First,
hitters had a tougher time hitting Minor and his balls in play were easier for
the Rangers’ subpar fielders to handle. Second, the Rangers’ fielders performed
better with Minor on the mound than they did otherwise, for whatever reason. Or
third, both factors had an effect, and the relative extent of those effects is
unknowable.
Any time a discrepancy of this type arises, bWAR uses the first interpretation. As such, it is also intermittently prone to startling swings toward or against certain pitchers. And that is how you get 2019 Mike Minor (3.59 ERA, 4.25 FIP, .288 BABIP) rated substantially ahead of Gerrit Cole (2.50 ERA, 2.64 FIP, .276 BABIP). For an even more extreme example, Aaron Nola in 2018 allowed a .254 BABIP for a team that bWAR thinks was below average in the field by 0.41 runs per 9 innings; as a result, bWAR rates Nola’s admittedly very good campaign as the single best season any pitcher has had in the last 15 years, narrowly outpacing Jacob deGrom’s 1.70 ERA/1.98 FIP effort the same year.
Where to go from here? We have three options for pitching
WAR, but all three seem to have the same failure point: the pitcher-fielder
interaction. In my opinion, all three systems handle this interaction too
rigidly by either by ignoring it entirely, assuming the pitcher can only affect
a few specific outcomes, or assuming
that all pitchers on the same team have the same fielding performance behind
them. It would probably be possible to amalgamate all three systems into some
sort of WAR hydra, allowing each system’s extremes to be smoothed out by the
other two.
Or we could abandon the WAR framework entirely and do
something completely different. Next time, we’ll start to explore the basis for
a new option.
