Next Gen Stats: Intro to new Route Recognition model

Conventional counting stats like receptions and receiving yards provide a way to measure an individual player's ability to catch and move the football, but they only tell part of the story. Advanced stats like depth of target, separation window and completion probability provide greater insight, but they still leave out an important factor. Namely, which route did the pass catcher run to get open before catching the ball?

With the help of player-tracking technology, the Next Gen Stats Analytics team set out to answer that exact question, decoding one of the key elements of an offensive play call by using player-tracking data to measure which routes pass catchers are running on any given pass play.

Last month, we revealed a new set of rushing metrics derived from the ability to calculate Expected Rushing Yards. Today, we are introducing another new machine-learning tool: the Route Recognition model, which classifies routes by type, in real time, with the help of player-tracking data.

Let's dig into the methodology behind the Route Recognition model:

How the model works

The Next Gen Stats player-tracking system records the x-y location, speed, acceleration, direction and orientation of all 22 players on the field in real time. Our new Route Recognition model leverages this data as inputs into a model that assigns a route type to every eligible receiver on every pass play, including tight ends and running backs. Our architectural approach uses a combination of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks trained on Amazon’s SageMaker platform. CNNs allow us to engage with the spatial nature of our dataset (that is, where each player is on the field in a given play), while LSTM networks allow us to engage with the temporal nature of our dataset (what happens as the play develops over time).

We approached routes run by players aligned in the backfield separately from routes run by players aligned out wide, in the slot or tight, because of clear differences in route archetypes. Below are the 15 unique route types assigned to all route runners, based on their location when the ball is snapped. Note that while NFL playbooks have hundreds of variations of routes, we've narrowed it down to these high-level categories, including 10 routes for those in typical wideout alignments and five for those aligned in the backfield:

Wideout Routes (10): Screen, flat, slant, crossing, out, in, hitch, corner, post, go
Backfield Routes (5): Screen, flat, angle, out, wheel

The model was trained and validated against all routes from every passing play from 2018 and '19, including both the regular season and the postseason. All route runners were included, regardless of whether or not they were targeted; given that we could not find a difference between the shapes of targeted routes and non-targeted routes, we saw no reason to train on only targeted routes. In total, the wideout model was trained on over 100,000 routes, while the backfield model was trained on over 15,000 routes.

To avoid noise in the data from broken plays (during which pass catchers often stop running their assigned routes) and player movement after the catch (which would not tell us much about the efficacy of any given route), all routes were capped at either the moment the ball was passed forward or at a given time (4.6 seconds after the snap for wideout routes and 4 seconds after the snap for backfield routes) -- whichever came first. The optimal time limits for these route types were determined by analyzing how pass attempts played out over the past two seasons; for reference, 4.4 seconds accounted for the 75th percentile of all pass attempts by time to throw in that span.

Here's a visualization of route paths sorted by our predicted route type:

For training purposes, the tracking data for the wideout model has been normalized such that all the pass catchers are to the left of the quarterback, with the rationale being that the route paths are symmetrical. We found the shape of the routes aligned with our expectations of route paths; no egregious model predictions seem apparent.

Future iterations of the model will look to delve deeper into the route tree to account for the nuance of route-running at the pro level.

What we can learn

Real-time route classification enables us to contextualize the passing game in new ways. We can study league-wide trends to gain a new understanding of offensive strategy and tendencies, and we can break down and rank individual players by advanced performance metrics.

The table below combines our more descriptive NGS receiving metrics with the results of our Route Recognition model. The values corresponding to each route represent league averages over the last two seasons. Only wideout routes are included (i.e., players aligned wide, in the slot or tight):

Wideout Tendencies by Route, NFL Average, 2018-19 Seasons

• The average pass catcher runs a go route on nearly a quarter of all routes (22.3%), the highest percentage of any route type in our data. However, those routes are targeted roughly 1 out of 10 times (10.8 percent), the lowest target rate of any route.
• The WR screen is the least-run route (3.4%), and it's the only route where the average target is behind the line of scrimmage. But it's also targeted at the highest rate (40.7%) and early in the play (1.6 seconds average time to throw).
• The most targeted routes outside of the WR Screen? The out (27.8%) and slant (25.2%) routes are the next most popular across the league.

How often a pass catcher runs a route can give us insight into strategy and tendencies at the league-wide, team and individual levels of the game. Evaluating route types by advanced performance metrics can tell us which routes are the most valuable on a per-target basis, as you can see in the chart below.

NOTE: EPA/target is expected points added per target; this measures the value of individual plays in terms of points comparing the down, distance and field position situation at the start of the play relative to the end of the play. CROE is catch rate over expectation, which measures performance relative to completion probabilities.

Production by Non-backfield Routes, 2018-19 Seasons, NFL Average

• The most valuable routes by expected points added per target were the post (+0.48) and corner (+0.43) routes. The go route (+0.19) ranked seventh on the list of 10 route types. One possible reason for this: It's harder to separate on go routes, which put the player on a straight path, than on posts or corners, which ask the player to make a cut. Targeted pass catchers on posts and corners average 2.4 yards and 2.3 yards of separation from the nearest defender, respectively, while pass catchers targeted on go routes average just 1.8 yards of separation.
• Target location on go routes has a dramatic impact on the average net value of the play. Go routes targeting a pass catcher outside the numbers average +0.13 EPA per target, while targeted pass catchers inside the numbers average +0.42 EPA per target. Over the last two seasons, pass catchers running go routes were targeted outside the numbers four times as much as they were targeted inside the numbers.
• The top three routes by catch rate (screen, flat and hitch) were the least valuable routes by EPA per target. As we found in our analysis of our completion probability model, there is a strong negative correlation between catch rate and air yards -- which indicates targets closer to the line of scrimmage are not as valuable on a per-target basis as deeper targets.
• It's worth noting four of the five most valuable routes by EPA per target are in-breaking routes: post (+0.48), in (+0.31), cross (+0.27) and slant (+0.26).

Contextualizing routes at the league-wide level gives a macro look into the value of route classification. At the individual player level, we can learn more about the micro nuances of route running.

Route versatility

Which wide receivers run the most diverse route tree relative to the average player? With our new route-classification model, we can evaluate which receivers are essentially the most predictable -- or different from the average. We calculate route versatility by averaging the sum of the absolute difference between a player's route percentage and the average NFL receiver's for each of the 10 route types (see the last paragraph of this article for further explanation).

The top five and bottom five most versatile route runners from the 2019 season among 72 wide receivers with at least 300 routes:

Wide Receiver Route Versatility Rankings, 2019 Season (min. 300 routes)

• In his second season as a pro -- and first in head coach Kliff Kingsbury's offense -- the Cardinals' Christian Kirk ranked as the most versatile route runner of the 2019 season. Kirk was targeted at least 13 times on five different routes: hitch (28 targets), crossing (17), WR screen (16), go (13) and out (13). While Kirk ran every route in the tree, it's important to note 77 percent of his routes came on the right side of the formation.
• Ted Ginn Jr., now a member of the Chicago Bears, ranked as our least versatile route runner of 2019 among qualifying wide receivers. Ginn ran a go route more often than any receiver in the sample (42% of routes). Ginn's production on those routes in 2019: 9 targets, 1 reception for 25 yards (targeted only 6% of the time).
• Not listed in the bottom five, but relevant based on his reputation: Seahawks receiver DK Metcalf ranked 66th out of 72 receivers by our measure of route versatility. Metcalf ran a go route at the second-highest rate of qualified receivers on this list (38% of routes), behind only Ginn (42%).

This only scratches the surface of the analysis possible with our route recognition. Who were the best-performing wide receivers by route type? NFL.com's Nick Shook takes a look at the top receivers of 2019 by route type.

-- Mike Band, Next Gen Stats Analyst. Follow Mike on Twitter @MBandNFL.

Route versatility calculation explanation: If Player A runs a go on 25 percent of routes, a hitch on 19 percent and an out on 12 percent, and the NFL averages are 22 percent, 18 percent and 10 percent respectively, the absolute difference from the average across those three routes would be 3 percent, 1 percent and 2 percent.