NFL Intelligence

Overview

The NFL shows live analytics from sensors in every player’s shoulder pads. This project gets the same kind of picture from the film alone: point it at the NFL Pro player and it puts every player on a top-down field in real yards, works out who is who, reads the formation and the coverage, and runs a “play value” with a plus/minus for the pass the quarterback could throw right now.

Six plays from two games, as the pipeline saw them. The film itself isn’t shown here; the top-down view and the panel are what it produces from it.

The challenge

Film has no sensors in it. Three things had to come from pixels before any football could be read: where the field is, who the players are, and what play this is.

The field is painted, which is enough. The pipeline finds the yard lines and hash marks, fits a level pan/tilt/zoom camera to them, and follows that camera frame to frame; every foot on screen becomes a point in yards. To pin those yards to the real field it lays the field flat through the camera model and reads the big painted numbers, whose arrows point at the nearer goal line.

The players come from a detector that loses people constantly: linemen in a pile, a corner pressed on his receiver, the quarterback behind the line. Fine-tuning it on 275 frames I corrected by hand helped; it did not make the problem go away. Tracking in yards with a Hungarian assignment, carrying a hidden player along with the one he is hidden behind, and giving a lost player his role back when he reappears made that survivable.

The play comes off the screen: on-device text recognition reads NFL Pro’s play header and on-field roster, and the header identifies the play’s row in the public play-by-play, which brings the real yard line, personnel, and afterwards the result.

Approach

Roles are named at the snap from where everyone stands, side of the ball first and jersey colour second, then corrected by what players do. The catch and yardage models behind the value are trained on the NFL’s own tracking data (Big Data Bowl 2026) and the plus/minus compares the play with a typical one from the same stretch of field at the same moment after the snap.

Every claim is measured. Role assignment is marked against 2,572 real snaps; formation reads are marked against FTN’s official charting; the snap and the value were checked against the frames by eye, and two early results that turned out to be artifacts were corrected and written down.

Outcome

On three games of film (day and night, 78 runs and passes identified), against official charting. The third game is one that no model here was trained on.

                                            first two games   third game
quarterback picked out                      42 of 42          23 of 24
passes given a live value                   24 of 24          10 of 12
backs in the backfield, within one          42 of 42          24 of 24
defenders in the box, within one            41 of 42          19 of 24
pass rushers, within one                    23 of 24          10 of 12
field anchored from the painted numbers     57 of 66 over all three
throughput, 3024x1654 capture, Apple M5     ~17 fps

The detector was fine-tuned on 275 frames corrected by hand: players found went from 80.5% to 91.1% on held-out frames. Against positions clicked by hand on five plays (1,531 player positions), a tracked player is 0.72 to 0.98 yards from where he stood, most of it a steady offset between the bottom of his box and his feet (0.32 to 0.37 yards without it), with 90% of players found. One of the five is from the game nothing was trained on, and it scores like the rest.

The value models on held-out weeks: catch AUC 0.76, calibrated within 3 points; throws to the model’s top option gained 12.0 yards, its third-or-lower 5.8. 10,700 lines of Python, 195 tests.

Still open: a player does not keep one identity through a pile (20 to 43 changes on the five labelled plays, against a target of 2), the throw can’t be timed from depth alone (within 0.6 s only 69% of the time), and the end-zone camera angle isn’t mapped. Five hand-labelled plays are a small sample.

What I learned

Measure before believing. The first version reported that the model’s best throw was the receiver who caught the touchdown; it was a coasting track on a receiver the detector had lost. The second version found the snap three seconds early because the huddle breaking looks like a snap. Both were obvious in the frames and invisible in the numbers.

Profile before optimising. The detector was assumed to be the slow part; image resizing of the 4K capture was costing more. Replacing it took the pipeline from 10 to 18 fps and moved nothing else.