Players and coaches for the Philadelphia Eagles and Kansas City Chiefs will spend hours and hours in film rooms this week in preparation for the Super Bowl. They’ll study positions, plays and formations, trying to pinpoint what opponent tendencies they can exploit while looking to their own film to shore up weaknesses.

New artificial intelligence technology being developed by engineers at Brigham Young University could significantly cut down on the time and cost that goes into film study for Super Bowl-bound teams (and all NFL and college football teams), while also enhancing game strategy by harnessing the power of big data.

BYU professor D.J. Lee, master’s student Jacob Newman and Ph.D. students Andrew Sumsion and Shad Torrie are using AI to automate the time-consuming process of analyzing and annotating game footage manually. Using deep learning and computer vision, the researchers have created an algorithm that can consistently locate and label players from game film and determine the formation of the offensive team — a process that can demand the time of a slew of video assistants.

“We were having a conversation about this and realized, whoa, we could probably teach an algorithm to do this,” said Lee, a professor of electrical and computer engineering. “So we set up a meeting with BYU Football to learn their process and immediately knew, yeah, we can do this a lot faster.”

While still early in the research, the team has already obtained better than 90% accuracy on player detection and labeling with their algorithm, along with 85% accuracy on determining formations. They believe the technology could eventually eliminate the need for the inefficient and tedious practice of manual annotation and analysis of recorded video used by NFL and college teams.

Lee and Newman first looked at real game footage provided by BYU’s football team. As they started to analyze it, they realized they needed some additional angles to properly train their algorithm. So they bought a copy of Madden 2020, which shows the field from above and behind the offense, and manually labeled 1,000 images and videos from the game.

They used those images to train a deep-learning algorithm to locate the players, which then feeds into a Residual Network framework to determine what position the players are playing. Finally, their neural network uses the location and position information to determine what formation (of more than 25 formations) the offense is using — anything from the Pistol Bunch TE to the I Form H Slot Open.

Lee said the algorithm can accurately identify formations 99.5% when the player location and labeling information is correct. The I Formation, where four players are lined up one in front of the next — center, quarterback, fullback and running back — proved to be one of the most challenging formations to identify.

Lee and Newman said the AI system could also have applications in other sports. For example, in baseball it could locate player positions on the field and identify common patterns to assist teams in refining how they defend against certain batters. Or it could be used to locate soccer players to help determine more efficient and effective formations.

The BYU algorithm is detailed in a journal article “Automated Pre-Play Analysis of American Football Formations Using Deep Learning,” recently published in a special issue of Advances of Artificial Intelligence and Vision Applications in Electronics.

“Once you have this data there will be a lot more you can do with it; you can take it to the next level,” Lee said. “Big data can help us know the strategies of this team, or the tendencies of that coach. It could help you know if they are likely to go for it on 4th Down and 2 or if they will punt. The idea of using AI for sports is really cool, and if we can give them even 1% of an advantage, it will be worth it.”