The intersection between art and technology is rarely more spectacularly on display than at a modern concert. Today’s audiences can expect to be wowed with the latest innovations in moving lights and high-resolution video, but these wonders are nothing without the programming or content to make them come to life. Innovation and inspiration can come from almost any source in our industry, and it follows that the intersection of augmented reality (AR) and facial recognition algorithms found in incredibly popular mobile apps like Snapchat, along with platforms like Facebook and Instagram would be a natural fit for the world of event production.
Enter Game Changer MVP, a Kansas City-based company specializing in digital multimedia and post-production with an emphasis on live events. “We made a move into interactive gaming about two years ago, and this was the first real product offering that people really connected with,” says Matt Coy, chief operating officer at Game Changer MVP and veteran sports presenter, referring to the Filter Fan Cam, a technology that allows AR-style filters to be overlaid on fans at live events in real time.
“It’s a take on the facial recognition that’s ubiquitous these days in the world of social media,” Coy adds. “I found myself very frustrated with the fact that we couldn’t find a way to utilize facial recognition technology on large-format event screens, especially since they’re in widespread use at almost any entertainment scenario. Not hearing back from the companies that did AR and face recognition led me to think of another way to make it work, so that we could bring the excitement of this technology to any large-scale event.”
Resolving to do something about this, he got together with his business partner Sean Hopkins, Game Ghanger MVP’s CTO, and developed the Filter Fan Cam, which Matt says uses the same facial-recognition software as the social media applications we’re all familiar with.
“The twist I introduced was to fashion our software so that live camera feeds at any type of event — be that sports, music, corporate conferences — can be fed to our software and sent back to the switcher with no delay, and add live, real-time 3D elements using facial recognition. Like the filters in popular social media software, custom elements can be created in very short turnout time — typically within a few days — and then added on top of the spectator’s faces in the live camera feed.”
Coy saw opportunity in the fairly static screens he saw at set changeovers and other intermissions at events where only a logo was displayed. “Sports events, typically, have their programming pretty locked-in, but music events are just now getting to the point of having real interactive content, commercials, and things of that nature during the down time between the performances. We’re trying to promote this time as something that can be fun, something that engages the fans, and it’s also something that can be sponsored, like the iHeartRadio Jingle Ball tour content we’ve done.”
Crowd-Pleasing Promos
That content, he explains, is a joint collaboration between Game Changer MVP and iHeartRadio, to promote the upcoming film Spider-Man: Into the Spider-Verse, the computer-animated film released Dec. 1, 2018 by Marvel Entertainment and Sony. In the film, a twist on the classic Spider-Man comics, character Miles Morales discovers he is one of several super heroes who share Spider-Man’s capabilities. To showcase that theme, Matt and the minds at Game Changer MVP created a filter that would transform the audience members at Jingle Ball into the iconic Stan Lee superhero for a few moments at a time.
“We custom-created the Spider-Man masks in 3D, which the facial recognition software overlays onto the audience’s faces, after which it can be sent to the I-Mag stream and shown in real-time, without noticeable lag or jumpiness,” explains Coy, adding that Game Changer MVP can create almost any content imaginable. “We can make filters for whomever, be that a big music festival like the Jingle Ball tour, musical or theatrical touring productions, sports teams and sporting events, anybody who wants to use this technology as a way to really engage the audience and bring them closer to the action.”
For the Spider-Man filter, Game Changer MVP wanted to make sure that everyone in the shot would have a chance to see themselves in the famous superhero mask. “The mask element that gets overlaid is entirely 3D, and can be shown from a variety of angles. There’s a lot of play in the 3D elements — you have about 70° of rotation on the faces 40° of vertical tilt, and the software tracks with the audience member’s face as it turns and moves,” Coy explains. “We programmed the filter to display the mask on up to 15 faces in every live camera shot, and included a bunch of the variations of the Spider-Man masks — as are shown in the movie. One goal we achieved was to show all the various masks together in a single shot to highlight the “multiple Spider-People” aspect of the new film. There are multiple options for the operators each night — for instance one iteration of the filter includes just the traditional red mask, and another includes just the special ‘hooded’ Spider-Gwen / Gwen Stacy mask, and a few that randomly pick from the entire selection of masks and applies them to the faces in the shot. The directors really have a real variety of looks to choose from each night.”
Coy says the Filter Fan Cam software intelligently picks faces out of the I-Mag feed, and has enough resiliency that a hat or pair of glasses aren’t likely to fool it. “Sometimes if you get extreme shadows or glare off of sunglasses, you can trick the system into not seeing a face, but it usually works pretty well. Game Changer MVP has a lot of sports team clients, and we’ve been using this system at outdoor events for a while now, where there are lots of shadows from ball caps and bright sunlight, and it works pretty well in those adverse conditions.” Nevertheless, the system is designed to fail elegantly when given a stimulus it doesn’t understand, or loses tracking. “As soon as someone turns more than 70°, or puts a beer can up to their face, it just pops off and starts looking for the next person.”
Easy to Deploy
When it comes to operation, Coy says the software is incredibly intuitive and simple, with minimal hardware requirements, a view echoed by the operator traveling with the iHeartRadio Jingle Ball Tour, Melissa Gonzales.
“From my end, it’s really simple — the software is really easy to use. Setup typically takes me about 10 minutes. Hardware-wise, there isn’t much to speak of. We have a stock laptop configured for gaming to deal with the 3D workload, and an input / output box that connects via USB. We get an I-Mag feed from the director’s rack, and the output goes straight back to the video director, where they can choose which shots to send to the screen as normal. There’s no human intervention required to select individual people or anything like that — the system is very good at recognizing faces and selects the clearest ones automatically,” says Gonzales. Like Coy, she’s equally confident in the software’s resiliency. “From my experience, there’s not much that the software can’t deal with. If they’re trying to take a selfie, for instance, and there’s a giant cell phone covering half of their face, the filter software will obviously not work with that. The shots need to be reasonably zoomed in on faces for the algorithm to pick up on, but those are easy requirements for the camera operators and the director to deal with.
“Audience wise, people get really excited when they see themselves on the screen, and now even more so with the filters in place,” Gonzales adds. “It’s a new thing that they haven’t seen before, and it tends to get the rest of the crowd really excited to participate. Who doesn’t want to be Spider-Man, if only for a few moments?”
What’s next for Game Changer MVP? Among other things, Coy says he wants to see the commercials that typically get played on the big screens during set changes get a facelift.
“Lots of festivals and events have ‘commercials’ that are just prerecorded, which can get repetitive and boring, especially after seeing them on a loop at a festival dozens of times. We want to convince the people putting on these events to do something more engaging, like a sponsored mobile game tie-in or real-time polling.” he tells us. “We want to move into a world where the audience can actually participate in real-time 3D games on the big screens at these events on their phone — the technology is already there to craft these incredibly detailed moments where people can play along. Imagine a scenario wherein you could increase fan participation with a brand by having a sponsored trivia game where whoever wins gets a prize, and everyone who finishes in the top ten gets a discount code they can redeem. We just want to take that in-between time and make it more engaging and less boring. As a fan, any time I can get free stuff just for playing a game at an event where I wouldn’t be doing anything else anyway, that’s a win for me. The possibilities are really limitless.”
Along with iHeartRadio’s Nov. 27-Dec. 16 Jingle Ball tour, Game Changer MVP’s Filter Fan Cam has been seen at a wide array of major sporting events. For more information, visit www.GCMVP.com.
