Using Motion Capture to Create Perfect Communication

The Animation Capture & Effects Lab (ACE-Lab) at Drexel University’s Westphal College of Media Arts & Design looks, on the surface, like a training program for VFX-led entertainment industries.

Drexel, however, prepares students, not only for visual effects applications that exist right now, but also those that are coming up five or even 10 years down the line.

The department’s news page is full of headlines about alumni working on high-profile projects such as Star Wars and Frozen II, but the ACE-Lab takes its students down less well-trodden paths too. In fact, it’s had a wide-ranging mission from the outset.

ACE-Lab resides in Drexel’s Digital Media Department, founded by Dr. Glenn Muschio, a doctor of anthropology. That anthropological influence is part of the lab’s DNA. “From the very beginning, it was all about communication,” says Nick Jushchyshyn, Program Director for VR & Immersive Media. “It just happens to be that entertainment is a really fun form of communication. But leveraging these technologies in teaching venues, in training and other experiences, has always been an integral component to our programs.”

EARLY ADOPTERS

Motion capture is a core part of the department’s offering. ACE-Lab was an early adopter of Vicon’s Vantage cameras, proud that its entertainment setup was one of the first Vantage installations in the US. The lab upgraded their T-Series system when the department moved to a larger 40 ft x 40 ft stage, complete with a huge green-screen cyclorama. “The big deal for us was covering our larger volume, higher frame rates, higher accuracy,” says Nick.

The fact that expanding the setup is relatively straightforward helped the decision to upgrade with Vicon. So did the presence of Vicon systems on many production stages, ensuring students are able to enter the workplace already fluent in the use of an industry-leading tool.Nick also points to the value that the system offers. “Price-to-performance was hands-down the best for what we needed. There was nothing at that price point that would deliver that type of performance – five megapixel cameras, and several hundreds of frames per second. That was awhole new order of accuracy for us.”

Nick notes that while a typical studio or lab might need only a handful of software licenses, his courses sometimes have as many as 60 or more students using the system and software. At that level, Vicon offers a market-leading price point.

A VERSATILE APPROACH

ACE-Lab uses the system in extremely varied ways. The department has brought in subjects ranging from martial artists to dancers to a troupe of performers in the mold of Cirque du Soleil for capture sessions.

The stage has been used for more esoteric projects, too. “We’ve had a grad student work with literally deconstructing motion capture. There was this piece where the performer was dancing in the volume with markers scattered all over the floor. Sometimes she would roll on the floor and extra markers would be picked up, and then she’d take off markers. Literally that data would get worse and worse and body parts would start falling off. That was surprising to me, after dedicating a career to striving for this photorealism and accuracy in motion capture, for her to completely subvert that was shocking to me – I loved it for that reason.”

Other collaborations have brought in the engineering college, the education college, the law school and nursing and medical students.

Engineering students studying robotics have used the stage to track a robot navigating the space with acoustic sensors. One computer sciences student utilized the system as part of a project to use AI machine learning to teach square dancing.

Virtual production is an increasingly crucial component of ACE-Lab’s programs, both for entertainment and for other sectors. “Right now we’re investigating and deploying virtual production technologies towards remote teaching and learning scenarios, incorporating motion capture into virtual production and to leveraging that for teaching,” Nick says.

“We’re looking at ways of creating virtual learning experiences that are better than in-person. What can we do in these spaces that isn’t even possible when you’re here in person?”

So while ACE-Lab alumni end up at games developers and big-name animation studios, such as Pixar and Dreamworks, just as many are hired in non-entertainment sectors. They might, for example, be recruited by an architect, where they could be the first person in the company with an understanding of tools like Unreal Engine and be tasked with starting a VR visualization program.

“We have some employers that are focusing on the use of digital media in museum spaces and public spaces,” Nick goes on. “So there are service providers where they’ll do 3D scanning, for example, of a tall ship and create animated experiences around that for the museum that that ship is docked next to.

“We’ve had law firms hire our students to do forensic reconstruction of accidents sites and crimescenes.These technologies are an incredibly effective means of communication. And so anywhere there’s communication, you’ll find that some of our students have been hired into those industries to create that perfect communication.”

The diversity of industries that the program feeds into means that Nick and his colleagues are constantly looking to the future. “The trajectory is then transferring these technologies that are being driven by entertainment production into non-entertainment fields,” he says.

“How would you use virtual production in aerospace design, automotive design? How do you use that in training law enforcement? How can you build up awareness and train people without putting them at risk? How can you create simulations that are genuinely visceral, that feel completely real, but they’re completely safe and allow people to experience all sides of a scenario?

“These technologies enable that. Because we can, with a motion capture system, have performers recreate genuine circumstances. We do that in the entertainment industry all the time. But that could be leveraged for social issues, it could be used for educational issues. It could be used to transform public spaces.

“Those are the directions that I see our practitioners moving towards in the years ahead.”