Bringing Virtual Humans to Life

For Dr Zerrin Yumak, a computer scientist specializing in socially interactive virtual humans at Utrecht University, there’s a missing piece in today’s crop of digital humans. “I think the most urgent question right now is how we can generate animations for non-verbal communication including with the face and body, but also holistically,” she says. “Multimodal generation of social animations is still missing and it’s a challenge that we need to address.” 

Dr Yumak is an assistant professor in the Information and Computing Sciences Department at Utrecht University and director of the Motion Capture and Virtual Reality Lab. A member of the Human Centered Computing group, she’s been working in the field of 3D digital humans for the last 15 years.

The field covers avatars for humans in virtual spaces, but also digital characters that could be used in video games or to embody chatbots or dialogue systems such as ChatGPT or IBM Watson.

“I’m working on how to make these characters interact naturally with us using natural ways of communication such as facial expressions, gestures and gaze behavior,” Dr Yumak explains. “Lately, I’ve mostly been working on speech- and music-driven animation. Given speech or music as input, we are automatically generating motion. We are aiming for natural and controllable motion synthesis and to automate these processes for faster and less costly pipelines.

“Our recent work on FaceXHubert and FaceDiffuser aims for generating facial motion using advanced deep learning algorithms both for 3D vertex-based and rigged characters. We have worked on a project in collaboration with Guerilla Games to generate gaze motion and our results show the advantage of a data-driven approach with respect to a procedural gaze model.”

These behaviors, whether applied to human avatars, non-player characters in video games or a digital character acting as an interface with a chatbot, are informed by motion capture data.