12/01/2022

The Rise in Multi-Modal Data Fusion in Motion Capture – Dr. Kim Duffy

Historically, Vicon has been a company that focuses on one technology: optical motion capture.

Now, like many others within biomechanics, we have shifted away from a focus on a single technology/ tracking modality towards multi-modal fused data. Multi-modal in this case means more than one modality, so more than one technology to track motion, and data fusion is combining different data to produce a result.

Why is this important and where is motion capture technology heading?

Before answering this question, we should look back at the history of motion capture. Eadweard Muybridge is not only considered a founder of biomechanics, but also of the technology used to capture and analyze human movement. The classic moving pictures that Muybridge captured, The horse in motion for example, is a perfect representation of a single modality; the single technology in this case is photography being used to capture human movement. The work conducted by Muybridge is where we can say human movement technology began.

Over the years, motion capture technology (optical, video and inertial sensors) assessing human movement has developed and expanded. I like to think that technology evolves in two different ways. First, it can evolve linearly, making a single technology better. For years, Vicon and other technology companies have worked to improve the technology and the modality to help the biomechanics community get better and better results.

One of the natural results in the linear evolution of technology is that if you continue for long enough, the improvements that you can make start to plateau and you don’t see big leaps as you did in the beginning. Rather than adopting a linear evolution approach, you can instead combine different technologies.

Mobile phones are a fantastic example of both linear and multimodal data fusion evolution. Firstly, progressing linearly, mobile phones were originally solely about wireless voice communication, using one technology, and the manufacturers worked to make them easier to use, smaller and lighter. Eventually they started to combine other types of technology, more powerful processing, cameras within mobile phones and the use of inertial sensors for interactions between device and user. By combining these different technologies, designers evolved mobile phones into smart phones.

Biomechanics technology is no different to the evolution of the mobile phone. While optical motion capture is the tool of choice for many usecases within life sciences, inertial and markerless technology allow users to expand their applications, moving away from the lab for example. These technologies have advantages and disadvantages and there is a lot of exciting potential in combining them.

At Vicon, we have been combining different technology for a long time. In the past, we focused on combining technologies that were sensing different things. For example, full-body optical motion capture combined with force plates to calculate joint forces. This allowed for synchronized collection of separate data (e.g. optical and EMG) in the same software.

Now, we are starting to use different technologies together to maximize their pros and balance out their cons by using multiple sources together to quantify the same movement. Several years ago, we acquired a company in New Zealand, IMeasureU that specialize in inertial sensing. We saw the advantages of inertial sensing and combined them with optical. Today we are moving past simply synchronizing capture, and taking two different sensing technologies measuring the same movement and using them together. Emerging techniques such as markerless add the ability to compare Apple’s AR Kit with the IMU Capture.U App. We also launched a partnership with Theia Markerless in 2020 to offer a combination of optical and markerless data. Today, rather than just having different synchronized data types, you are able to collect the same data. If you have inertial and optical measurement, and the optical marker gets covered, you could fill the data in with inertial. So, frame by frame you can use one or the other modality and you shouldn’t lose data. We see this as a big advantage. Video user-needs and developments are also expanding, and we are excited to welcome CONTEMPLAS, a well known video system solution provider, to the Vicon Family this year to help address the need for quick, reliable video analysis.

Vicon has a longstanding history within Life Sciences, and we will continue to work with the scientific community to expand multi-modal data capture and derive outcomes that are based on validated and published methods. Although this ultimately takes more time, our belief is that there must always be science behind our products.

There is value and exciting possibility when you start combining different modalities together, and that is where we can see the industry moving.

Nowhere is that more obvious than among our community of users. Every conversation we’ve had for The Standard has come down to one of our customers pushing the boundaries of what motion capture can do. We’re seeing incredible strides in the applications Vicon technology is used for and we, in turn, are inspired to expand our offering to fit. The result is a virtuous cycle that continues to push motion capture in exciting new directions.