Vicon Study of Dynamic Object Tracking Accuracy
Tim Massey, Vicon Engineering & VR Product Manager
There is already a wide body of work in which assessments of various motion capture systems, Vicon included, have produced a wide range of different results.
These studies produce useful and relevant results for specific use cases, but they frequently measure different things and the methods, or even the terminology that is used, vary significantly in several important ways.
The nature of optical motion capture systems means that there is a large range of variables, making answering questions about accuracy complicated, and some of the most important factors in system performance are also unintuitive.
Conditions affecting data quality can include: the size of the space, the geometry and speed of the object, the nature of the movement, and changes in ambient conditions – as well as the characteristics of the system itself – such as the number, resolution and quality of cameras; the quality and size of markers used; the camera calibration process; and the algorithms used to compute the rigid object pose from the camera data.
Even with a completely controlled environment, the motion of objects can affect the measurements. For example, we can continuously observe the length of a test object. It’s possible to observe a highly significant difference, depending on whether the object is at rest or in motion. This shows how crucial consistency and control of the variables are before comparisons can be made.
Terminology can often mean different things in different contexts, with terms such as ‘accuracy’ and ‘precision’ taking on very different meanings. In addition to this, they are often case-specific: for example the meaning of the term ‘accuracy’ may be understood differently in the study of biomechanics to robotics to visual effects.
As part of this work, we really wanted to provide information that would be informative and help customers to understand what level of performance they could expect. This demanded a method that was appropriate to the audience that most frequently asked about this topic, was easily explainable, and produced sensible results across a range of different uses.
We also wanted to start a conversation – internally during development, as well as externally – with a method that was understood, so that future developments and improvements could be communicated in that context.
As a company, we felt it was important to use an externally-published method, to be fully transparent about that method and therefore about what results are achieved.
Vicon pulled together a group with a wide array of experience in the details of how motion capture worked: in practice – not just in theory – after installations with customers. They were a group who knew how we could help make sense of this problem in a way that would be useful and informative for our customers.
Over a period of months we reviewed a range of external methods, trying some of the most promising, to assess their effectiveness and relevance to how the systems were being used.
We decided that a method developed and assessed by the subcommittee for 3D measurement systems at the American Society of Tests and Measures (ASTM E57) was most appropriate. The method E3064 (“Standard Test Method for Evaluating the Performance of Optical Tracking Systems that Measure Six Degrees of Freedom (6DOF) Pose”) is a simple one that allows a uniform measurement between two rigid objects while moving in a controlled way in an optical motion capture volume.
This method had the advantage of being simple to understand and explain. But it also had the benefit of assessing a continuously moving object (a critical component of a motion capture system, and a variable that can make a significant difference!) and assessing errors quickly in many locations within the capture volume.
It’s important, though, to mention that there is no ‘correct’ method and many different alternatives, and each method has different merits. However, we felt this was an informative method to help explain to customers the type of performance they may be able to expect with an assessment that was relevant to how Vicon systems are used by our engineering customers.
We tested two separate but similar setups, made up of Vantage 5 MP or 16 MP cameras, and measured the distance between the two rigid objects. This was a known fixed distance that was constructed of thermally-neutral plastic to eliminate, as much as possible, physical differences that might arise from movement or temperature changes.
This enabled the measurement accuracy to be measured consistently against this known fixed reference as it moved around the volume, and the measurements to be assessed. Observation was consistent even though the object was moved across the capture volume and in continuous motion.
This work now enables us to better communicate how new developments improve the quality that Vicon provides.
A great example of this is the recent release of Vantage+ where we’re now able to clearly explain the huge benefits that the increased frame rate can deliver, but also the inherent trade-offs that exist within the technology.
We’re excited to be able to move forward and better help our customers to understand better how to get the best possible results for their own amazing projects.
Find out more about Vicon accuracy in our FAQ section.
