I thought I knew a lot about sign language. The elementary school I attended during first through fifth grade was attached to the city's school for the deaf. The school offered many extra-curricular activities for all students to come and learn sign language. They also had the school band practice in the gymnasium located at that end of the school, for obvious reasons. Even as a youngster I understood the unique opportunity that the school for the deaf presented me, and I excitedly signed up for the after school classes they offered. I learned the American Sign Language alphabet, as well as a smattering of kid based vocabulary, and it was fun to be able to have small, slow conversations with my friends during our normal class without the teacher hearing us. I remember the classes fondly, but after fifth grade I moved to a new school where none of the other kids knew anything about sign language, and most of the vocabulary was quickly forgotten.
Recently, sign language brought me to Gallaudet University in Washington D.C., which is the only federally chartered University dedicated to the higher education of the Deaf and Hearing Impaired population. The University had purchased a T-Series system from Vicon, which was made possible by a W.M. Keck Foundation Grant which was awarded to Dr. Laura-Ann Petitto, Principal Investigator, with Melissa Malzkuhn (Co-PI), for a research project entitled, "Seeing the Rhythmic Temporal Beats of Human Language." For more information see http://www.gallaudet.edu/petitto.html.
The system purchase included an on-site installation and training session that we provide for new customers. The installation and training gives us an opportunity to meet the end user, learn more about their projects, and provide a quality inspection and professional installation of the equipment, as well as offering the user the opportunity for personalized hands on training with their new system. While many of the installation and training sessions I've performed in the past share some similarities, I knew that this would be an altogether new and exciting challenge. I did not know, however, the true scope of the challenge, or how much I was about to learn.
After I arrived at the University and the American Sign Language (ASL) interpreters introduced themselves and the team, I asked the customary question "What is the ultimate goal of your motion capture system?". The answer was so much more than I expected. The project that Melissa and the team at the Motion Light Lab are pursuing is about more than just capturing sign language. Their goal is to use motion capture to help build stimuli that they can show to young infants, while testing with thermal infrared cameras and fNIRS neuroimaging tools to find temporal rhythmic patterns that young babies are sensitive to. In addition, they will continue the development of ASL nursery rhymes and storytelling using motion capture to study the temporal rhythmic patterns of our natural visual language. As is often the case, motion capture is just a small part of their overall goal, and is just another tool in the researcher's lab. As we proceeded with the installation, I started to become familiar with the ebb and flow of a conversation through an ASL interpreter, and quickly realized that I knew very little about sign language.
Originally the setup was pretty typical, although I did place more cameras in the front of the lab as I knew the hands would mostly be in front of the person signing. Once we had a person in a suit, with a full set of finger markers, I quickly realized I was looking in the wrong direction. The markers are placed on the back of the hands and the fingers, and as the hands move in sign, most of the marker motion stays in a relatively small pocket in front of the speaker. The motion is small and complex, executed with precision, intention and SPEED! I can still remember the ASL alphabet, but words that were spelled out were a blur of fast finger motion and mostly passed me by. I also quickly realized that the facial expressions used while signing are as important as any of the hand motions. It's the visual punctuation and tone to what's being said, and cannot be left out of a fully understandable sign language avatar. While listening to my interpreter, I would watch the face of whomever was speaking so that I could fully understand what was being said.
After a couple of ideas for camera configurations were tested, I ended up with more cameras behind the speaker than in front. Looking at it now, it's the only reasonable solution, but it wasn't until I understood more about the motion we were trying to capture that I could come to that solution. Motion capture has many challenges, and each day I learn of a new one. While the basics are the same, each solution is unique and tailored to the application. It's the endless technical creativity of solving these new problems that has captured my mind and fuels my passion for motion capture.
As I finished the training session and began to leave for the airport, I thanked Melissa and her team for all that they had taught me. It's not often that I leave a training session having learned more than I taught. I may not know as much about sign language as I thought, but I now know the sign for motion capture, avatar, and a smashing joke about King Kong! Ask me to show you all three next time I see you!