Spatial and Temporal Pyramids for Grammatical Expression Recognition of American Sign Language

Nicholas Michael, Dimitris Metaxas and Carol Neidle

Eleventh International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2009)
Pittsburgh, PA, USA, October 26-28, 2009

---

Abstract

Given that sign language is the primary modality for communication for deaf people, development of sign language recognition technologies would be a major step forward in making computers equally accessible to everyone. However, most research in the field of sign language recognition has focused on the manual component of signs, despite the fact that there is critical grammatical information expressed through facial expressions and head gestures.

We propose a novel framework for robust tracking and analysis of facial expression and head gestures, with an application to sing language recognition. We then apply it to recognize with excellent accuracy two classes of grammatical expressions, namely wh-questions and negative expressions. Our method is signer-independent and builds on the popular ``bag-of-words" model, utilizing spatial pyramids to model facial appearance and temporal pyramids to represent patterns of pose derivatives.