Sign language (SL) recognition, although it has been explored for many years, is still a challenging problem for real practice. The complex background and illumination conditions affect the hand tracking and make the SL recognition very difficult. Fortunately, Kinect is able to provide depth and color data simultaneously, based on which the hand and body action can be tracked more accurate and easier. This demo will show our primary efforts on sign language recognition and translation with Kinect.

Sign Language Recognition and Translation with Kinect. FG 2013, IEEE conference on Automatic Face and Gesture Recognition

MSRA Best Social Impact Project Award in 2012
MSRA Best Demo Student TechFest Award 2012 (3/26)

On touch-screen mobile phones, text-related applications (e.g., email, notes) usually involve a cursor in editing tasks. However, accurately positioning the cursor to the editing position by touch is often a challenge because of the “fat finger” problem. To improve the accuracy of one-touch cursor positioning, we propose SpeechTouch, a multimodal method that combines imprecise touch input and simple speech input for precise cursor positioning. Our evaluation study shows that this method can significantly improve user performance in cursor positioning on touch screen mobiles.

SpeechTouch: Precise Cursor Positioning on Touch Screen Mobiles MobileHCI 2013, ACM Conference on Human Computer Interaction with Mobile Devices and Services

In this paper, we introduce the design and implementation of TempoString, an easy-to-use tool which assists children with music creation. It provides such a fun and novel platform by allowing children to "draw" music on a canvas and then edit it using a rope. The main contribution of our work is the novel access which allows children to "paint" music on a canvas and then edit using a rope.

TempoString: a tangible tool for children's music creation. Ubicomp 2012, ACM conference on Ubiquitous Computing

When designers explain their early design ideas to others, they usually use face-to-face communication along with sketches. In practice, however, sometimes face-to-face meetings are not possible, and designers have to rely on asynchronous communication. Important contextual information that is available in face-to-face meetings often becomes missing in such asynchronous communications, which can lead to confusion and misunderstanding. To address this challenge, we present SketchComm: an enhanced tool to support rich and flexible asynchronous communication of early design ideas. The key of the system is to allow designers to capture and communicate important contextual information to the audience in addition to sketches. A user study with designers and audience demonstrated effectiveness of asynchronous early design communication using SketchComm.

SketchComm: A tool to support rich and flexible asynchronous communication of early design ideas. CSCW2012, ACM conference on Computer Supported Cooperative Work

With the fast economic growth and urbanization of many developing countries come concerns that their children now have fewer opportunities to express creativity and develop collaboration skills, or to experience their local cultural heritage. We propose to address these concerns by creating technologies inspired by traditional arts, and allowing children to create and collaborate through playing with them. ShadowStory is our first attempt in this direction, a digital storytelling system inspired by traditional Chinese shadow puppetry. We present the design and implementation of ShadowStory and a 7-day field trial in a primary school. Findings illustrated that ShadowStory promoted creativity, collaboration, and intimacy with traditional culture among children, as well as interleaved children's digital and physical playing experience.

ShadowStory: Creative and collaborative digital storytelling inspired by cultural heritage. CHI 2011, ACM Conference on Human Factors in Computing Systems