System and method for integrating gesture and sound for controlling device

• US 8,793,134 B2
• Filed: 12/21/2011
• Issued: 07/29/2014
• Est. Priority Date: 12/23/2010
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Han, Mun Sung, Kim, Hyun, Kim, Jae Hong, Sohn, Joo Chan, Jung, Young Giu
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US13/333,773
• Publication Number:
  US 20120166200A1
• Time in Patent Office: 951 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 23/00 Non-electrical signal transmission systems, e.g. optical systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Disclosed is a system for integrating gestures and sounds including: a gesture recognition unit that extracts gesture feature information corresponding to user commands from image information and acquires gesture recognition information from the gesture feature information; a background recognition unit acquiring background sound information using the predetermined background sound model from the sound information; a sound recognition unit that extracts the sound feature information corresponding to user commands from the sound information and extracts the sound feature information based on the background sound information and acquires the sound recognition information from the sound feature information; and an integration unit that generates integration information by integrating the gesture recognition information and the sound recognition information.

Claims

1. A system for integrating gestures and sounds, comprising one or more processors configured to embody a plurality of functionalities including:

a gesture recognition unit that extracts gesture feature information corresponding to user commands from image information and acquires gesture recognition information from the gesture feature information;

a sound recognition unit that extracts sound feature information corresponding to user commands in a sound section selected by using the gesture recognition information from sound information and acquires sound recognition information from the sound feature information; and

an integration unit that generates integration information by integrating the gesture recognition information and the sound recognition information.

2. The system of claim 1, further comprising a background sound model that collects background sounds, extracts the sound feature information of the collected background sounds, and classifies and models the background sounds based on the extracted sound feature information,

wherein the sound recognition unit acquires the background sounds and the sound recognition information by using the background sound model and the sound model.

3. The system of claim 1, wherein the gesture recognition unit includes:

a human detection module that detects and tracks a part of the human body from the image information;

a gesture feature extraction module that extracts gesture feature information from detection and tracking results of the human detection module; and

a gesture recognition module that acquires gesture recognition information from the gesture feature information.

4. The system of claim 1, further comprising a synchronization unit that synchronizes a sound frame of the sound information with an image frame of the image information.

5. The system of claim 4, wherein the synchronization unit selects a sound command candidate section based on a point in time when the gestures are first recognized in the sound frame, and

the sound recognition unit extracts the sound feature information within the sound command candidate section.

6. The system of claim 5, wherein the synchronization unit selects a predetermined section from a point in time when the gestures are first recognized as the sound command candidate section.

7. The system of claim 5, wherein the synchronization unit selects as the sound command candidate section the predetermined section from a point in time after a predetermined time of the point in time when the gestures are first recognized.

8. The system of claim 5, wherein the sound recognition unit includes:

a dynamic end point detection (EPD) module that moves windows in a frame unit within the sound command candidate section;

a sound feature extraction module that extracts sound features for each moved window and extracts a sound command section and the sound feature information using probability information corresponding to each sound feature and a previously built EPD probability model; and

a sound recognition module that acquires the sound recognition information from the sound feature information.

9. The system of claim 8, wherein the sound recognition module acquires the sound recognition information based on the previously built sound model.

10. A method for integrating gestures and sounds, wherein the method is performed by one or more processors, comprising:

extracting gesture feature information corresponding to user commands from image information and acquiring gesture recognition information from the gesture feature information;

extracting sound feature information corresponding to user commands in a sound section selected by using the gesture recognition information from sound information and acquiring sound recognition information from the sound feature information; and

generating integration information by integrating the gesture recognition information and the sound recognition information.

11. The method of claim 10, further comprising selecting a sound command candidate section based on a point in time when the gestures are first recognized in the sound frame of the sound information, and

wherein the acquiring of the sound recognition information extracts the sound feature information within the sound command candidate section.

12. The method of claim 11, wherein the acquiring of the sound recognition information includes:

extracting sound features for each moved window while moving windows in a frame unit within the sound command candidate section and extracting a sound command section and the sound feature information by using probability information corresponding to each sound feature and a previously built end point detection (EPD) probability model; and

acquiring the sound recognition information from the sound feature information.

13. The method of claim 10, wherein the acquiring of the sound recognition information acquires background sounds and the sound recognition information using the previously built background sound model and the sound model.