Compound gesture recognition

• US 8,972,902 B2
• Filed: 08/22/2008
• Issued: 03/03/2015
• Est. Priority Date: 08/22/2008
• Status: Expired due to Fees

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Nishihara, H. Keith, Hsu, Shi-Ping, Ferren, Bran, Jangaard, Lars, Kaehler, Adrian
• Primary Examiner(s): Lo, Weilun
• Assistant Examiner(s): Hassan, Rashedul
• Application Number:
  US12/196,767
• Publication Number:
  US 20100050133A1
• Time in Patent Office: 2,384 Days
• Field of Search: 715/863, 715/856, 715/857, 382/103, 382/154, 382/203, 345/156, 345/157
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

One embodiment of the invention includes a method for executing and interpreting gesture inputs in a gesture recognition interface system. The method includes detecting and translating a first sub-gesture into a first device input that defines a given reference associated with a portion of displayed visual content. The method also includes detecting and translating a second sub-gesture into a second device input that defines an execution command for the portion of the displayed visual content to which the given reference refers.

Claims

1. A method for executing and interpreting gesture inputs in a gesture recognition interface system, the method comprising:

detecting and translating a first sub-gesture provided by a first hand of a user in a three-dimensional gesture recognition environment into a first device input that defines a given reference associated with a portion of displayed visual content;

detecting and translating a second sub-gesture provided by a second hand of a user while concurrently maintaining the first sub-gesture with the first hand of the user in the three-dimensional gesture recognition environment into a second device input that defines an execution command for the portion of the displayed visual content to which the given reference refers; and

detecting and translating a third sub-gesture into a third device input that is configured to execute a command associated with manipulation of the portion of the displayed visual content to which the given reference refers while at least one of the first and second sub-gestures is maintained.

2. The method of claim 1, further comprising maintaining the second sub-gesture to interact with the portion of the displayed visual content.

3. The method of claim 1, wherein the first sub-gesture corresponds to pointing at the portion of the displayed visual content, and wherein the reference corresponds to a mouse cursor.

4. The method of claim 1, wherein the second device input corresponds to one of a double mouse-click and a mouse-click-and-hold.

5. The method of claim 1, wherein translating the first and second sub-gestures comprises:

obtaining a plurality of sequential images of a first bare hand and a second bare hand of a user;

determining a three-dimensional location of each bare hand in each of the plurality of sequential images of the respective bare hands;

determining motion of each bare hand based on changes in the three-dimensional location of the respective bare hands in each of the plurality of sequential images of the each respective bare hand; and

correlating the motion of each bare hand into one of a plurality of predefined gestures that each has an associated device input.

6. The method of claim 5, wherein translating the first sub-gesture further comprises:

buffering the plurality of sequential images in a memory;

accessing the buffered plurality of sequential images from the memory subsequent to translating the second gesture; and

determining to what the given reference refers on the displayed visual content based on the accessed plurality of sequential images.

7. The method of claim 5, wherein obtaining the plurality of sequential images of each bare hand comprises obtaining a plurality of sequential images of each bare hand concurrently from a plurality of stereo cameras, and wherein determining the three-dimensional location of each bare hand comprises:

illuminating each bare hand with infrared (IR) light;

determining a location of each bare hand in each the plurality of sequential images of each of the plurality of stereo cameras based on an IR brightness contrast between each bare hand and a background; and

determining the three-dimensional location of each bare hand based on a parallax separation of each bare hand in each of the concurrently obtained sequential images of each of the respective plurality of stereo cameras.

8. A gesture recognition interface system comprising:

means for displaying visual content;

means for obtaining a plurality of sequential images of a gesture input environment that is associated with the visual content;

means for buffering the plurality of sequential images of the gesture input environment;

means for determining compound gesture inputs including a first input gesture and a second input gesture associated with at least one input object based on three-dimensional locations of at least one feature of the at least one input object in each of the plurality of sequential images of the gesture input environment; and

means for translating the compound gesture inputs into a first device input based on the first input gesture and a second device input based on the second input gesture, the first device input being configured to reference a portion of the visual content and the second device input being configured to execute at least one command that allows a user to interact with the selected portion of the displayed visual content using the first input gesture while the second input gesture is maintained, the means for translating determining the reference to the portion of the visual content based on accessing the buffered plurality of sequential images subsequent to translating the second input gesture.

9. The system of claim 8, wherein the means for obtaining the plurality of images comprises plural means for concurrently obtaining the plurality of sequential images from different perspectives, and wherein the means for determining the compound gesture inputs comprises:

means for illuminating the at least one input object with infrared (IR) light; and

means for determining the three-dimensional locations of the at least one feature of the at least one input object based on an IR brightness contrast between the at least one input object and a background, and based on a parallax separation of the at least one feature of the at least one input object in the different perspectives of the concurrently obtained plurality of sequential images.