Compound Gesture Recognition

• US 20100050133A1
• Filed: 08/22/2008
• Published: 02/25/2010
• Est. Priority Date: 08/22/2008
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Kaehler, Adrian, Jangaard, Lars, Nishihara, H. Keith, Ferren, Bran, Hsu, Shi-Ping
• Granted Patent: US 8,972,902 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

One embodiment of the invention includes a method for executing and interpreting gesture inputs in a gesture recognition interface system. The method includes detecting and translating a first sub-gesture into a first device input that defines a given reference associated with a portion of displayed visual content. The method also includes detecting and translating a second sub-gesture into a second device input that defines an execution command for the portion of the displayed visual content to which the given reference refers.

Claims

1. A method for executing and interpreting gesture inputs in a gesture recognition interface system, the method comprising:

detecting and translating a first sub-gesture into a first device input that defines a given reference associated with a portion of displayed visual content;

detecting and translating a second sub-gesture into a second device input that defines an execution command for the portion of the displayed visual content to which the given reference refers.

2. The method of claim 1, wherein detecting and translating the first sub-gesture comprises detecting and translating the first sub-gesture that is provided by a first hand of a user.

3. The method of claim 2, wherein the first sub-gesture comprises pointing to the portion of the displayed visual content with at least one extended finger, and wherein the second sub-gesture comprises one of moving a thumb associated with the first hand of the user, rapidly extending and retracting the first hand of the user, rotating the at least one extended finger, forming a grip with the thumb and at least one finger of the first hand of the user, forming a fist, and snapping fingers of the first hand of the user.

4. The method of claim 2, wherein detecting and translating the second sub-gesture comprises detecting and translating the second sub-gesture that is provided by the second hand of the user, the method further comprising maintaining the first sub-gesture with the first hand of the user concurrently with employing the second hand of the user to provide the second sub-gesture.

5. The method of claim 1, further comprising maintaining the second sub-gesture to interact with the selected portion of the displayed visual content.

6. The method of claim 1, wherein the first sub-gesture corresponds to pointing at the portion of the displayed visual content, and wherein the reference corresponds to a mouse cursor.

7. The method of claim 1, wherein the second device input corresponds to one of a single mouse-click, a double mouse-click, and a mouse-click-and-hold.

8. The method of claim 1, wherein translating the first and second sub-gestures comprises:

obtaining a plurality of sequential images of an input object;

determining a three-dimensional location of the input object in each of the plurality of sequential images of the input object;

determining motion of the input object based on changes in the three-dimensional location of the input object in each of the plurality of sequential images of the input object; and

correlating the motion of the input object into one of a plurality of predefined gestures that each have an associated device input.

9. The method of claim 8, wherein the first sub-gesture and the second sub-gesture are each implemented by the first input object, wherein translating the first sub-gesture further comprises:

buffering the plurality of sequential images in a memory;

accessing the buffered plurality of sequential images from the memory subsequent to translating the second gesture; and

determining to what the given reference refers on the displayed visual content based on the accessed plurality of sequential images.

10. The method of claim 8, wherein obtaining the plurality of sequential images of the input object comprises obtaining a plurality of sequential images of the input object concurrently from a plurality of stereo cameras, and wherein determining the three-dimensional location of the input object comprises:

illuminating the input object with infrared (IR) light;

determining a location of the input object in each the plurality of sequential images of each of the plurality of stereo cameras based on an IR brightness contrast between the input object and a background; and

determining the three-dimensional location of the input object based on a parallax separation of the input object in each of the concurrently obtained sequential images of each of the respective plurality of stereo cameras.

11. The method of claim 1, further comprising:

detecting and translating a third sub-gesture into a third device input that is configured to execute a command associated with manipulation of the portion of the displayed visual content.

12. A gesture recognition interface system configured to implement the method of claim 1.

13. A method for executing and interpreting gesture inputs in a gesture recognition interface system, the method comprising:

obtaining a plurality of sequential images of a gesture input environment;

detecting a first sub-gesture based on a three-dimensional location of at least one feature of a first input object relative to displayed visual content in each of the plurality of sequential images of the gesture input environment;

translating the first sub-gesture into a first device input that defines a given reference associated with a portion of the displayed visual content;

detecting a second sub-gesture based on changes in the three-dimensional location of at least one feature of at least one of the first input object and a second input object in each of the plurality of sequential images of the gesture input environment; and

translating the second sub-gesture into a second device input that defines an execution command for the portion of the displayed visual content to which the given reference refers.

14. The method of claim 13, wherein detecting the first sub-gesture comprises determining a three-dimensional location of at least one finger of a first hand of a user relative to the displayed visual content, and wherein detecting the second sub-gesture comprises determining changes in the three-dimensional location of at least one finger of at least one of the first hand of the user and a second hand of the user in each of the plurality of sequential images of the gesture input environment.

15. The method of claim 14, wherein determining the three-dimensional location of the at least one finger of a first hand comprises determining the portion of the visual content based on a location on the visual content where the at least one finger is pointing, and wherein the second sub-gesture comprises one of moving a thumb associated with the first hand of the user, rapidly extending and retracting the first hand of the user, rotating the at least one finger, forming a grip with a thumb and at least one finger of one of the first and second hand, and snapping fingers of one of the first and second hand.

16. The method of claim 14, wherein determining the second sub-gesture comprises maintaining the first sub-gesture with the first hand of the user concurrently with determining the changes in the three-dimensional location of the at least one finger of the second hand of the user.

17. The method of claim 13, further comprising:

detecting a third sub-gesture based on changes in a three-dimensional location of at least one feature of the at least one of the first input object and a second input object in each of the plurality of sequential images of the gesture input environment; and

translating the third sub-gesture into a third device input to the computer, the third device input being configured to execute a command associated with manipulation of the portion of the visual content.

18. The method of claim 13, wherein detecting the third sub-gesture comprises determining changes in the three-dimensional location of the at least one feature of the first input object in each of the plurality of sequential images of the gesture input environment, and wherein translating the second sub-gesture comprises accessing the plurality of sequential images prior to the changes in the three-dimensional location of the at least one feature of the first input object to determine the portion of the visual content on which to execute the command.

19. A gesture recognition interface system comprising:

means for displaying visual content;

means for obtaining a plurality of sequential images of a gesture input environment that is associated with the visual content;

means for determining compound gesture inputs associated with at least one input object based on three-dimensional locations of at least one feature of the at least one input object in each of the plurality of sequential images of the gesture input environment; and

means for translating the compound gesture inputs into a first device input and a second device input, the first device input being configured to reference a portion of the visual content and the second device input being configured to execute a command associated with the portion of the visual content to which the first device input refers in at least one of the plurality of sequential images.

20. The system of claim 19, wherein the means for means for obtaining the plurality of images comprises plural means for concurrently obtaining the plurality of sequential images from different perspectives, and wherein the means for determining the compound gesture inputs comprises:

means for illuminating the at least one input object with infrared (IR) light; and

means for determining the three-dimensional locations of the at least one feature of the at least one input object based on an IR brightness contrast between the at least one input object and a background, and based on a parallax separation of the at least one feature of the at least one input object in the different perspectives of the concurrently obtained plurality of sequential images.

21. The system of claim 19, wherein the means determining the compound gestures determines changes in the three-dimensional location of the at least one feature of the first input object in each of the plurality of sequential images of the gesture input environment, and wherein the means for translating the compound gesture inputs accesses the buffered plurality of sequential images prior to the changes in the three-dimensional location of the at least one feature of the first input object to determine the portion of the visual content on which to execute the command.