CONTROLLING A COMPUTING-BASED DEVICE USING HAND GESTURES

• US 20140208274A1
• Filed: 01/18/2013
• Published: 07/24/2014
• Est. Priority Date: 01/18/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Smyth, Samuel Gavin, Ansell, Peter John, O'Prey, Christopher Jozef, Shotton, Jamie Daniel Joseph, Sharp, Toby, Izadi, Shahram, Banks, Richard Malcolm, O'Hara, Kenton, Butler, David Alexander, Hodges, Stephen E, Goldberg, Mitchel Alan, Sellen, Abigail Jane, Harper, Richard Harry Robert, Greveson, Eric John
• Application Number:
  US13/744,823
• Publication Number:
  US 20140208274A1
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

Methods and system for controlling a computing-based device using both input received from a traditional input device (e.g. keyboard) and hand gestures made on or near a reference object (e.g. keyboard). In some examples, the hand gestures may comprise one or more hand touch gestures and/or one or more hand air gestures.

Claims

1. A method of controlling a computing-based device, the method comprising:

receiving input from a traditional input device;

receiving an image stream of a reference object;

performing gesture recognition on the received image stream to identify hand gestures performed on or near the reference object; and

controlling the computing-based device using both the input received from the traditional input device and the identified hand gestures.

2. The method according to claim 1, wherein the hand gestures comprise at least one touch hand gesture.

3. The method according to claim 1, wherein the hand gestures comprise at least one free-air hand gesture.

4. The method according to claim 1, wherein the hand gestures comprise at least one touch hand gesture and at least one free-air hand gesture.

5. The method according to claim 1, wherein the reference object is a keyboard.

6. The method according to claim 1, wherein performing gesture recognition on the received image stream comprises identifying the background of the received image stream, wherein identifying the background of the received image stream comprises deeming an image element of the received image stream to be part of the background if the object associated therewith does not move a predetermined distance within a predetermined period.

7. The method according to claim 1, wherein performing gesture recognition on the received image stream comprises performing connected component analysis on at least a part of the image stream to identify components that have at least one image element that lies on a front edge of the image stream.

8. The method according to claim 1, wherein performing gesture recognition on the received image stream comprises detecting at least one arm in the received image stream, wherein the at least one arm is detected by identifying an object in the image stream that has at least one image element that lies on a front edge of the image stream and is greater than a predetermined threshold size.

9. The method according to claim 1, wherein performing gesture recognition on the received image stream comprises applying a classifier to at least a portion of the image elements of the received image stream to classify each image element of the portion of image elements as being part of at least one of a particular body part and a particular state.

10. The method according to claim 9, wherein the classifier is trained using images of at least one synthetic hand and at least one real hand.

11. The method according to claim 9, wherein the classifier produces classification data for each image element of the portion of image elements, the classification data indicting the probability that the image element is part of each of a plurality of possible body parts and each of a plurality of possible states.

12. The method according to claim 11, wherein performing gesture recognition on the received image stream further comprises identifying the location of the centre of mass of a particular body part using the classification data.

13. The method according to claim 12, wherein identifying the centre of mass of the body part comprises summing the product of the location of each image element identified as relating to the body part and the probability that the image element is part of the body part divided by the sum of the probabilities that the image element is part of the body part.

14. The method according to claim 11, wherein performing gesture recognition on the received image stream further comprises identifying the location of a user'"'"'s digit tip, wherein identifying the location of the user'"'"'s digit tip comprises:

determining the centre of mass of the user'"'"'s hand;

determining the centre of mass of the user'"'"'s wrist;

establishing a virtual line between the centre of mass of the hand and the centre of mass of the wrist; and

deeming the user'"'"'s digit tip to be the image element on the virtual line that is furthest away from the centre of mass of the wrist.

15. The method according to claim 1, wherein performing gesture recognition on the received image stream comprises identifying the plane of a desktop on which the reference object is situated, wherein identifying the plane of the desktop comprises identifying a set of candidate image element likely to correspond to the desktop and iteratively generating a plane and modifying the set of candidate image elements until the plane is a good match for the set of candidate image elements.

16. The method according to claim 1, wherein at least one gesture is identified when a predetermined action is performed in a predetermined location with respect to the reference object.

17. The method according to claim 1, wherein at least one gesture is identified when a user moves a first hand in a predetermined pattern with respect to a second hand.

18. A system to control a computing-based device, the system comprising:

the computing-based device configured to;

receive input from a traditional input device;

receive an image stream of a reference object from a capture device;

perform gesture recognition on the received image stream to identify hand gestures performed on or near the reference object; and

control the computing-based device using both the input received from a traditional input device and the identified hand gestures.

19. The system according to claim 18, the computing-based device being at least partially implemented using hardware logic selected from any one of more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.

20. A method of controlling a computing-based device, the method comprising:

receiving input from a keyboard;

receiving an image stream of the keyboard;

performing gesture recognition on the received image stream to identify hand gestures performed on, near or above the keyboard; and

controlling the computing-based device using both the input received from the keyboard and the identified hand gestures.