Recognizing Hand Poses and/or Object Classes

• US 20080317331A1
• Filed: 06/19/2007
• Published: 12/25/2008
• Est. Priority Date: 06/19/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Winn, John, Agarwal, Ankur, Deselaers, Thomas
• Granted Patent: US 8,103,109 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

There is a need to provide simple, accurate, fast and computationally inexpensive methods of object and hand pose recognition for many applications. For example, to enable a user to make use of his or her hands to drive an application either displayed on a tablet screen or projected onto a table top. There is also a need to be able to discriminate accurately between events when a user'"'"'s hand or digit touches such a display from events when a user'"'"'s hand or digit hovers just above that display. A random decision forest is trained to enable recognition of hand poses and objects and optionally also whether those hand poses are touching or not touching a display surface. The random decision forest uses image features such as appearance, shape and optionally stereo image features. In some cases, the training process is cost aware. The resulting recognition system is operable in real-time.

Claims

1. A method comprising:

receiving at least one image of an item to be classified as one of a plurality of specified classes;

accessing a plurality of decision trees which have been formed in a training process using information about classification accuracy and information about computational cost;

classifying the image into one of the classes at least by applying the plurality of decision trees to at least part of the image; and

storing the classified image.

2. A method as claimed in claim 1 which further comprises segmenting the at least one image to identify a foreground region and applying the plurality of decision trees to that foreground region.

3. A method as claimed in claim 1 wherein the process of receiving at least one image comprises receiving a stereo image pair.

4. A method as claimed in claim 1 wherein the process of receiving at least one image comprises receiving a depth map.

5. A method as claimed in claim 1 which further comprises receiving a touch map and wherein the process of classifying the image comprises using information from the touch map.

6. A method as claimed in claim 1 wherein the plurality of specified classes comprises classes in which items are touching a display surface and classes in which items are not touching a display surface.

7. A method as claimed in claim 1 wherein the plurality of specified classes comprise a plurality of classes of hand poses and a plurality of classes of objects.

8. A method as claimed in claim 1 which further comprises inputting information about the classified image into a user interface.

9. A method as claimed in claim 1 wherein the step of accessing the decision trees comprises accessing decision trees comprising tests using any of appearance and shape image features.

10. A method as claimed in claim 3 wherein the step of accessing the decision trees comprises accessing decision trees comprising tests using appearance image features, shape image features and stereo image features.

11. A method as claimed in claim 4 wherein the step of accessing the decision trees comprises accessing decision trees comprising tests using depth map features.

12. A method as claimed in claim 1 wherein the step of classifying the image comprises, for each decision tree, computing a histogram using results of applying that decision tree to at least part of the image.

13. A method as claimed in claim 12 wherein the step of classifying the image further comprises, concatenating the histograms and inputting the concatenated histogram to a multi-class classifier.

14. A method comprising:

receiving at least one image of an item to be classified as one of a plurality of specified classes, those classes comprising hand pose classes and object classes;

accessing a plurality of decision trees which have been formed in a training process using information at least about classification accuracy;

classifying the image into one of the classes using a unified recognition process at least by applying the plurality of decision trees to at least part of the image; and

storing the classified image.

15. A method as claimed in claim 14 wherein the process of accessing the plurality of decision trees comprises accessing decision trees which have been formed in a training process using information about classification accuracy and information about computational cost.

16. A method as claimed in claim 14 wherein the process of receiving at least one image comprises receiving a stereo image pair.

17. A method as claimed in claim 14 which further comprises inputting information about the classified image into a user interface in order to control a display.

18. A method as claimed in claim 17 wherein the process of receiving the image comprises receiving an image comprising one or more items against a background of the display.

19. An apparatus comprising

an input arranged to receive at least one image of an item to be classified as one of a plurality of specified classes;

a memory arranged to store a plurality of decision trees which have been formed in a training process using information about classification accuracy and information about computational cost; and

a processor arranged to classifying the image into one of the classes at least by applying the plurality of decision trees to at least part of the image, and to store the classified image.

20. An apparatus as claimed in claim 19 which further comprises a user interface arranged to provide a display and wherein the processor is arranged to provide information about the classified image to the user interface in order to control the display.