Recognizing hand poses and/or object classes

• US 8,103,109 B2
• Filed: 06/19/2007
• Issued: 01/24/2012
• Est. Priority Date: 06/19/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Winn, John, Criminisi, Antonio, Agarwal, Ankur, Deselaers, Thomas
• Primary Examiner(s): Rahmjoo, Mike
• Application Number:
  US11/765,264
• Publication Number:
  US 20080317331A1
• Time in Patent Office: 1,680 Days
• Field of Search: 382/224, 382/128, 382/129, 382/130, 382/131, 382/132, 382/173, 382/287, 382/294, 382/305, 382/318
• US Class Current: 382/224
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

There is a need to provide simple, accurate, fast and computationally inexpensive methods of object and hand pose recognition for many applications. For example, to enable a user to make use of his or her hands to drive an application either displayed on a tablet screen or projected onto a table top. There is also a need to be able to discriminate accurately between events when a user'"'"'s hand or digit touches such a display from events when a user'"'"'s hand or digit hovers just above that display. A random decision forest is trained to enable recognition of hand poses and objects and optionally also whether those hand poses are touching or not touching a display surface. The random decision forest uses image features such as appearance, shape and optionally stereo image features. In some cases, the training process is cost aware. The resulting recognition system is operable in real-time.

Claims

1. One or more computer-readable storage devices to store processor executable instructions that, when the instructions are implemented by one or more processors, configure the one or more processors to implement a method comprising:

receiving at least one image of an item to be classified as one of a plurality of specified classes, those classes comprising hand pose classes and object classes;

accessing a plurality of decision trees which have been formed in a training process using information at least about classification accuracy;

classifying the image, using the one or more processors, into one of the classes using a unified recognition process at least by applying the plurality of decision trees to at least part of the image, the unified recognition process based in part on the plurality of decision trees being applied by a single module and are used to recognize hand poses, objects, and touch/no touch user interactions; and

storing the classified image in memory.

2. The one or more computer-readable storage devices as claimed in claim 1 wherein accessing the plurality of decision trees comprises accessing decision trees which have been formed in a training process using information about classification accuracy and information about computational cost.

3. The one or more computer-readable storage devices as claimed in claim 1 wherein the process of receiving at least one image comprises receiving a stereo image pair.

4. The one or more computer-readable storage devices as claimed in claim 1 which further comprises inputting information about the classified image into a user interface in order to control a display.

5. The one or more computer-readable storage devices as claimed in claim 4 wherein receiving the image comprises receiving an image comprising one or more items against a background of the display.

6. An apparatus comprising:

an input arranged to receive at least one image of an item to be classified as one of a plurality of specified classes;

a memory arranged to store a plurality of decision trees which have been formed in a training process using information about classification accuracy and information about computational cost; and

a processor arranged to;

classify the image into one of the classes at least by applying the plurality of decision trees to a plurality of pixels of the image;

record an outcome for each decision tree applied to the plurality of pixels, the outcome being an index of a leaf on the decision tree that is reached during the application of the decision tree to the image;

compute a histogram for each decision tree based in part on the recorded outcomes; and

store the classified image into memory.

7. An apparatus as claimed in claim 6 which further comprises a user interface arranged to provide a display and wherein the processor is arranged to provide information about the classified image to the user interface in order to control the display.