ARCHITECTURE FOR CONTROLLING A COMPUTER USING HAND GESTURES

• US 20090268945A1
• Filed: 06/30/2009
• Published: 10/29/2009
• Est. Priority Date: 03/25/2003
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Oliver, Nuria M., Wilson, Andrew D.
• Application Number:
  US12/494,303
• Publication Number:
  US 20090268945A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 13/105 where the programme performs an input/output emulation function
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

Claims

1. A method of determining a command, comprising:

capturing an image of an object with a camera;

determining a gesture based at least partly upon the image;

detecting an audio input; and

determining, at one or more processors, the command based at least partly upon the gesture and the audio input.

2. The method of claim 1, further comprising:

determining a depth of the object; and

determining the command based at least partly upon the depth of the object.

3. The method of claim 2, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

4. The method of claim 1, wherein the camera is a video camera.

5. The method of claim 1, wherein the camera detects visible light.

6. The method of claim 1, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.

7. A computer-readable medium having instruction that cause a processor to execute steps, the steps comprising:

capturing an image of an object with a camera;

determining a gesture based at least partly upon the image;

detecting an audio input; and

determining, at one or more processors, a command based at least partly upon the gesture and the audio input.

8. The computer-readable medium of claim 7, the steps further comprising:

determining a depth of the object; and

determining the command based at least partly upon the depth of the object.

9. The computer-readable medium of claim 8, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

10. The computer-readable medium of claim 7, wherein the camera is a video camera.

11. The computer-readable medium of claim 7, wherein the camera detects visible light.

12. The computer-readable medium of claim 7, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.

13. A command determining system, comprising:

a camera configured to capture an image of an object;

a first determiner configured to determine a gesture based at least partly upon the image;

an audio detection unit configured to detect an audio input; and

a second determiner configured to determine the command based at least partly upon the gesture and the audio input.

14. The command determining system of claim 13, further comprising:

a third determiner configured to determine a depth of the object, wherein the second determiner is further configured to determine the command based at least partly upon the depth of the object.

15. The command determining system of claim 14, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

16. The command determining system of claim 13, wherein the camera is a video camera.

17. The command determining system of claim 13, wherein the camera detects visible light.

18. The command determining system of claim 13, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.