Architecture for controlling a computer using hand gestures

• US 8,745,541 B2
• Filed: 12/01/2003
• Issued: 06/03/2014
• Est. Priority Date: 03/25/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Oliver, Nuria M.
• Primary Examiner(s): Nunez, Jordany
• Application Number:
  US10/724,950
• Publication Number:
  US 20040193413A1
• Time in Patent Office: 3,837 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

A 3-D imaging system for recognition and interpretation of gestures to control a computer. The system includes a 3-D imaging system that performs gesture recognition and interpretation based on a previous mapping of a plurality of hand poses and orientations to user commands for a given user. When the user is identified to the system, the imaging system images gestures presented by the user, performs a lookup for the user command associated with the captured image(s), and executes the user command(s) to effect control of the computer, programs, and connected devices.

Claims

1. A computer-implemented system having memory, processor, and data storage subsystems that facilitates a user interface, the computer-implemented system comprising;

a tracking component to detect motion of a group of moving pixels included in two or more images by employing multiple hypotheses on the two or more images utilizing two or more trackers to track a detected object, such that a second tracker continues tracking the detected object upon failure of a first tracker, wherein the tracking component detects motion when a sum of absolute differences of at least one pixel within the two or more images is above a specified threshold, wherein a vision algorithm comprising 3-D tracking and 3-D depth computations is employed in association with the tracking component;

at least two video capturing sources configured to receive a user command to control the computer-implemented system, wherein the user command is received from a gesture extracted from the group of moving pixels included in the two or more images captured by the at least two video capturing sources; and

a 3-D imaging component that receives the gesture in a form of a gesture image, processes the gesture image, and interprets the gesture image to execute the user command for control of the computer-implemented system via a plurality of object hypothesis models which detect associated multiple objects and remove any redundantly tracked objects.

2. The computer-implemented system of claim 1, further comprising a voice communication system that receives voice signals that are used singly or in combination with the gesture to control the computer-implemented system.

3. The computer-implemented system of claim 1, wherein the 3-D imaging component determines when an operator is looking in a direction of the computer-implemented system.

4. The computer-implemented system of claim 1, wherein the gesture is formed by using at least one hand.

5. The computer-implemented system of claim 1, wherein the gesture is formed using a head movement.

6. The computer-implemented system of claim 1, wherein the 3-D imaging component initiates a confirmation request signal in response to receiving the gesture.

7. The computer-implemented system of claim 1, wherein the 3-D imaging component detects gesture characteristics in the gesture, which gesture characteristics include at least one of hand movement, finger count, finger orientation, hand orientation, and hand rotation.

8. The computer-implemented system of claim 1, wherein the 3-D imaging component continually monitors location of a given user bearing a tag via a triangulation system, and associates location data with captured image data such that gestures from that location will be processed against an associated user profile to properly execute the user command.

9. The computer-implemented system of claim 1, wherein the 3-D imaging component controls windows in the computer-implemented system based on dwell time of the received gesture wherein the dwell time is a time after having engaged the computer-implemented system, that the user holds a hand position stationary such that a system icon remains over a particular window.

10. The computer-implemented system of claim 1, further comprising a classifier that tracks, processes, compares and updates a user profile when a command gesture associated with a particular user command is changed within a specified criterion after a calibration process.

11. The computer-implemented system of claim 1, wherein the control of the computer-implemented system comprises controlling computer programs by manipulating on screen objects without a cursor.

12. The computer-implemented system of claim 1, wherein the 3-D imaging component permits different users to select different commands from a plurality of user commands executable by the computer-implemented system, to associate with the received gesture such that the received gesture causes execution of the user command based on a user profile and on a particular application executing on the computer-implemented system, wherein the gesture image causes execution of the user command by the computer-implemented system.

13. A computer-implemented method for facilitating a user interface, comprising:

detecting a first object;

tracking the first object based at least in part upon image comparison of a plurality of images relative to a course mapping of the images, the tracking performed in association with multiple object hypotheses utilizing multiple trackers comprising at least a first tracker and a second tracker to select and track the first object based on one or more criteria such that the second tracker continues to track the first object upon failure of the first tracker to track the first object, wherein motion is detected when a sum of absolute differences of at least one pixel within the plurality of images is above a specified threshold, and wherein a vision algorithm comprising 3-D tracking and 3-D depth computations is employed in association with the tracking of the first object;

providing iterative seeding in relation to the object hypotheses based at least in part upon a presence of the first object and the image comparison;

selectively removing at least one object from the object hypotheses or at least one object hypothesis from the object hypotheses

interpreting an object characteristic of the first object; and

executing a command in response to the object characteristic.

14. The computer-implemented method of claim 13, wherein the object characteristic comprises at least one of object motion, object orientation, object features, and object rotation.

15. The computer-implemented method of claim 13, wherein the command is executed based at least in part on audio input data.

16. The computer-implemented method of claim 13, wherein the first object is a hand controlled to provide a hand gesture.

17. The computer-implemented method of claim 13, wherein the command is executed based at least in part on at least one of keyboard input data and mouse input data.

18. A computer-implemented system having memory, processor, and data storage subsystems that facilitates a user interface, the computer-implemented system comprising;

a tracking component that tracks a first object based at least in part upon image comparison of a plurality of images relative to a course mapping of the images, the tracking performed in association with multiple object hypotheses utilizing multiple trackers comprising at least a first tracker and a second tracker to select and track the first object based on one or more criteria such that the second tracker continues to track the first object upon failure of the first tracker to track the first object, wherein the tracking component detects motion when a sum of absolute differences of at least one pixel within the plurality of images is above a specified threshold, wherein a vision algorithm comprising 3-D tracking and 3-D depth computations is employed in association with the tracking component;

a seeding component that iterative seeds the tracking component with object hypotheses based at least in part upon a presence of the first object and the image comparison;

a filtering component that selectively removes at least one object from the object hypotheses or at least one object hypothesis from the object hypotheses; and

an interpreting component that interprets an object characteristic of the first object and executes a command in response thereto.

19. The computer-implemented system of claim 18, wherein the object characteristic comprises at least one of object motion, object orientation, object features, and object rotation.

20. The computer-implemented system of claim 18, wherein the command is executed based at least in part on audio input data.

21. The computer-implemented system of claim 18, wherein the first object is a hand controlled to provide a hand gesture.

22. The computer-implemented system of claim 18, wherein the command is executed based at least in part on at least one of keyboard input data and mouse input data.