Method and apparatus for real-time gesture recognition

• US 6,256,033 B1
• Filed: 08/10/1999
• Issued: 07/03/2001
• Est. Priority Date: 10/15/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Planet Electric Incorporated
• Inventors: Nguyen, Katerina H.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Thai, Cuong T.
• Application Number:
  US09/371,214
• Time in Patent Office: 693 Days
• Field of Search: 345/326, 345/327, 345/358, 345/156, 382/107, 382/209, 382/217, 382/218
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system and method are disclosed for providing a gesture recognition system for recognizing gestures made by a moving subject within an image and performing an operation based on the semantic meaning of the gesture. A subject, such as a human being, enters the viewing field of a camera connected to a computer and performs a gesture, such as flapping of the arms. The gesture is then examined by the system one image frame at a time. Positional data is derived from the input frames and compared to data representing gestures already known to the system. The comparisons are done in real-time and the system can be trained to better recognize known gestures or to recognize new gestures. A frame of the input image containing the subject is obtained after a background image model has been created. An input frame is used to derive a frame data set that contains particular coordinates of the subject at a given moment in time. This series of frame data sets is examined to determine whether it conveys a gesture that is known to the system. If the subject gesture is recognizable to the system, an operation based on the semantic meaning of the gesture can be performed by a computer.

Claims

1. A computer-implemented method of storing and recognizing an aspect of a subject within an image, the method including:

a) building a background model by obtaining at least one frame of an image;

b) obtaining a data frame containing a subject;

c) removing background for said data frame based on said background model whereby the subject is isolated;

d) analyzing the data frame thereby determining particular coordinates of the subject at a particular time while the subject is moving;

e) adding the particular coordinates to a frame data set; and

f) examining the particular coordinates such that the particular coordinates are compared to positional data making up a plurality of recognizable aspects, wherein a recognizable aspect is made up of at least one dimension such that the positional data describes dimensions of the recognized aspect.

2. A method as recited in claim 1 wherein the computer includes a network connection capable of being coupled to a network.

3. A method as recited in claim 1 wherein the data frame is transmitted over the network.

4. A method as recited in claim 1 wherein the positional data is transmitted over the network.

5. A method as recited in claim 1 and further comprising:

e) repeating a through d for a plurality of data frames; and

f) determining whether the plurality of the data frames when examined in a particular sequence, conveys a subject aspect by the subject that resembles a recognizable aspect, thereby causing an operation based on a predetermined meaning of the recognizable aspect be performed by a computer.

6. A computer program embodied on a computer-readable medium that stores and recognizes an aspect of a subject within an image, including:

a) a code segment that builds a background model by obtaining at least one frame of an image;

b) a code segment that obtains a data frame containing a subject;

c) a code segment that subtracts background for said data frame based on said background model;

d) a code segment that analyzes the data frame thereby determining particular coordinates of the subject at a particular time while the subject is moving;

e) a code segment that adds the particular coordinates to a frame data set; and

f) a code segment that examines the particular coordinates such that the particular coordinates are compared to positional data making up a plurality of recognizable aspects, wherein a recognizable aspect is made up of at least one dimension such that the positional data describes dimensions of the recognized aspect.

7. A computer program embodied on a computer-readable medium as recited in claim 6 wherein the computer includes a network connection capable of being coupled to a network.

8. A computer program embodied on a computer-readable medium as recited in claim 6 wherein the data frame is transmitted over the network.

9. A computer program embodied on a computer-readable medium as recited in claim 6 wherein the positional data is transmitted over the network.

10. A computer program embodied on a computer-readable medium as recited in claim 6 and further comprising:

e) a code segment that repeats a through d for a plurality of data frames; and

f) a code segment that determines whether the plurality of the data frames when examined in a particular sequence, conveys a subject aspect by the subject that resembles a recognizable aspect, thereby causing an operation based on a predetermined meaning of the recognizable aspect be performed by a computer.

11. A computer-implemented method of storing and recognizing aspect of a subject within an image, the method including:

building a background model by obtaining at least one frame of an image;

storing a plurality of samples of a subject;

subtracting background for said data frame based on said background model;

inputting a number of key points that fit in an aspect of the subject;

inputting a corresponding time value representing the time of the aspect of the subject to complete; and

wherein the key points and corresponding time value are adapted for being used to recognize aspect of the subject.

12. A method as recited in claim 11 further including:

determining locations of key points in a model representative of the aspect of the subject.

13. A method as recited in claim 12 further including:

refining the model such that the plurality of samples of the aspect of the subject fit within the model.

14. A method as recited in claim 11 further including:

calculating a probability distribution for key points indicating the likelihood that a certain output will be observed.

15. A method as recited in claim 14 further including:

calculating a confusion matrix wherein the aspect of the subject is compared with previously stored recognizable aspects so that similarities between the new aspect to previously stored recognizable aspect can be determined.

16. A method as recited in claim 11 further including:

inputting a number of dimensions of the aspect of the subject.

17. A computer program embodied on a computer-readable medium that stores and recognizes an aspect of a subject within an image, including:

a code segment that builds a background model by obtaining at least one frame of an image;

a code segment that stores a plurality of samples of a subject;

a code segment that subtracts background for said data frame based on said background model;

a code segment that inputs a number of key points that fit in an aspect of the subject;

a code segment that inputs a corresponding time value representing the time of the aspect of the subject to complete; and

wherein the key points and corresponding time value are adapted for being used to recognize aspect of the subject.

18. A computer program embodied on a computer-readable medium as recited in claim 17 further including:

a code segment that determines locations of key points in a model representative of the aspect of the subject.

19. A computer program embodied on a computer-readable medium as recited in claim 18 further including:

a code segment that refines the model such that the plurality of samples of the aspect of the subject fit within the model.

20. A computer program embodied on a computer-readable medium as recited in claim 17 further including:

a code segment that calculates a probability distribution for key points indicating the likelihood that a certain output will be observed.

21. A computer program embodied on a computer-readable medium as recited in claim 20 further including:

a code segment that calculates a confusion matrix wherein the aspect of the subject is compared with previously stored recognizable aspects so that similarities between the new aspect to previously stored recognizable aspect can be determined.

22. A computer program embodied on a computer-readable medium as recited in claim 17 further including:

a code segment that inputs a number of dimensions of the aspect of the subject.