Gesture recognition system recognizing gestures within a specified timing

• US 7,000,200 B1
• Filed: 09/15/2000
• Issued: 02/14/2006
• Est. Priority Date: 09/15/2000
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Martins, Fernando C. M.
• Primary Examiner(s): Vu, Kieu D.
• Application Number:
  US09/662,679
• Time in Patent Office: 1,978 Days
• Field of Search: 315/863, 315716-776, 382/190, 382/209, 434/250, 434/252, 463/7, 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Gesture recognition in which timing data is used to temporally segment video data into video clips. The timing data can be beat data extracted from an audio signal.

Claims

1. A method for recognizing gestures in video data at times associated with music comprising:

receiving audio data which represents music that has a beat formed of an audibly perceptible periodic pulse contained within the audio data;

obtaining beat data indicative of said beat data for said audio data;

playing said audio data, and using said beat to determine a gesture window associated with said audio data;

obtaining video data during the time of said gesture window during which a target gesture should occur;

segmenting said video data to form individual frames of video data and investigating said frames to recognize gestures in said frames and to determine gesture probabilities which represent whether said gestures represent target gestures; and

determining if a gesture probability associated with said gestures represents that the target gesture has been determined during said frame.

2. The method of claim 1, wherein determining the probability that the video data contains each of the target gestures includes evaluations of Hidden Markov Models.

3. The method of claim 1, further comprising displaying a target gesture to be performed by the subject of the video data.

4. The method of claim 1, further comprising generating a feature vector for each video frame of the video data.

5. The method of claim 1, further comprising generating a score based on whether the video data contains the target gesture.

6. The method of claim 5, further comprising displaying the score.

7. A system for recognizing gestures in video data at times associated with music comprising:

an audio receiving part that receives audio data representing music that has a beat formed of an audibly perceptible periodic pulse contained within the audio data;

a recognition subsystem and timing data source, operative to determine said beat data indicative of said beat, and to define a gesture window of time based on said beat, during which gesture window, a target gesture should occur;

a video device, receiving video, and segmenting said video into a plurality of different frames; and

wherein said recognition subsystem and timing data source is also operative to recognize gestures within said frames and determine probabilities of which gestures are represented within the frames and whether the gestures represent the target gesture.

8. The system of claim 7, wherein the recognition subsystem and timing data source operates based on a plurality of Hidden Markov Models.

9. The system of claim 7, further comprising:

a video source, in communication with the recognition subsystem and timing data source, to provide the video to the recognition subsystem and timing data source.

10. The system of claim 7, further comprising a move subsystem, in communication with the recognition subsystem and timing data source, to provide said target gesture to be performed by the subject of the video.

11. The system of claim 10, wherein the target gesture is a dance move that is to be performed by the subject of the video.

12. The system of claim 10, further comprising a scoring subsystem, in communication with the recognition subsystem and timing data source and the move subsystem, to determine a score based on whether the video contains the target gesture.

13. The system of claim 12, further comprising a display subsystem, in communication with the scoring subsystem, to display a score that is a function of whether the video contains the target gesture.

14. The system of claim 13, wherein the display subsystem is in communication with the move subsystem and is configured to display a gesture request based on the target gesture.

15. A system as in claim 7, wherein said recognition subsystem and timing data source is also operative to analyze said audio data to automatically determine beat data associated with the audio data.

16. A system as in claim 15, wherein said audio data includes MIDI data and said recognition subsystem and timing data source analyzes the MIDI data to automatically determine the data from a specified channel of the MIDI data.

17. A computer program product, tangibly stored on a computer readable medium, for recognizing gestures in video data at times associated with music, comprising instructions operable to cause a programmable processor to:

receive audio data which represents music that has a beat formed of an audibly perceptible periodic pulse contained within the audio data;

obtaining beat data indicative of said beat associated with said audio data;

play said audio data, and using said beat data to determine a gesture window associated with the audio data;

obtain video data during a time defined by said gesture window during which a target gesture should occur;

segment said video data to form individual frames of video data and investigate said individual frames to determine gesture probabilities indicative of whether target gestures are contained in said frames; and

determine if a gesture probability represents that the target gesture has been determined during said frame.

18. A product as in claim 17, wherein said obtaining beat data comprises extracting beat data automatically from the audio data.

19. A product as in claim 18, wherein said extracting comprises analyzing a MIDI sequence representing said audio data, and extracting beat data from a specified channel of the audio data that represents a drumbeat.

20. A computer program product as in claim 17, further comprising instructions to automatically analyze said audio data to automatically determine beat data associated with the audio data.

21. A computer program product as in claim 20, wherein said audio data includes MIDI data, and said instructions analyze a specified channel of the MIDI data to determine the data from said specified channel.