Recognizing actions of animate objects in video

• US 8,396,247 B2
• Filed: 07/31/2008
• Issued: 03/12/2013
• Est. Priority Date: 07/31/2008
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Li, Wanqing, Liu, Zicheng
• Primary Examiner(s): Wang, Claire X
• Application Number:
  US12/183,078
• Publication Number:
  US 20100027835A1
• Time in Patent Office: 1,685 Days
• Field of Search: 823/103, 823/118, 382/103, 382/118
• US Class Current: 382/103
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 30/1988 Graph matching graphical image representation G06V30/18181
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A system that facilitates automatically determining an action of an animate object is described herein. The system includes a receiver component that receives video data that includes images of an animate object. The system additionally includes a determiner component that accesses a data store that includes an action graph and automatically determines an action undertaken by the animate object in the received video data based at least in part upon the action graph. The action graph comprises a plurality of nodes that are representative of multiple possible postures of the animate object. At least one node in the action graph is shared amongst multiple actions represented in the action graph.

Claims

1. A method comprising the following computer-executable acts:

receiving video data, wherein the video data includes a plurality of video frames that comprise an animate object;

accessing a data store that comprises data representable by an action graph responsive to receiving the video data, the action graph including a plurality of nodes, representative of a plurality of possible postures of the animate object, and at least one node of the action graph corresponding to multiple actions determinable through use of the action graph;

probabilistically identifying a sequence of nodes in the action graph based upon respective postures of the animate object in the plurality of video frames; and

determining an action undertaken by the animated object in the plurality of video frames based at least in part upon the probabilistically identifying the sequence of nodes in the action graph.

2. The method of claim 1, wherein the animate object is a human being.

3. The method of claim 1, wherein the act of determining comprises determining a most likely path in the action graph that corresponds to images in the plurality of video frames.

4. The method of claim 3, wherein the act of determining comprises:

extracting features from the received video data pertaining to the animate object; and

determining at least one posture of the animate object in the video data based at least in part upon the extracted features.

5. The method of claim 3, further comprising determining a likelihood of each of the multiple determinable actions based at least in part upon the received video data.

6. The method of claim 5, wherein the act of determining the likelihood of each of the multiple determinable actions comprises using one or more of Action Specific Viterbi Decoding, Unigram with Global Viterbi Decoding, Bi-gram with Global Viterbi Decoding, Uni-gram with Maximum Likelihood Decoding, or Bi-gram with Maximum Likelihood Decoding.

7. The method of claim 1, wherein the action graph includes transitional probabilities between multiple postures.

8. The method of claim 1, further comprising annotating the video data based at least in part upon the determined action.

9. The method of claim 1, further comprising:

receiving training data;

determining shape and motion dissimilarities with respect to images of animate objects in the training data; and

clustering postures based at least in part upon the determined shape and motion dissimilarities.

10. The method of claim 9, further comprising estimating transitional probabilities between clustered postures.

11. The method of claim 10, further comprising:

receiving a video of an animate object undertaking a new action;

determining postures that describe the new action;

comparing determined actions with postures existent in the action graph; and

removing from the action graph determined postures that are found to be similar to postures existent in the action graph.

12. A system that facilitates automatically determining an action of an animate object in received video data comprises the following computer-executable components:

a receiver component that receives video data that includes images of an animate object; and

a determiner component that accesses a data store that includes an action graph and automatically determines an action undertaken by the animate object in the received video data based at least in part upon the action graph, wherein the action graph comprises a plurality of nodes that are representative of multiple possible postures of the animate object, and wherein at least one node in the action graph is shared amongst multiple actions represented in the action graph.

13. The system of claim 12, wherein actions represented in the action graph comprise running, walking, and sliding.

14. The system of claim 12, wherein the animate object is a human being.

15. The system of claim 12, wherein the received video data comprises a plurality of silhouettes of the animate object.

16. The system of claim 12, further comprising:

a posture recognizer component that recognizes a plurality of salient postures of the animate object in the received video data;

a path determiner component that determines a most likely path in the action graph that corresponds to a sequence of received images in the video data; and

a probability determiner component that determines a likelihood of each action represented in the action graph.

17. The system of claim 12, further comprising an annotater component that annotates portions of the video data with information pertaining to the determined action.

18. The system of claim 12, further comprising a learner component that receives posture samples and learns the action graph based at least in part upon the received posture samples.

19. The system of claim 18, further comprising:

a shape dissimilarity determiner component that determines dissimilarity between different shape descriptors, wherein the shape descriptors are based at least in part upon the posture samples;

a motion dissimilarity determiner component that determines motion dissimilarity between motion feature vectors pertaining to the posture samples;

a clusterer component that creates clusters of posture samples based at least in part upon dissimilarity between different posture samples.

20. A computer-readable data storage device comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving a plurality of video frames, wherein the plurality of video frames include a sequence of silhouettes of a human being;

determining a plurality of postures of the human being in the sequence of silhouettes;

comparing the plurality of postures with postures represented in an action graph, wherein the action graph includes multiple postures pertaining to numerous actions, and wherein a first posture in the action graph is linked to a second posture in the action graph by a probability that for a first action the human being will transition from the first posture to the second posture, and wherein at least one posture in the action graph corresponds to more than one action;

determining a most likely action that corresponds to the determined plurality of postures based at least in part upon the comparison; and

outputting the most likely action as a determined action.