Method and system for detecting and recognizing social interactions in a video

• US 9,159,362 B2
• Filed: 06/13/2013
• Issued: 10/13/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Hodgins, Jessica, Rehg, James, Fathi, Alizreza
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Wendmagegn, Girumsew
• Application Number:
  US13/917,267
• Publication Number:
  US 20140270707A1
• Time in Patent Office: 852 Days
• Field of Search: 348/169, 386/278
• US Class Current: 1/1
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A system and method for providing a plurality of frames from a video, the video having been taken from a first person'"'"'s perspective, identifying patterns of attention depicted by living beings appearing in the plurality of frames, identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time and using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.

Claims

1. A method of detecting and recognizing social interactions in a video, the method comprising:

obtaining a video recording captured from a video recording device, the video comprising a plurality of frames, the video having been taken from a perspective of a person capturing the video within an environment, individual ones of the frames depicting a first-person perspective of the person within the environment;

identifying patterns of attention depicted by living beings appearing in one or more of the plurality of frames;

identifying one or more social interactions associated with the plurality of frames using the identified patterns of attention over one or more periods of time, such that a first social interaction is identified over a first period of time;

associating individual ones of the plurality of frames and/or sets of one or more of the plurality of frames with one or more of the identified social interactions, a first set of frames being associated with the first social interaction; and

using the identified social interactions and associated frames to affect one or more of a subsequent use, storage, presentation, or processing of one or more of the plurality of frames, such that the association of the first set of frames with the first social interaction affects one or more of the subsequent use, storage, presentation, or processing of one or more of the plurality of frames.

2. The method of claim 1, wherein the social interactions are identified based on temporal changes in the identified patterns of attention.

3. The method of claim 2, wherein individual ones of the plurality of frames are represented by a vector.

4. The method of claim 3, wherein individual ones of the vectors include one or more of a location of faces present within an image depicted in individual ones of the frames with respect to the first-person'"'"'s perspective, an identification of roles present within the image, a representation of first-person head movement within the image, an identification of a speaker, a speech analysis, or an analysis of shifts in attention.

5. The method of claim 1, wherein the identifying patterns of attention comprises:

identifying a plurality of faces within individual ones of the frames; and

determining a location of individual ones of the identified faces with respect to the first-person perspective of the corresponding frame.

6. The method of claim 5, wherein determining a location comprises classifying individual ones of the plurality of faces based on a distance from the recording device and an angle from the recording device.

7. The method of claim 5, wherein identifying patterns of attention further comprises:

determining an orientation of individual ones of the plurality of faces.

8. The method of claim 1, wherein identifying patterns of attention comprises one or more of:

identifying a plurality of faces depicted in a given frame;

determining a number of faces that are looking at a particular one of the other faces;

determining whether the first person is attending to the particular one face;

identifying whether the particular one face is attending to the first person;

or

determining a number of faces within the given frame that are looking at a same location as the first person.

9. The method of claim 8, wherein individual ones of the plurality of faces are classified into a role based on the identified patterns of attention.

10. The method of claim 1, wherein the identifying social interactions comprises:

splitting frame vertically and horizontally into a plurality of sub-windows;

splitting a flow vector field in individual ones of the sub-windows into vertical and horizontal components; and

determining a vector that represents a mean value of motion for individual ones of the sub-windows.

11. The method of claim 1, further comprising:

automatically editing the video recording based on the social interactions and associated frames, such that the automatically edited video includes the first set of frames associated with the first social interaction.

12. The method of claim 2, wherein the social interactions are further identified based on one or more of a Hidden Conditional Random Field (“

HCRF”

) model, a Conditional Random Field (“

CRF”

) model, a Hidden Markov Model (“

HMM”

), or a Granger causality model.

13. A video analysis system configured for detecting and recognizing social interactions in a video, the system comprising:

a storage media storing a video recording captured from a video recording device, the video comprising a plurality of frames, the video having been taking from a perspective of a person capturing the video within an environment, individual ones of the frames depicting a first-person perspective of the person within the environment; and

one or more physical processors configured by computer-readable instructions to;

identify patterns of attention depicted by living beings appearing in one or more of the plurality of frames;

identify one or more social interactions associated with the plurality of frames using the identified patterns of attention over one or more periods of time, such that a first social interaction is identified over a first period of time;

associating individual ones of the plurality of frames and/or sets of one or more of the plurality of frames with one or more of the identified social interactions, a first set of frames being associated with the first social interaction; and

using the identified social interactions and associated frames to affect one or more of a subsequent use, storage, presentation, or processing of the plurality of frames, such that the association of the first set of frames with the first social interaction affects one or more of the subsequent use, storage, presentation, or processing of one or more of the plurality of frames.

14. The system of claim 13, wherein the one or more physical processors are further configured by computer-readable instructions such that the social interactions are identified based on temporal changes in the patterns of attention.

15. The system of claim 14, wherein the one or more physical processors are further configured by computer-readable instructions such that the social interactions are further identified based on one or more of a Hidden Conditional Random Field (“

HCRF”

) model, a Conditional Random Field (“

CRF”

) model, a Hidden Markov Model (“

HMM”

), or a Granger causality model.

16. The system of claim 13, wherein the storage media is further configured such that individual one of the plurality of frames are represented by a vector.

17. The system of claim 16, wherein the storage media is further configured such that individual ones of the vectors include one or more of a location of faces present within an image depicted in individual ones of the frames with respect to the first-person'"'"'s perspective, an identification of roles present within the image, a representation of first-person head movement within the image, an identification of a speaker, a speech analysis, or an analysis of shifts in attention.

18. The system of claim 13, wherein the one or more physical processors are further configured by computer-readable instructions such that identifying patterns of attention comprises:

identifying a plurality of faces within individual ones of the frames; and

determining a location of individual ones of the identified faces with respect to the first-person perspective of the corresponding frame.

19. The system of claim 18, wherein the one or more physical processors are further configured by computer-readable instructions such determining a location comprises classifying individual ones of the plurality of faces based on a distance from the recording device and an angle from the recording device.

20. The system of claim 18, wherein the one or more physical processors are further configured by computer-readable instructions such that identifying patterns of attention further comprises:

determining an orientation of individual ones of the plurality of faces.

21. The system of claim 13, wherein the one or more physical processors are further configured by computer-readable instructions such that identifying patterns of attention comprises one or more of:

identifying a plurality of faces depicted in the frame;

determining a number of faces that are looking at a particular one of the other faces;

determining whether the first person is attending to the particular one face;

identifying whether particular one face attending to the first person;

or

determining a number of faces within the frame that are looking at a same location as the first person.

22. The system of claim 21, wherein the one or more physical processors are further configured by computer-readable instructions such that individual ones of the plurality of faces are classified into a role based on the patterns of attention.

23. The system of claim 13, wherein the one or more physical processors are further configured by computer-readable instructions such that identifying social interactions comprises:

splitting frame vertically and horizontally into a plurality of sub-windows;

splitting a flow vector field in individual ones of the sub-windows into vertical and horizontal components; and

determining a vector that represents a mean value of motion for individual ones of the sub-windows.

24. The system of claim 13, wherein the one or more physical processors are further configured by computer-readable instructions to:

automatically edit the video recording based on the social interactions and associated frames, such that the edited video includes the first set of frames associated with the first social interaction.