Method and System for Detecting and Recognizing Social Interactions In a Video

• US 20140270707A1
• Filed: 06/13/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Fathi, Alireza, Hodgins, Jessica, Rehg, James
• Granted Patent: US 9,159,362 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A system and method for providing a plurality of frames from a video, the video having been taken from a first person'"'"'s perspective, identifying patterns of attention depicted by living beings appearing in the plurality of frames, identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time and using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.

Claims

1. A method, comprising:

providing a plurality of frames from a video, the video having been taken from a first person'"'"'s perspective;

identifying patterns of attention depicted by living beings appearing in the plurality of frames;

identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time; and

using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.

2. The method of claim 1, wherein the social interaction is classified using a method that analyzes temporal changes in the patterns of attention.

3. The method of claim 2, wherein the method is one of a Hidden Conditional Random Field (“

HCRF”

) model, a Conditional Random Field (“

CRF”

) model, a Hidden Markov Model (“

HMM”

), and a Granger causality model.

4. The method of claim 2, wherein each of the frames is represented by a vector.

5. The method of claim 4, wherein the vector includes one of a location of faces present within the image with respect to the first-person'"'"'s perspective, an identification of roles present within the image, a representation of first-person head movement within the image, an identification of a speaker, a speech analysis, and an analysis of shifts in attention.

6. The method of claim 1, wherein the identifying patterns of attention comprises:

identifying a plurality of faces within each of the frames; and

determining a location of each identified face with respect to a first person perspective of the frame.

7. The method of claim 6, wherein the act of determining a location comprises classifying each of the plurality of faces based on a distance from a recording device that recorded the video recording and an angle from the recording device.

8. The method of claim 6, wherein the act of identifying patterns of attention further comprises:

determining an orientation of each of the plurality of faces.

9. The method of claim 1, wherein identifying patterns of attention comprises one of:

identifying a plurality of faces depicted in the frame;

determining a number of faces that are looking at a particular one of the other faces;

determining whether the first person is attending to the particular one face;

identifying whether the particular one face is attending to the first person; and

determining a number of faces within the frame that are looking at a same location as the first person.

10. The method of claim 9, wherein each of the plurality of faces is classified into a role based on the patterns of attention.

11. The method of claim 1, wherein the identifying social interactions comprises:

splitting frame vertically and horizontally into a plurality of sub-windows;

splitting a flow vector field in each sub-window into vertical and horizontal components; and

determining a vector that represents a mean value of motion for each of the sub-windows.

12. The method of claim 1, further comprising:

automatically editing the video recording based on the social interactions.

13. A video analysis system, comprising:

a memory storing a video taken from a first person'"'"'s perspective; and

a processor executing a set of instructions that cause the processor to perform a method, comprising

identifying patterns of attention depicted by living beings appearing in the plurality of frames;

identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time; and

using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.

14. The system of claim 13, wherein the social interaction is classified using a method that analyzes temporal changes in the patterns of attention.

15. The system of claim 14, wherein the method is one of a Hidden Conditional Random Field (“

HCRF”

) model, a Conditional Random Field (“

CRF”

) model, a Hidden Markov Model (“

HMM”

), and a Granger causality model.

16. The system of claim 13, wherein each of the frames is represented by a vector.

17. The system of claim 16, wherein the vector includes one of a location of faces present within the image with respect to the first-person'"'"'s perspective, an identification of roles present within the image, a representation of first-person head movement within the image, an identification of a speaker, a speech analysis, and an analysis of shifts in attention.

18. The system of claim 13, wherein the identifying patterns of attention comprises:

identifying a plurality of faces within each of the frames; and

determining a location of each identified face with respect to a first person perspective of the frame.

19. The system of claim 18, wherein the act of determining a location comprises classifying each of the plurality of faces based on a distance from a recording device that recorded the video recording and an angle from the recording device.

20. The system of claim 18, wherein the act of identifying patterns of attention further comprises:

determining an orientation of each of the plurality of faces.

21. The system of claim 13, wherein identifying patterns of attention comprises:

identifying a plurality of faces depicted in the frame;

determining a number of faces that are looking at a particular one of the other faces;

determining whether the first person is attending to the particular one face;

identifying whether particular one face attending to the first person; and

determining a number of faces within the frame that are looking at a same location as the first person.

22. The system of claim 21, wherein each of the plurality of faces is classified into a role based on the patterns of attention.

23. The system of claim 13, wherein the identifying social interactions comprises:

splitting frame vertically and horizontally into a plurality of sub-windows;

splitting a flow vector field in each sub-window into vertical and horizontal components; and

determining a vector that represents a mean value of motion for each of the sub-windows.

24. The system of claim 13, wherein the method further comprises:

automatically editing the video recording based on the social interactions.