3D Visual Proxemics: Recognizing Human Interactions in 3D From a Single Image

• US 20140270363A1
• Filed: 08/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Javed, Omar, Chakraborty, Ishani, Cheng, Hui
• Granted Patent: US 9,268,994 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A unified framework detects and classifies people interactions in unconstrained user generated images. Previous approaches directly map people/face locations in two-dimensional image space into features for classification. Among other things, the disclosed framework estimates a camera viewpoint and people positions in 3D space and then extracts spatial configuration features from explicit three-dimensional people positions.

Claims

1-3. -3. (canceled)

4. A method for recognizing a human interaction depicted in a two-dimensional image, the method comprising, algorithmically:

detecting a plurality of human face locations of people depicted in the image;

determining a three-dimensional spatial arrangement of the people depicted in the image based on the detected human face locations;

performing a proxemics-based analysis of the three-dimensional spatial arrangement of the people depicted in the image, wherein the proxemics-based analysis identifies cues in the three-dimensional spatial arrangement that are relevant to human interactions; and

classifying the image as depicting a type of human interaction using visual proxemes, wherein the visual proxemes comprise a set of prototypical patterns that represent commonly occurring people interactions.

5. The method of claim 4, wherein the image is created by a camera positioned at a camera viewpoint relative to a reference plane, and the method comprises estimating the camera viewpoint and using the estimated camera viewpoint to classify the image.

6. The method of claim 5, comprising classifying the camera viewpoint as a high-angle viewpoint, an eye-level viewpoint, or a low-angle viewpoint.

7. The method of claim 4, comprising analyzing the plurality of detected human face locations using a linear camera model, identifying a face location that does not fit the linear camera model as an outlier, identifying a face location that fits the linear camera model as an inlier, determining the position of the outlier in relation to the inlier, and classifying the image as depicting a type of human interaction based on the position of the outlier in relation to the inlier.

8. The method of claim 7, comprising analyzing the position of the outlier in relation to the inlier using one or more visual proxemics-based constraints.

9. The method of claim 4, comprising detecting, in the image, a person standing in front of another person by applying a proxemics-based visibility constraint.

10. The method of claim 4, comprising detecting, in the image, a child and an adult by applying a proxemics-based localized pose constraint.

11. The method of claim 4, comprising classifying the image as depicting a group interaction, a family photo, a group photo, a couple with an audience, a crowd scene, or a speaker and an audience.

12. The method of claim 4, comprising alternating between estimating a camera parameter of the camera used to create the image and applying proxemics-based constraints to the three-dimensional spatial arrangement of the human face locations detected in the image to identify the type of human interaction depicted by the image.

13. The method of claim 4, comprising detecting a plurality of feature cues in the image, wherein each of the feature cues relates to a proxemics-based attribute.

14. The method of claim 13, wherein the plurality of feature cues comprises a shape cue that indicates a shape of the spatial arrangement of the detected face locations, a shot composition cue that indicates a visual distribution of the people depicted in the image, a distance cue that measures distances between the detected face locations in the image, a camera pose cue that estimates the height of the camera used to capture the image in relation to the people depicted in the image relative to a ground plane, and a shape layer cue that indicates whether the people depicted in the image are arranged in a single group or in separate subgroups.

15. The method of claim 4, comprising creating a collection of classified images by repeating the detecting, determining, performing, and classifying for a plurality of two-dimensional images and arranging the classified images in a collection according to human interaction type.

16. The method of claim 15, comprising searching the collection using search criteria including a human interaction type.

17. The method of claim 15, comprising retrieving an image from the collection based on a human interaction type.

18. A framework to detect and classify people interactions depicted in a two-dimensional digital image, the framework comprising:

a visual proxemics knowledge base comprising a semantic vocabulary that associates inter-person distances with classes of people interactions;

a perspective rectification module to estimate a three-dimensional position of a person depicted in the image relative to at least one other person depicted in the image; and

a proxemics classification module to classify the image as depicting a type of people interaction using the visual proxemics knowledge base.

19. The framework of claim 18, comprising a camera pose estimation and classification module to estimate a position of a camera used to capture the image, wherein the framework relates the camera position to a proxemics-based attribute.

20. The framework of claim 18, comprising an outlier detection module to detect human face locations in the image that do not conform to a linearized model of people heights and poses and identify the detected human face locations as outliers.

21. The framework of claim 20, wherein the outlier detection module reasons about the locations of the outliers in relation to human face locations detected in the image that do conform to the linearized model by applying proxemics-based semantic constraints.

22. The framework of claim 18, wherein the proxemes classification module determines a proxemes class likely associated with the image.

23. The framework of claim 18, wherein the framework estimates a three dimensional face depth and a camera height from a two-dimensional face location detected in the image.