Recognizing Entity Interactions in Visual Media

• US 20140270482A1
• Filed: 09/09/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Chakraborty, Ishani, Cheng, Hui, Javed, Omar
• Granted Patent: US 9,330,296 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

An entity interaction recognition system algorithmically recognizes a variety of different types of entity interactions that may be captured in two-dimensional images. In some embodiments, the system estimates the three-dimensional spatial configuration or arrangement of entities depicted in the image. In some embodiments, the system applies a proxemics-based analysis to determine an interaction type. In some embodiments, the system infers, from a characteristic of an entity detected in an image, an area or entity of interest in the image.

Claims

1. A method for recognizing an entity interaction depicted in a two-dimensional image or a video, the entity interaction involving an entity depicted in the image, the method comprising, with a computing system, algorithmically:

detecting the entity in the image;

estimating a three-dimensional spatial configuration of the entity in the image; and

classifying the image as depicting a type of entity interaction involving the detected entity based on the estimated three-dimensional spatial configuration of the detected entity.

2. The method of claim 1, wherein the entity comprises a person depicted in the image, and the method comprises classifying the image as depicting a type of human interaction involving the person.

3. The method of claim 2, comprising detecting a non-person entity in the image, estimating a three-dimensional arrangement of the person and the non-person entity as depicted in the image, and classifying the image as depicting a type of human interaction involving the person and the non-person entity based on the estimated three-dimensional arrangement of the person and the non-person entity.

4. The method of claim 2, comprising detecting a plurality of persons depicted in the image, estimating a three-dimensional position of each of the persons in the detected plurality of persons, and classifying the image as depicting a type of human interaction involving at least one of the detected persons based on the estimated three-dimensional position of each of the persons in the detected plurality of persons.

5. The method of claim 2, comprising detecting a plurality of persons depicted in the image, estimating a three-dimensional arrangement of the detected plurality of persons, and classifying the image as depicting a type of human interaction involving the detected plurality of persons based on the estimated three-dimensional arrangement of the detected plurality of persons.

6. The method of claim 5, comprising determining a proxemics class associated with the three-dimensional arrangement of the persons depicted in the image, wherein the proxemics class represents a commonly occurring human interaction.

7. The method of claim 6, comprising estimating a parameter of a device likely used to capture or record the image, and using the estimated parameter of the device likely used to capture or record the image to determine the proxemics class.

8. The method of claim 1, comprising estimating a three-dimensional arrangement of a plurality of entities depicted in the image, and classifying the image as depicting a type of entity interaction involving the plurality of entities based on the estimated three-dimensional arrangement of the plurality of entities.

9. The method of claim 1, wherein the image depicts a live physical reality observed by a user of a device used to capture the image, and the method further comprises presenting a suggestion to the user based on the type of human interaction associated with the classified image.

10. The method of claim 1, comprising tagging the image with a plurality of semantic elements usable to describe the type of human interaction associated with the classified image and storing the semantic elements in computer memory.

12. The method of claim 1, comprising repeating the detecting and estimating for a second entity depicted in the image, and modifying the classification of the image based on the estimated three-dimensional position of the second entity.

13. The method of claim 1, comprising inferring an area of interest or an entity of interest in the image based on the estimated three-dimensional spatial configuration of the at least one detected entity.

14. A computer application comprising one or more of an image/video tagger, an information retrieval system, and an intelligent assistant, wherein the computer application is embodied in one or more machine accessible storage media, and the computer application comprises instructions accessible by one or more processors to cause the one or more processors to execute the method of claim 1.

15. A computing system comprising the one or more processors and the one or more machine accessible storage media having embodied therein the computer application of claim 14.

11. The method of claim A10, comprising converting user input into a computer-executable search query comprising one or more of the semantic elements.

16. A method for inferring an area of interest in a two-dimensional image depicting at least one entity, the method comprising, with a computing system, algorithmically:

locating the at least one entity in the image;

determining, from the image, a spatial configuration of at least a portion of the at least one entity located in the image; and

inferring an area of interest in the image based on the location of the at least one entity and the determined spatial configuration, the area of interest at least partially spaced from the at least one entity, the area of interest having a size that is greater than zero and less than the size of the entire image.

17. The method of claim 16, wherein at least one of the entities comprises a person, and the method comprises inferring the area of interest based on the location of the person in the image and the determined spatial configuration.

18. The method of claim 17, wherein the plurality of characteristics comprises a face pose and a hand location of the person, and the method further comprises estimating the face pose, estimating the hand location based on the estimated face pose, and inferring the area of interest based on the estimated face pose and the estimated hand location.

19. The method of claim 17, wherein the spatial configuration comprises an estimated face pose of the person, and the method comprises computing a direction of the person'"'"'s gaze from the estimated face pose, and inferring the area of interest based on the computed direction of the person'"'"'s gaze.

20. The method of claim 19, further comprising estimating a location of the person'"'"'s hand and inferring the area of interest based on the location of the person'"'"'s hand.

21. The method of claim 16, wherein the area of interest comprises a plurality of entities of possible interest, and the method comprises filtering the plurality of entities of possible interest to a smaller number of entities of possible interest based on the determined spatial configuration of the at least one entity depicted in the image.

22. The method of claim 16, wherein the inferring an area of interest comprises inferring a three-dimensional area of interest in the recorded two-dimensional image.

23. The method of claim 16, wherein the at least one entity comprises a person and the method comprises estimating a three-dimensional position of the person from the determined plurality of characteristics, analyzing the three-dimensional position of the person using a proxemics analysis, determining a type of human interaction likely depicted in the image based on the proxemics analysis, and inferring the area of interest based on the determined type of human interaction.

24. A computer application comprising one or more of an image/video tagger, an information retrieval system, and an intelligent assistant, wherein the computer application is embodied in one or more machine accessible storage media, and the computer application comprises instructions accessible by one or more processors to cause the one or more processors to execute the method of claim 16.

25. A computing system comprising the one or more processors and the one or more machine accessible storage media having embodied therein the computer application of claim 24.

26. The method of claim 17, comprising detecting an attribute of the person depicted in the image, computing the size of the attribute detected in the image, comparing the size of the detected attribute to a threshold size, and inferring the area of interest in the image based on the comparison of the size of the detected attribute to the threshold size.

27. The method of claim 26, wherein the attribute of the person comprises the person'"'"'s face, and the method comprises computing the size of the detected face, comparing the size of the detected face to a threshold face size and inferring the area of interest in the image based on the comparison of the size of the detected face to the threshold face size.

28. The method of claim 16, wherein the inferred area of interest comprises a plurality of entities of possible interest and the method comprises using one or more characteristics of one or more of the entities of possible interest to filter the plurality of entities of possible of interest to a smaller number of entities of possible interest.

29. The method of claim 16, wherein the at least one entity comprises a surface, and the method comprises determining a three-dimensional arrangement of the surface and another entity depicted in the image based on a characteristic of the surface.

30. The method of claim 29, comprising associating a surface affordance with the surface based on the characteristic of the surface and the three-dimensional arrangement of the surface and a person depicted in the image.

31. A method for detecting a characteristic of a gaze of a person depicted in a recorded two-dimensional image, the method comprising, with a computing system, algorithmically:

detecting the person in the recorded two-dimensional image;

estimating a three-dimensional spatial configuration of at least a portion of the person in the image; and

inferring a characteristic of the person'"'"'s gaze in the image based on the estimated spatial configuration.

32. The method of claim 31, wherein the estimated spatial configuration comprises an estimated head pose of the detected person and the method comprises inferring a characteristic of the person'"'"'s gaze in the image based on the estimated head pose.

33. The method of claim 31, comprising classifying the recorded two-dimensional image as depicting a type of human interaction based on the inferred characteristic of the person'"'"'s gaze.

34. The method of claim 33, wherein the recorded image comprises a plurality of images and the method comprises tracking the person'"'"'s gaze by repeating the detecting, estimating, and inferring for the plurality of images over a time period, and reclassifying the recorded image as depicting a different type of human interaction based on the tracking of the person'"'"'s gaze over the time period.

35. The method of claim 33, comprising determining a proxemics class associated with the recorded two-dimensional image based on the estimated spatial configuration, and inferring an area of interest in the image at least partially spaced from the detected person based on the proxemics class.

36. The method of claim 31, comprising inferring a direction of the person'"'"'s gaze, and inferring an area of interest in the recorded two-dimensional image based on the direction of the person'"'"'s gaze, wherein the area of interest at least partially spaced from the person, and the area of interest has a size that is greater than zero and less than the size of the entire recorded two-dimensional image.

37. The method of claim 36, wherein the inferring an area of interest comprises inferring a three-dimensional area of interest from the recorded two-dimensional image.

38. A computer application comprising one or more of an image/video tagger, an information retrieval system, and an intelligent assistant, wherein the computer application is embodied in one or more machine accessible storage media, and the computer application comprises instructions accessible by one or more processors to cause the one or more processors to execute the method of claim 31.

39. A computing system comprising the one or more processors and the one or more machine accessible storage media having embodied therein the computer application of claim 38.