System and method for computing the visual profile of a place

• US 9,298,982 B2
• Filed: 07/26/2011
• Issued: 03/29/2016
• Est. Priority Date: 07/26/2011
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Perronnin, Florent, Gurin, Nicolas, Saunders, Craig John
• Primary Examiner(s): Park, Edward
• Application Number:
  US13/190,753
• Publication Number:
  US 20130028508A1
• Time in Patent Office: 1,708 Days
• Field of Search: 382/224, 382/231
• US Class Current: 1/1
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

A system and method for computing a place profile are disclosed. The method includes providing a geographical definition of a place, retrieving a set of images based on the geographical place definition. With a classifier, image-level statistics for the retrieved images are generated. The classifier has been trained to generate image-level statistics for a finite set of classes, such as different activities. The image-level statistics are aggregated to generate a place profile for the defined place which may be displayed to a user who has provided information for generating the geographical definition or used in an application such as a recommender system or to generate a personal profile for the user.

Claims

1. A method for computing a place profile, comprising:

providing a geographical definition of a place;

retrieving a set of images based on the geographical place definition;

for each of the images in the set of images, generating an image-level representation comprising extracting a set of patch statistics for each of a set of patches of the image and aggregating the patch statistics into an image-level representation, the patch statistics being based on features of pixels of the respective patch;

with a classifier, generating image-level statistics for the retrieved images based on the image-level representations including labeling images with class labels, the classifier having been trained to generate image-level statistics for a finite set of classes, and filtering a class label of a retrieved image based on a computed geographical density of other retrieved images with a same class label; and

aggregating the image-level statistics to generate a place profile for the geographically-defined place,

wherein at least the generating and aggregating are performed with a computer processor.

2. The method of claim 1, further comprising training the classifier on a training set of images labeled according to class.

3. The method of claim 1, wherein the providing of the geographical definition of the place comprises receiving geographical coordinates for a place and defining a region based on the received coordinates.

4. The method of claim 1, wherein the providing of the geographical definition of the place comprises providing for a map to be presented to a user and receiving a user'"'"'s selection of a place on the map.

5. The method of claim 1, wherein the retrieving of the set of images based on the geographical definition of the place comprises retrieving images based on geoposition information associated with the retrieved images.

6. The method of claim 1, wherein the generating image-level statistics further comprises labeling images with class label where the classifier score for the class exceeds a threshold value.

7. The method of claim 1, wherein the aggregating of the image-level statistics to generate a place profile comprises computing, for each class in the set, at least one of:

an optionally weighted sum of images in the set of images that are labeled by the classifier with the class; and

a sum of the different users whose retrieved images in the set of images are labeled with the class.

8. The method of claim 1 wherein the classes comprise tourist activities.

9. The method of claim 1, further comprising outputting the place profile.

10. The method of claim 9, wherein the outputting comprises generating a representation of the place profile for display to a user on a graphical user interface.

11. The method of claim 1, further comprising generating a recommendation of a place for display to a user based on a user'"'"'s profile and place profiles for a set of places, each of the place profiles being computed according to the method of claim 1.

12. The method of claim 1, further comprising comparing a first place profile with at least one second place profile, the first and second place profiles being generated according to the method of claim 1.

13. The method of claim 1, further comprising generating a user profile based on a plurality of place profiles generated according to the method of claim 1.

14. A system for generating a place profile comprising memory which stores instructions for performing the method of claim 1 and a processor, in communication with the memory, for executing the instructions.

15. A computer program product comprising a non-transitory recording medium which stores instructions which, when executed by a computer, perform the method of claim 1.

16. A graphical user interface comprising a display which receives a selection of a place from a user and displays a place profile which has been generated by the method of claim 1.

17. The method of claim 1, further comprising:

for a predefined set of activity classes, training the classifier on training images labeled according to the activity classes in the set of activity classes; and

with the trained classifier, generating the image-level statistics for the retrieved images in the set of retrieved images.

18. The method of claim 1, wherein the aggregating the image-level statistics to generate a place profile comprises generating a histogram of class scores generated through aggregation of the image-level statistics.

19. A system for computing a place profile, comprising:

a place definition component for generating a geographical definition of a place, based on information received from a user;

an image selection component for retrieving a set of images corresponding to the geographical definition of the place from an associated database of images tagged with geoposition information;

a signature generator which generates, for each of the retrieved images, an image signature based on statistics computed on image data of the image, the image data being based on features of pixels of the image;

an image classification component which has been trained on a finite set of classes for classifying the retrieved images to generate image-level statistics, the classifier classifying the retrieved images with class labels based on their image signatures and filtering a class label of a retrieved image based on a computed geographical density of other retrieved images with a same class label;

a profile component for aggregating the image-level statistics to generate a place profile for the geographically-defined place; and

a processor which implements the place definition component, image selection component, signature generator, image classification component, and profile component.

20. The system of claim 19, further comprising a graphical user interface for a user to query the system about a particular place and to visualize the place profile.