Photography assistant and method for assisting a user in photographing landmarks and scenes

• US 8,332,429 B2
• Filed: 06/22/2010
• Issued: 12/11/2012
• Est. Priority Date: 06/22/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Poirier, Herve, Perronnin, Florent, Jarmasz, Mario Agustin Ricardo
• Primary Examiner(s): Vo, Truong
• Application Number:
  US12/820,647
• Publication Number:
  US 20110314049A1
• Time in Patent Office: 903 Days
• Field of Search: 707/781
• US Class Current: 707/781
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: H04N 1/00183 Photography assistance, e.g. displaying suggestions to the user
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

A method and system to help photographers to take better quality pictures of landmarks and scenes are disclosed. A user is guided with examples of existing quality images, which are extracted from a database, of the same or similar landmarks or scenes. The method includes taking a query photograph that may include an image associated with a GPS location and other metadata, and using information extracted from the image to retrieve existing, similar images. The images retrieved may be ordered according to different criteria. When a user selects one as a model image, the user is provided with assistance for taking a target photograph of similar quality.

Claims

1. A method for assisting a user in taking photographs comprising:

receiving a query photograph comprising an image;

with a processor, extracting an image signature from the query photograph based on content-related features extracted from patches of the image, the extracting of the image signature including extracting a descriptor from each patch, each descriptor comprising a feature vector or histogram, the image signature comprising a vectorial representation of the descriptors;

querying a database of photographs based on the image signature to retrieve photographs similar to the query image;

presenting at least a subset of the retrieved photographs for the user to select a model photograph from among the presented photographs;

based on a user-selected model photograph, generating instructions for taking a target photograph; and

presenting the instructions to the user for taking the target photograph.

2. The method of claim 1, wherein the presenting of the at least a subset of the images includes presenting images based on at least one sorting criterion.

3. The method of claim 1, wherein the sorting criterion is selected from the group consisting of a quality indication, a visual similarity, location information, setting information for an image capture device, timing information, and combinations thereof.

4. The method of claim 1, wherein the query photograph comprises a landmark or a scene.

5. The method of claim 1, wherein the query photograph comprises metadata, and wherein the querying of the database is based on at least some of the metadata.

6. The method of claim 5, wherein the metadata comprises at least one of:

a global positioning system (GPS) location, settings of an image capture device which generated the photograph, model information of the image capture device, timing information, and information related to the query photograph.

7. The method of claim 1, wherein presenting the instructions comprises at least one of:

a) where the model photograph is associated with a GPS location, at least one of;

indicating to the user on a map a location where the model photograph was taken, and

providing directions to that location;

b) where the model photograph is associated with settings, providing suggestions to the user related to at least one setting for taking the target photograph, based on the settings of the model photograph;

or

c) presenting the model photograph in one display of a double display or displaying a landmark or scene within the model photograph in a background while the target photograph is concurrently displayed on the foreground using a transparency effect.

8. The method of claim 1, wherein presenting the instructions comprises computing geometrical transformation data between a view within a frame of an image capture device and the model photograph and at least one of:

presenting directions for increasing a similarity level between the view and the model photograph to the user, and

providing a signal when the view within the frame of the image capture device is determined to be substantially similar to the model photo.

9. The method of claim 1, wherein presenting the assistance comprises:

detecting a conflict for taking the target photograph having a similar quality to the model photograph, based on time information associated with the model photograph, and

proposing to the user a suitable future time to take the target photograph when environmental conditions are predicted to be similar to those when the model photograph was taken.

10. The method of claim 9, wherein the environmental conditions comprise at least one of an amount of sunlight and a weather condition.

11. The method of claim 1, wherein the querying of the database comprises comparing the image signature of the query photograph to image signatures of images in the database.

12. The method of claim 1 further comprising filtering the retrieved images based on at least one of a GPS location associated with the images and comments associated with the images which relate to at least one of a location of the image and a quality of the image.

13. The method of claim 1, wherein the image capture device comprises a camera or a mobile phone capable of communicating the query image and receiving retrieved images over a wireless network.

14. A computer program product comprising a non-transitory recording medium encoding instructions which, when executed by a computing device, perform the method of claim 1.

15. The method of claim 1, wherein the extracting an image signature includes extracting the descriptors from the patches based on at least two of texture, color, shape, and structure.

16. The method of claim 1, wherein the patches are obtained by image segmentation, by applying specific interest point detectors, by considering a regular grid, or by random sampling.

17. The method of claim 1, wherein the extracting an image signature comprises assigning each of the descriptors to its closest visual word in a previously trained vocabulary of visual words or to all the visual words in a probabilistic manner.

18. The method of claim 17, wherein the extracting an image signature from the query photograph includes modeling the query image as a set of mixture weights, one for each of a set of Gaussian mixture models.

19. The method of claim 1, wherein the descriptors comprise histograms of gradients.

20. The method of claim 1, wherein the querying of the database of photographs based on the image signature to retrieve photographs similar to the query image comprises computing a vector set comparison measure based on the respective image signatures of the query image and database photographs.

21. A system for assisting a user in taking photographs, comprising:

memory which stores;

a user selection component for receiving a user'"'"'s selection of a query photograph and of a model photograph;

an image shuttle for transferring the query photograph to an associated query system for querying a database based on an image signature extracted from the query image and for receiving images retrieved from the database in response to the query, the image signature comprising a vectorial representation of descriptors extracted from patches of the query image extracted by assigning each of the descriptors to its closest visual word in a previously trained vocabulary of visual words or to all the visual words in a probabilistic manner;

a display generator for generating a display of at least a subset of the retrieved images on an associated display device from which the model photograph is selected; and

an instruction generator configured to present instructions to the user for taking a target photograph based on the selected model photograph; and

a processor in communication with the memory for implementing the user selection component, image shuttle, display generator, instruction generator.

22. The system of claim 21, wherein the memory further stores a sorting component for sorting the retrieved photographs based on at least one of a quality indication, a similarity weighting, a location information, setting information for an image capture device, and a timing information related to the query photograph.

23. The system of claim 21, wherein the query system queries the database based on metadata associated with the image.

24. The system of claim 21, wherein the image signature further comprises information variables having at least one of a viewpoint, zooming, light, occlusion and quality information.

25. The system of claim 21, further comprising a geometrical transformation engine configured to compute a geometrical transformation between a view in a frame of an image capture device and the model photograph and for assisting a user in aligning the view with the model photograph.

26. The system of claim 21, further comprising an image categorization mechanism coupled to the instruction generator and configured to categorize the model photograph and the target photograph into a time of day, and/or weather information for providing suggestions to the user.

27. An image capture device comprising the system of claim 21, and the display device.

28. The image capture device of claim 27, wherein the image capture device comprises a camera or a mobile phone capable of communicating the query photograph and receiving the retrieved photographs over a wireless network.

29. A method for assisting a user in taking photographs executed by at least one processor with at least one memory storing executable instructions for performing the method, comprising:

at a computer device, receiving a selected query photograph submitted from an image capture device remote from the computer device, the photograph comprising at least one digital image;

extracting information from the received query photograph, the information including an image signature derived from the digital image;

retrieving photographs from a database based on a comparison of the information extracted from the query photograph with information extracted from photographs in the database, the information extracted from photographs in the database including, for each photograph, an image signature derived from he digital image;

presenting at least a subset of the retrieved photographs for a user of the image capture device to select a model photograph therefrom;

receiving a selection of a model photograph from among the presented photographs; and

generating instructions to guide the user for taking a target photograph comprising a similar quality to the selected model photograph, including calculating a geometrical transformation between the model photograph and an image being framed by the image capture device to be the target photograph.

30. The method of claim 29, wherein the information further comprises metadata associated with the query image and database images respectively.