Scene classification for place recognition

• US 8,238,671 B1
• Filed: 12/07/2009
• Issued: 08/07/2012
• Est. Priority Date: 12/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Flynn, John, Neven, Hartmut, Babenko, Boris
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US12/632,284
• Time in Patent Office: 974 Days
• Field of Search: None
• US Class Current: 382/224
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

Aspects of the invention pertain to identifying whether or not an image from a user'"'"'s device is of a place. Before undertaking time and resource consuming analysis of an image using specialized image analysis modules, pre-filtering classification is conducted based on image data and metadata associated with the image. The metadata may include geolocation information. One classification procedure analyzes the metadata to perform a high level determination as to whether the image is of a place. If the results indicate that it is of a place, then a further classification procedure may be performed, where the image information is analyzed, with or without the metadata. This process may be done concurrently with a place match filtering procedure. The results of the further classification will either find a match with a given place or not. The output is a place match either with or without geolocation information.

Claims

1. A method of classifying an image, comprising:

receiving an image and metadata of the image from a user device;

performing a first classification of the image using only the metadata to determine whether the image is a place;

performing a second classification of the image using both the metadata and information about the image to determine whether the image is the place; and

if either the first or second classification determines that the image is of the place, then providing place information to the user device;

wherein if neither the first nor the second classification determines that the image is of the place, then the method further comprises performing filtering to determine whether the image is of an object or a person.

2. The method of claim 1, further comprising performing place match filtering on the image by comparing features of the image against a database of stored imagery.

3. The method of claim 2, wherein performing the second classification is done in parallel with performing the place match filtering.

4. The method of claim 1, wherein the first classification is performed prior to the second classification.

5. The method of claim 1, wherein if different filter modules identify the image as being of different types, then the first and second classifications are used to select which filter module identification is correct.

6. The method of claim 1, wherein the image information is selected from the group consisting of edges, color settings, texture, zoom level and shutter speed.

7. The method of claim 1, wherein the metadata includes a location of the user device when the image was taken.

8. The method of claim 1, wherein the metadata is selected from the group consisting of location coordinates, device orientation, accelerometer information and location accuracy.

9. The method of claim 8, wherein the device orientation includes at least one of angle, pitch, roll or compass point direction.

10. An apparatus for classifying an image, the apparatus comprising:

means for communicating with a mobile user device; and

at least one processor coupled to the communicating means, the at least one processor configured to;

receive an image and metadata of the image from a user device;

store the image and metadata in memory;

perform a first classification of the image using only the metadata to determine whether the image is a place; and

perform a second classification of the image using both the metadata and information about the image to determine whether the image is the place;

wherein if either the first or second classification determines that the image is of the place, then the processor is configured to provide place information to the user device; and

wherein if neither the first nor the second classification determines that the image is of the place, then the processor is further configured to perform filtering to determine whether the image is of an object or a person.

11. The apparatus of claim 10, wherein the at least one processor is further configured to perform place match filtering on the image by comparing features of the image against a database of stored imagery.

12. The apparatus of claim 11, wherein the at least one processor performs the second classification in parallel with the place match filtering.

13. The apparatus of claim 10, wherein the metadata includes location information of the user device when the image was taken.

14. The apparatus of claim 13, wherein the location information includes at least one of location coordinates, device orientation, accelerometer information and location accuracy.

15. The apparatus of claim 10, wherein the at least one processor is configured to perform a training procedure on a set of training images to set false positive and false negative thresholds for the first and second classifications.

16. The apparatus of claim 15, wherein the training procedure employs a statistical machine learning process to obtain measurement weights and measurement thresholds.