Scene classification for place recognition

• US 8,798,378 B1
• Filed: 08/08/2013
• Issued: 08/05/2014
• Est. Priority Date: 12/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Babenko, Boris, Adam, Hartwig, Flynn, John, Neven, Hartmut
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US13/962,352
• Time in Patent Office: 362 Days
• Field of Search: None
• US Class Current: 382/224
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

Aspects of the disclosure pertain to identifying whether or not an image from a user'"'"'s device is of a place or not. As part of the identification, a training procedure may be performed on a set of training images. The training procedure includes performing measurements of image data for each image in the set to derive a result. The result includes a series of variables for each training image in the set. The series of variable is evaluated for each training image to obtain one or more measurement weights and one or more measurement thresholds. These weights and thresholds are adjusted to set a false positive threshold and a false negative threshold for identifying whether an actual image is of a place type or is some other type of image.

Claims

1. A computer-implemented training method for classifying an image, the method comprising:

providing, by one or more computing devices, a set of training images, each training image being labeled to indicate whether it is a place type;

performing, by the one or more computing devices, measurements of image data for each image in the set of training images to derive a result, the result comprising a series of variables for each training image in the set;

for each training image, determining by the one or more computing devices and based at least in part on the series of variables, one or more measurement weights and one or more measurement thresholds;

adjusting, by the one or more computing devices, the one or more measurement weights and one or more measurement thresholds to set a false positive threshold and a false negative threshold for identifying whether an actual image is of the place type;

performing, by the one or more computing devices, a first classification of the actual image based on at least one of the false positive threshold and the false negative threshold using only metadata of the actual image to determine whether the actual image is of the place type;

performing, by the one or more computing devices, a second classification of the actual image based on at least one of the false positive threshold and the false negative threshold using both the metadata and information about the actual image to determine whether the actual image is of the place type; and

when neither the first nor the second classification determines that the actual image is of the place type, then the method further comprises performing filtering to determine whether the actual image is of an object or a person.

2. The method of claim 1, further comprising resizing, by the one or more computing devices, one or more images in the set of training images prior to performing the measurements.

3. The method of claim 1, further comprising identifying, by the one or more computing devices, image orientation of each image prior to performing the measurements.

4. The method of claim 1, further comprising applying, by the one or more computing devices, the measurement weights and measurement thresholds on the actual image to determine whether the actual image is of the place type.

5. The method of claim 4, wherein each measurement weight (α

_i) and each measurement thresholds (θ

_i) is applied to obtain a weighted output expressed according to the following equation;

weighted output=Σ

_iα

_i1[x_i>

θ

_i],

wherein (x_i) indicates each respective measurement.

6. The method of claim 4, wherein the one or more computing devices determine whether the actual image is of the place type, at least in part, by applying a final threshold against the weighted output.

7. The method of claim 6, further comprising adjusting, by the one or more computing devices, the final threshold to achieve at least one of a determined false positive rate and a determined false negative rate.

8. The method of claim 1, wherein the image data is selected from the group consisting of:

any blurry or sharp regions, color settings, color correction settings, a color histogram, vertical edges, horizontal edges, line counts, any image processing filtering, a zoom level, a focal length, a focus distance, a shutter speed, a texture, and an indication of whether there is any text in a given image in the set of training images.

17. The method of claim 1, wherein when different filter modules identify the actual image as being of different types, then the first and second classifications are used to select which filter module identification is correct.

9. A non-transitory computer-readable recording medium storing a computer program therein, the computer program, when executed by one or more computing devices, causing the one or more computing devices to perform a training method for classifying an image, the method comprising:

providing, by the one or more computing devices, a set of training images, each training image being labeled to indicate whether it is a place type;

performing, by the one or more computing devices, measurements of image data for each image in the set of training images to derive a result, the result comprising a series of variables for each training image in the set;

for each training image, determining by the one or more computing devices and based at least in part on the series of variables, one or more measurement weights and one or more measurement thresholds; and

adjusting, by the one or more computing devices, the one or more measurement weights and one or more measurement thresholds to set a false positive threshold and a false negative threshold for identifying whether an actual image is of the place type;

performing, by the one or more computing devices, a first classification of the actual image based on at least one of the false positive threshold and the false negative threshold using only metadata of the actual image to determine whether the actual image is of the place type;

performing, by the one or more computing devices, a second classification of the actual image based on at least one of the false positive threshold and the false negative threshold using both the metadata and information about the actual image to determine whether the actual image is of the place type; and

when neither the first nor the second classification determines that the actual image is of the place type, then the method further comprises performing filtering to determine whether the actual image is of an object or a person.

18. The non-transitory computer-readable recording medium of claim 9, wherein when different filter modules identify the actual image as being of different types, then the first and second classifications are used to select which filter module identification is correct.

10. A system for training classifying of imagery, the system comprising:

memory storing a set of training images, each training image being labeled to indicate whether it is a place type;

one or more computing devices coupled to the memory, the one or more computing devices configured to;

perform measurements of image data for each image in the set of training images to derive a result, the result comprising a series of variables for each training image in the set;

for each training image, determining, based at least in part on the series of variables, one or more measurement weights and one or more measurement thresholds; and

adjust the one or more measurement weights and one or more measurement thresholds to set a false positive threshold and a false negative threshold for identifying whether an actual image is of the place type;

perform a first classification of the actual image based on at least one of the false positive threshold and the false negative threshold using only metadata of the actual image to determine whether the actual image is of the place type;

perform a second classification of the actual image based on at least one of the false positive threshold and the false negative threshold using both the metadata and information about the actual image to determine whether the actual image is of the place type; and

when neither the first nor the second classification determines that the actual image is of the place, then the one or more computing devices are further configured to perform filtering to determine whether the actual image is of an object or a person.

11. The system of claim 10, wherein the one or more computing devices are further configured to resize one or more images in the set of training images prior to performing the measurements.

12. The system of claim 10, wherein the one or more computing devices are further configured to identify an image orientation of each image prior to performing the measurements.

13. The system of claim 10, wherein the one or more computing devices are further configured to apply the measurement weights and measurement thresholds on the actual image to determine whether the actual image is of the place type.

14. The system of claim 13, wherein each measurement weight (α

_i) and each measurement thresholds (θ

_i) is applied to obtain a weighted output expressed according to the following equation;

weighted output=Σ

_iα

_i1[x_i>

θ

_i],

wherein (x_i) indicates each respective measurement.

15. The system of claim 13, wherein the one or more computing devices are configured to determine whether the actual image is of the place type, in part, by applying a final threshold against the weighted output.

16. The system of claim 15, wherein the one or more computing devices are further configured to adjust the final threshold to achieve at least one of a determined false positive rate and a determined false negative rate.

19. The system of claim 10, wherein when different filter modules identify the actual image as being of different types, then the first and second classifications are used to select which filter module identification is correct.