LOCATION-AIDED RECOGNITION

• US 20120321175A1
• Filed: 06/17/2011
• Published: 12/20/2012
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sinha, Sudipta, Zitnick, Charles Lawrence, Szeliski, Richard, Hedau, Varsha
• Granted Patent: US 9,152,882 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A mobile device having the capability of performing real-time location recognition with assistance from a server is provided. The approximate geophysical location of the mobile device is uploaded to the server. Based on the mobile device'"'"'s approximate geophysical location, the server responds by sending the mobile device a message comprising a classifier and a set of feature descriptors. This can occur before an image is captured for visual querying. The classifier and feature descriptors are computed during an offline training stage using techniques to minimize computation at query time. The classifier and feature descriptors are used to perform visual recognition in real-time by performing the classification on the mobile device itself.

Claims

1. A computer-implemented method, comprising:

using a geophysical location to obtain a classifier, the classifier associated with one or more classes, each class used to describe an image;

capturing an image associated with the geophysical location, the image having an unknown class; and

applying the classifier to the image to determine a class for the image.

2. The method of claim 1, further comprising:

requesting the classifier from a first device, the request containing the geophysical location of the first device; and

obtaining the classifier from a second device remotely located from the first device.

3. The method of claim 1, further comprising:

training the classifier from one or more training images corresponding to the geophysical location prior to capturing the image, each training image associated with one or more classes.

4. The method of claim 3, further comprising:

extracting one or more candidate features from each training image of a class; and

determining a subset of features from the candidate features.

5. The method of claim 4, further comprising:

rectifying each training image prior to extracting the candidate features.

6. The method of claim 4, further comprising:

representing each feature of the subset of features as a feature descriptor;

calculating a table having a similarity score for each feature descriptor corresponding to each training image by searching for a most similar feature descriptor using min hash to accelerate the search; and

training the classifier using the similarity scores.

7. The method of claim 6, further comprising:

configuring each feature descriptor as a BRIEF feature descriptor.

8. The method of claim 7, further comprising:

configuring the classifier as a random decision forest.

9. The method of claim 1, further comprising:

obtaining a plurality of feature descriptors associated with one or more classes at the geophysical location;

densely matching the feature descriptors across the captured image; and

generating a similarity score for each feature descriptor, the similarity score representing a closest match to an image patch found in the captured image.

10. The method of claim 9, the generating step further comprising:

calculating a hamming distance for each feature descriptor representing a smallest distance between the feature descriptor and a closest matching image patch in the captured image.

11. A computer-readable storage medium storing thereon processor-executable instructions, comprising:

instructions to obtain one or more feature descriptors;

instructions to determine a similarity score for each feature descriptor with respect to a captured image, the similarity score representing a closest match of a feature descriptor to an image patch in the captured image, the similarity score computed for those feature descriptors matching an image patch in the captured image, the matching feature descriptors determined by a min-hash process; and

instructions to recognize the captured image through use of the similarity scores.

12. The computer-readable storage medium of claim 11, further comprising:

instructions to determine a geophysical location that is used to obtain the feature descriptors, the feature descriptors associated with the geophysical location.

13. The computer-readable storage medium of claim 11, further comprising:

instructions to obtain a classifier associated with a geophysical location, the geophysical location associated with the feature descriptors; and

instructions to apply the classifier to the feature descriptors and similarity scores to determine at least one probability that the captured image pertains to a class.

14. The computer-readable storage medium of claim 11, further comprising:

instructions to rectify the captured image prior to determination of the similarity score.

15. The computer-readable storage medium of claim 11, further comprising:

instructions to represent the similarity score as a hamming distance; and

instructions that compute the hamming distance for those feature descriptors identified as having a matching image patch.

16. A computer-readable storage medium storing thereon processor-executable instructions, comprising:

instructions to obtain one or more feature descriptors from one or more training images, the training images associated with a common geophysical location;

instructions to determine a similarity score for each feature descriptor with respect to each training image, the similarity score representing a similarity between a feature descriptor and a most similar feature descriptor amongst all feature descriptors corresponding to image patches in a training image, the matching feature descriptors determined by performing Hamming distance computations which is accelerated using a min-hash process applied to the feature descriptors with respect to a training image; and

instructions to train a classifier through use of the similarity scores.

17. The computer-readable storage medium of claim 16, further comprising:

instructions to select the feature descriptors from image patches in the training images having more salient features.

18. The computer-readable storage medium of claim 16, further comprising:

instructions that generate one or more random decision trees as the classifier, each random decision tree trained using the feature descriptors and the similarity scores.

19. The computer-readable storage medium of claim 16, further comprising:

instructions to represent the similarity score as a hamming distance; and

instructions that compute the hamming distance for those feature descriptors matching an image patch in a training image.

20. The computer-readable storage medium of claim 19, further comprising:

instructions that min-hash the feature descriptors to determine those feature descriptors having similar min-hash values; and

instructions that compute the Hamming distance for those feature descriptors having similar min-hash sketches.