METHOD AND APPARATUS FOR RECOGNIZING AND LOCALIZING LANDMARKS FROM AN IMAGE ONTO A MAP

• US 20110150324A1
• Filed: 12/22/2009
• Published: 06/23/2011
• Est. Priority Date: 12/22/2009
• Status: Active Grant

• Current Assignee: Chinese University of Hong Kong
• Original Assignee: Chinese University of Hong Kong
• Inventors: Ngan, King Ngi, Cheng, Bangsheng
• Granted Patent: US 8,180,146 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/39 Urban scenes

Abstract

Method and apparatus for recognizing landmark buildings in an image and then locating the recognized landmark buildings onto a map together with related information wherein a first database is employed to store models formed by mathematical set descriptions of landmark buildings which are learned from a set of training images of a model-learning module captured by an imaging device for each building, and a second database is employed to store the related information of each landmark building. The model of each landmark building is represented as a set of features and the geometric relationship between them by clustering the salient features extracted from a set of training images of the landmark building.

Claims

1. A method for recognizing landmark buildings shown in an input image and locating recognized landmark buildings on a map with related information, the method comprising:

capturing a set of training images of one landmark building and learning a model of the landmark building from the set of training images;

determining similarity scores for image pairs of different ones of said training images;

clustering the training images into a set of prototypes based upon the similarity scores of said each pair of training images;

storing said set of models learned for a set of landmark buildings to a storage device to build a first model database, wherein each model of one landmark building from said set of landmark buildings has a record in said database indexed by an identifier code for said landmark building, and each prototype has a record in said database linked to the record of corresponding landmark building model;

capturing an input image of a candidate building;

calculating the similarity score between said input image and each model stored in the model database;

recognizing the building in said input image based on the largest value of the similarity scores between said input image and the models stored in the first database;

locating the recognized landmark building on an electronic map;

retrieving information related to the recognized landmark building from a second database operative under control of a processor circuit; and

displaying the related information of the recognized landmark building on a display device to an end user.

2. The method according to claim 1 wherein the prototypes producing step for one landmark building comprises:

calculating a prototypic score for each image in the training image set of one landmark building by summing all similarity scores of said each image with other images in said training image set, and selecting the image with the largest prototypic score as the first prototype of said landmark building;

excluding the selected prototype image and the images which have similarity scores as said selected prototype image greater than a threshold from said training image set, then updating prototypic score for each image in the updated training image set by summing all similarity scores of said each image with other images in said updated training image set, and selecting the image with the largest prototypic score as the next prototype of said landmark building;

repeating the above process until there is less than two images in said training image set, and if there is only one image in the training image set, selecting said only one image as the last prototype of said landmark building; and

storing the local salient features extracted from each prototype image as feature records to the first database linked to one prototype record of said prototype, which is linked to the model record of said landmark building.

3. The method according to claim 1 wherein producing the similarity score of one pair of training images comprises:

extracting a set of local salient features from each image of said training image pair;

generating descriptors for each local salient feature extracted from the images of said training image pair;

calculating the similarity score of the set of local salient features extracted from one image of said image pair with the set of local salient features extracted from the other image of said image pair; and

returning the similarity score of said two sets of local salient features as the similarity score of said training image pair.

4. The method claimed in claim 3 wherein the step of producing a set of local salient features for one image comprises:

producing an image pyramid by successively smoothing and sub-sampling said image;

for each image level of the image pyramid, producing a set of multi-scale saliency maps; and

locating pixel saliency maxima in the set of multi-scale saliency maps with corresponding scale value.

5. The method according to claim 3 comprising producing a plurality of descriptors for each local salient feature, said descriptors producing step comprising:

extracting a local image region around a feature point with size proportional to scale value of said feature point;

calculating the gradient field of said local image region;

calculating dominant orientations of the said extracted local region;

for each dominant orientation, normalizing the gradient field according to this dominant orientation, and subdividing said local image region to several overlapped sub-regions relative to the dominant orientation;

for each sub-region, creating an orientation histogram based on the magnitude and normalized orientation field according to the dominant orientation, and transforming this histogram to a binary bit string; and

assembling said binary bit strings of said sub-regions into one global binary bit string according to an order relative to said dominant orientation as one descriptor corresponding to the feature point with said dominant orientation.

6. The method according to claim 3 wherein the similarity score for two sets of local salient features is produced by:

calculating Hamming distance of the descriptors of each feature pair from said two sets of local salient features, with one feature each from the said two sets of local salient features, and adding each said feature pair to a candidate matching feature pair set if said Hamming distance less than a threshold;

estimating the geometric transformation parameter value by matching locations, scale values, and dominant orientations of the two features from each feature pair, and voting the resultant parameter value to bins of a histogram of geometric transformation parameter space;

selecting the histogram bin with the largest voting value, and finding the set of features voting positive values to the bin so selected to form a refined candidate matching feature pair set;

applying a RANSAC algorithm to said refined candidate matching feature pair set to estimate the geometric transformation parameter, and finding the inlier set of matching feature pairs from said refined candidate matching feature pair set as the true matching feature pair set; and

returning the number of feature pairs in said true matching feature set as the similarity score of said two sets of local salient features if said number is greater than a threshold;

otherwise, returning the similarity score of said two sets of local salient features as zero.

7. The method according to claim 5 comprising producing a plurality of dominant orientations for the local image region corresponding to one feature point, said dominant orientations producing step comprising:

calculating the gradient field of said local image region;

calculating gradient orientation and gradient magnitude for each pixel in the local image region, the gradient magnitude being weighted by a Gaussian function with deviation proportional to the size of the said local image region;

for each pixel in the local image region, voting the corresponding orientation with the value of weighted orientation magnitude;

selecting as candidates of dominant orientations maxima of the orientation histogram; and

selecting as the dominant orientations of the local image region the orientation with the largest vote value and such other orientations with maximal vote value not less than a threshold relative to the largest vote value.

8. The method according to claim 1 further comprising producing a tag to indicate the recognized landmark building in one input image, said tag producing step comprising:

calculating similarity scores of said input image with each landmark building model stored in the first database;

calculating the largest value of the similarity scores of said input image with all landmark building models stored in the first database, and recording the corresponding landmark building'"'"'s ID; and

returning said landmark building'"'"'s ID as the tag of recognized landmark building in said input image if the said largest similarity score is greater than a threshold;

otherwise, returning nothing to indicate no landmark building is recognized in said input image.

9. The method according to claim 8 wherein producing the similarity score for said input image with one prototype of one landmark building stored in the landmark building model database comprises:

extracting the set of local salient features from said input image and creating descriptors for each local salient feature;

retrieving the set of local features of said prototype from the landmark building model database; and

calculating the similarity score value of the set of local features from said input image and the set of local features from said prototype, and return this similarity score value as the similarity score of said input image with said prototype.

10. The method according to claim 9 wherein said similarity score producing step comprises comparing similarity scores of said input images with all prototypes of the said landmark building, and selecting the largest similarity score value as the similarity score of said input image with the said landmark building model.

11. The method according to claim 1 wherein the step of localizing the recognized landmark building in an input image on a map with related information comprises:

retrieving the location information of the landmark building from information stored in a storage device indexed by the landmark buildings, then the marking on the location of said recognized landmark building, and then retrieving for display from the information database indexed by landmark building the related information of the recognized landmark building.

12. A system for recognizing landmark buildings in an image and for locating the recognized buildings on a map with related information, the system comprising:

at least one processor circuit and at least one storage, said circuit and storage configured for;

learning models of landmark buildings from a set of training images stored in a first database operative under control of at the least one processor circuit, said learning for each landmark building comprising;

determining a similarity score between each pair of the training images within said set;

clustering the training images into a set of prototypes based on the similarity score of each pair of training images; and

storing the set of prototypes to the at least one storage to build a model database of the landmark buildings for subsequent comparison with input images of candidate buildings; and

at least one processor circuit, at least one storage and a display device configured for;

inputting an image of a candidate building to the at least one processor circuit;

calculating a similarity score of the input image to a model of each landmark building, said calculating comprising comparing local salient features extracted from the input image with local salient features of prototypes of landmark buildings stored in a model database, in order to recognize the landmark building;

locating the recognized building on an electronic map;

retrieving information related to the recognized building from a second database operative under control of the at least one processor circuit; and

displaying the related information of the recognized landmark building on the display device to an end user.

13. The apparatus according to claim 12 wherein the at least one processor and storage are further configured for:

clustering the local salient features extracted from the set of training images, the training images being taken from different positions, different viewpoints and different times of each landmark building.

14. An apparatus for use in a system for recognizing landmark buildings in an image and for locating the recognized buildings on a map with related information, the apparatus comprising:

at least one processor circuit;

at least one storage; and

a display device,

said at least one processor circuit, said at least one storage and said display device configured for;

inputting an image of a candidate building to the at least one processor circuit;

calculating a similarity score of the input image to a model of each landmark building, said calculating comprising comparing local salient features extracted from the input image with local salient features of prototypes of landmark buildings stored in a model database, in order to recognize the landmark building;

locating the recognized building on an electronic map;

retrieving information related to the recognized building from a second database operative under control of the at least one processor circuit; and

displaying the related information of the recognized landmark building on the display device to an end user.