IMAGE RECOGNITION BY IMAGE SEARCH

• US 20140376819A1
• Filed: 06/21/2013
• Published: 12/25/2014
• Est. Priority Date: 06/21/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Ce, Weiss, Yair, Torralba Barriuso, Antonio
• Granted Patent: US 9,754,177 B2
• Time in Patent Office: Days
• US Class Current: 382/205
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/52 Scale-space analysis, e.g. wavelet analysis multi-scale boundary representations G06V10/42
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects

Abstract

One or more aspects of the subject disclosure are directed towards identifying objects within an image via image searching/matching. In one aspect, an image is processed into bounding boxes, with the bounding boxes further processed to each surround a possible object. A sub-image of pixels corresponding to the bounding box is featurized for matching with tagged database images. The information (tags) associated with any matched images is processed to identify/categorize the sub-image and thus the object corresponding thereto.

Claims

1. A computer-implemented method, comprising, processing an image, including computing a bounding box corresponding to a potential object within an image, extracting a sub-image corresponding to the bounding box, querying an image database using data corresponding to the sub-image to find a set of one or more matched images for the sub-image, and identifying the sub-image based upon data associated with the set of one or more matched images.

2. The method of claim 1 wherein computing the bounding box comprises segmenting the image into segments, generating bounding boxes from the segments, and keeping bounding boxes that meet a variance threshold.

3. The method of claim 2 further comprising, grouping bounding boxes together based upon similarity.

4. The method of claim 1 extracting features from the sub-image to obtain the data corresponding to the sub-image.

5. The method of claim 4 wherein extracting the features comprises using multi-orientation steerable filters.

6. The method of claim 1 wherein identifying the sub-image comprises integrating or pruning the search results, or both integrating and pruning the search results to fit to the data corresponding to the sub-image.

7. The method of claim 6 wherein integrating or pruning the search results, or both integrating and pruning the search results comprises determining which tags of the one or more matched images are semantically coherent.

8. The method of claim 6 wherein integrating or pruning the search results, or both integrating and pruning the search results comprises pruning noisy tags.

9. The method of claim 1 further comprising, computing another bounding box corresponding to another potential object within the image, extracting another sub-image corresponding to the other bounding box, querying the image database using other data corresponding to the other sub-image to find another set of one or more matched images for the other sub-image, and identifying the other sub-image based upon the data associated with the other set of one or more matched images.

10. The method of claim 1 further comprising, outputting a parsed image corresponding to the image in which the sub-image is labeled based upon the identifying of the sub-image.

11. A system comprising, a bounding box generation stage of a pipeline configured to select a sub-image from an input image, an image-to-image search stage of the pipeline configured to query an image database with data corresponding to the sub-image to obtain identification-related or category-related information of one or more matched images, and an identification stage configured to process the identification-related and/or category-related information corresponding to the one or more matched images to identify and/or categorize the sub-image.

12. The system of claim 11 wherein the pipeline is incorporated into at least one of:

a search engine, a mobile computing device, a personal computing device, or a cloud service.

13. The system of claim 11 wherein the identification stage is further configured to output a parsed image labeled with an identifier or a category.

14. The system of claim 11 wherein the image-to-image search stage is further configured to extract features from the sub-image, and to query an image database with at least some of the features to obtain the identification-related or category-related information of one or more matched images.

15. The system of claim 14 wherein the features comprise global features.

16. The system of claim 14 wherein the identification stage is configured to aggregate the identification-related and/or category-related information from the matched images based upon at least one local expert classifier.

17. One or more computer-readable storage media or logic having executable instructions, which when executed perform steps, comprising, inputting an image, generating bounding boxes to determine sub-images corresponding to possible objects within the image, determining features of the sub-images, and for each sub-image, querying an image database having tagged images with features corresponding to the sub-image to match an object corresponding to the sub-image with zero or more matched tagged images in the image database.

18. The one or more computer-readable storage media or logic of claim 17 having further executable instructions comprising, for each image, processing information associated with the zero or more matched tagged images in the image database to categorize or identify the object.

19. The one or more computer-readable storage media or logic of claim 17 wherein generating the bounding boxes comprises segmenting the image into segments and processing the segments.

20. The one or more computer-readable storage media or logic of claim 17 having further executable instructions comprising, labeling objects in the image that are matched to at least one tagged image in the image database.